-----============= acceptance-small: sanity-flr ============----- Thu Oct 23 11:59:28 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg316-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:59:46 (1761235186) === oleg316-client.virtnet: executing check_config_client /mnt/lustre oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg316-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ccec5426000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ccec5426000.idle_timeout=debug disable quota as required oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:00:20 (1761235220) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8186] == sanity-flr test 0a: lfs mirror create with -N option == 12:00:22 (1761235222) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:00:34 (1761235234) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:00:37 (1761235237) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:00:40 (1761235240) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:01:01 (1761235261) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:01:03 (1761235263) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:01:06 (1761235266) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ccec5426000.stats=clear osc.lustre-OST0001-osc-ffff8ccec5426000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:01:14 (1761235274) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:01:23 (1761235283) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:01:30 (1761235290) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:01:33 (1761235293) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 12:01:41 (1761235301) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:01:47 (1761235307) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:01:54 (1761235314) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:02:01 (1761235321) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:02:10 (1761235330) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:02:16 (1761235336) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.11278 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553436 s, 18.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:02:25 (1761235345) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.333693 s, 40.9 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:02:33 (1761235353) Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (62s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:03:36 (1761235416) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.367327 s, 20.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (30s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:04:06 (1761235446) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (53s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:04:59 (1761235499) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.87565 s, 18.2 MB/s -rw-r--r-- 1 root root 52428800 Oct 23 12:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.905s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.798s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.693s user 0m0.000s sys 0m0.187s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ccec5426000.ost_server_uuid in FULL state after 0 sec PASS 33b (67s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:06:06 (1761235566) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:06:09 (1761235569) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:06:12 (1761235572) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 12:06:14 (1761235574) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 12:06:19 (1761235579) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.389288 s, 32.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:06:28 (1761235588) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323856 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244914 s, 42.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:06:35 (1761235595) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:06:42 (1761235602) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000740053 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000761693 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000929325 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000852229 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:06:51 (1761235611) create test files with size 15754512 .. 1df52f97d75abaa67bf541c05ba7a28d - 870aa14bb3cc0ef7af47d92a396f2f28 - a5b8e25db587b90a5b794742ef41350e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 12:07:08 (1761235628) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:07:20 (1761235640) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:07:25 (1761235645) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959842 s, 10.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028599 s, 36.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 12:07:32 (1761235652) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0571172 s, 36.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.236212 s, 17.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170973 s, 12.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 12:07:41 (1761235661) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:07:44 (1761235664) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:07:47 (1761235667) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456264 s, 23.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 12:07:53 (1761235673) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.215784 s, 14.6 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:08:00 (1761235680) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:08:08 (1761235688) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.2713 s, 38.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:08:18 (1761235698) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ccec8b43000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296653 s, 35.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8cced0c4e800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:08:31 (1761235711) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (29s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:09:00 (1761235740) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (27s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:09:28 (1761235768) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:09:30 (1761235770) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202032 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0934845 s, 33.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 6f3f90adbdeb6fdc1a82f1501906f8ca - mirror 1 checksum: 68358b5b0638b72d6d599b4e0da39f06 - mirror 2 checksum: 6f3f90adbdeb6fdc1a82f1501906f8ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:09:34 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:09:34 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6f3f90adbdeb6fdc1a82f1501906f8ca - mirror 1 checksum: 68358b5b0638b72d6d599b4e0da39f06 - mirror 2 checksum: 68358b5b0638b72d6d599b4e0da39f06 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:09:34 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:09:34 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:09:38 (1761235778) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340428 s, 30.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:09:44 (1761235784) Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 12:09:54 (1761235794) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 12:10:13 (1761235813) targets are mounted 12:10:13 (1761235813) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395695 s, 26.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:10:25 (1761235825) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0528587 s, 387 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.014893 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:10:32 (1761235832) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0435542 s, 470 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00755942 s, 2.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:10:39 (1761235839) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00922053 s, 1.8 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:10:45 (1761235845) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0686855 s, 298 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:10:51 (1761235851) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123307 s, 332 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.047744 s, 85.8 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:10:57 (1761235857) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:11:14 (1761235874) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:11:42 (1761235902) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:12:04 (1761235924) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:12:26 (1761235946) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:12:33 (1761235953) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55913 mirror split pid 55914 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 55913 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 55914 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:13:40 (1761236020) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6021 skipping zero length zero range skipping zero length zero range truncating to largest ever: 0x32780 000004 1761236022.195668 trunc from 00000000 to 0x03277f (0x32780 bytes) 000005 1761236022.215584 mapwrite 0x01e515 thru 0x027069 (0x8b55 bytes) 000007 1761236022.287125 punch from 0x0002aa to 0x007bf5 (0x794c bytes) 000008 1761236022.295736 trunc from 0x0214b4 to 0x02f359 (0xdea6 bytes) 000009 1761236022.302900 trunc from 0x000a97 to 0x00b87b (0xade5 bytes) fallocating to largest ever: 0x40000 000010 1761236022.310087 fallocate 0x03e275 thru 0x03ffff (0x1d8b bytes) 000011 1761236022.318691 fallocate 0x001487 thru 0x00f5da (0xe154 bytes) 000012 1761236022.323680 trunc from 0x040000 to 0x016924 (0x296db bytes) 000013 1761236022.339954 mapread 0x005938 thru 0x007c80 (0x2349 bytes) 000015 1761236022.351635 trunc from 0x016925 to 0x00b285 (0xb69f bytes) 000016 1761236022.370220 mapwrite 0x03091c thru 0x03a97c (0xa061 bytes) 000017 1761236022.425651 mapwrite 0x03ffd9 thru 0x03ffff (0x27 bytes) 000019 1761236022.465932 write 0x013fd8 thru 0x0234d7 (0xf500 bytes) 000020 1761236022.483434 mapread 0x022834 thru 0x023142 (0x90f bytes) 000021 1761236022.484249 mapwrite 0x01c59c thru 0x023b46 (0x75ab bytes) 000022 1761236022.502395 trunc from 0x040000 to 0x01cae5 (0x2351a bytes) 000023 1761236022.526674 trunc from 0x01cae6 to 0x01ff1f (0x343a bytes) 000024 1761236022.553565 mapread 0x009e2a thru 0x016922 (0xcaf9 bytes) 000026 1761236022.556186 mapwrite 0x030afe thru 0x0316b9 (0xbbc bytes) 000027 1761236022.592085 trunc from 0x0200d4 to 0x02e3fa (0xe327 bytes) 000028 1761236022.597433 fallocate 0x03f97a thru 0x03ffff (0x686 bytes) 000029 1761236022.603830 mapwrite 0x014986 thru 0x018661 (0x3cdc bytes) 000030 1761236022.634756 mapwrite 0x023ab7 thru 0x02f133 (0xb67d bytes) 000031 1761236022.691992 mapread 0x02135d thru 0x02d508 (0xc1ac bytes) 000032 1761236022.693048 mapread 0x00ce7c thru 0x00faae (0x2c33 bytes) 000033 1761236022.699257 mapwrite 0x03b4d9 thru 0x03ffff (0x4b27 bytes) 000034 1761236022.782771 mapread 0x0250bb thru 0x031881 (0xc7c7 bytes) 000036 1761236022.789254 read 0x01dd5d thru 0x02422f (0x64d3 bytes) 000037 1761236022.795243 MIRROR_EXTEND to 2 mirrors 000038 1761236023.487564 trunc from 0x040000 to 0x011e0e (0x2e1f1 bytes) 000039 1761236023.532755 mapwrite 0x006253 thru 0x0145e3 (0xe391 bytes) 000040 1761236023.610518 fallocate 0x000ff5 thru 0x00fe7e (0xee8a bytes) 000042 1761236023.625586 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000043 1761236023.780033 fallocate 0x020a12 thru 0x024763 (0x3d52 bytes) 000044 1761236023.786103 read 0x0121c1 thru 0x0145e3 (0x2423 bytes) 000045 1761236023.790721 read 0x013287 thru 0x0145e3 (0x135d bytes) 000046 1761236023.791280 punch from 0x012abe to 0x0145e3 (0x1b26 bytes) 000047 1761236023.797442 trunc from 0x0122a8 to 0x0145e3 (0x233c bytes) 000048 1761236023.803146 trunc from 0x012634 to 0x0145e3 (0x1fb0 bytes) 000049 1761236023.808859 write 0x036c4d thru 0x03b32a (0x46de bytes) 000050 1761236023.817053 trunc from 0x01e394 to 0x01efba (0xc27 bytes) 000051 1761236023.822518 write 0x02c3f4 thru 0x033fdf (0x7bec bytes) 000052 1761236023.833650 trunc from 0x02f558 to 0x02f842 (0x2eb bytes) 000053 1761236023.847307 trunc from 0x03b32b to 0x01970a (0x21c20 bytes) 000054 1761236023.861195 trunc from 0x01970b to 0x03cc73 (0x23569 bytes) 000055 1761236023.876047 trunc from 0x02b362 to 0x03298d (0x762c bytes) 000057 1761236023.882396 trunc from 0x03cc74 to 0x0144bd (0x287b6 bytes) 000058 1761236023.900266 trunc from 0x002064 to 0x005123 (0x30c0 bytes) 000060 1761236023.906112 mapread 0x00e5cf thru 0x0144bd (0x5eef bytes) 000061 1761236023.907111 mapwrite 0x00a668 thru 0x00e66e (0x4007 bytes) 000062 1761236023.924935 mapwrite 0x011b40 thru 0x01cee6 (0xb3a7 bytes) 000063 1761236024.003544 mapread 0x006e4e thru 0x0119cc (0xab7f bytes) 000065 1761236024.004907 punch from 0x013a2d to 0x01c379 (0x894d bytes) 000066 1761236024.012568 read 0x0190dd thru 0x01cee6 (0x3e0a bytes) 000067 1761236024.024095 trunc from 0x01cee7 to 0x03ff2f (0x23049 bytes) 000068 1761236024.041539 mapwrite 0x031212 thru 0x038d81 (0x7b70 bytes) 000069 1761236024.068830 trunc from 0x019c21 to 0x01ef81 (0x5361 bytes) 000070 1761236024.073726 trunc from 0x002020 to 0x00f9a7 (0xd988 bytes) 000071 1761236024.080012 mapread 0x01b142 thru 0x023a8f (0x894e bytes) 000072 1761236024.100017 read 0x01f2b3 thru 0x027b3b (0x8889 bytes) 000073 1761236024.100784 trunc from 0x01f8ce to 0x02d3c2 (0xdaf5 bytes) 000074 1761236024.105579 trunc from 0x03ff30 to 0x034d22 (0xb20d bytes) 000075 1761236024.128951 fallocate 0x03a16e thru 0x03cc0b (0x2a9e bytes) 000077 1761236024.135861 write 0x00db4a thru 0x0111d6 (0x368d bytes) 000078 1761236024.151187 trunc from 0x020c41 to 0x030334 (0xf6f4 bytes) 000079 1761236024.155443 punch from 0x01ec14 to 0x027e32 (0x921f bytes) 000081 1761236024.159941 trunc from 0x008f24 to 0x017300 (0xe3dd bytes) 000082 1761236024.174184 punch from 0x00e3cc to 0x01a851 (0xc486 bytes) 000083 1761236024.178021 mapwrite 0x01ad5e thru 0x0263fa (0xb69d bytes) 000084 1761236024.246505 read 0x0004ec thru 0x00821b (0x7d30 bytes) 000085 1761236024.257409 read 0x010285 thru 0x011963 (0x16df bytes) 000086 1761236024.258036 mapwrite 0x01f3d3 thru 0x01f746 (0x374 bytes) 000088 1761236024.270539 write 0x01003d thru 0x0167ca (0x678e bytes) 000089 1761236024.271757 write 0x00299a thru 0x00e581 (0xbbe8 bytes) 000091 1761236024.274256 punch from 0x01a8e8 to 0x01afb3 (0x6cc bytes) 000092 1761236024.284233 mapwrite 0x01efa3 thru 0x0220ab (0x3109 bytes) 000093 1761236024.304889 fallocate 0x002feb thru 0x00eac0 (0xbad6 bytes) 000095 1761236024.312036 punch from 0x020381 to 0x02292a (0x25aa bytes) 000096 1761236024.319057 mapwrite 0x025f55 thru 0x02e488 (0x8534 bytes) 000099 1761236024.350044 fallocate 0x03919c thru 0x03ffff (0x6e64 bytes) 000101 1761236024.358786 mapread 0x020bb9 thru 0x02a87e (0x9cc6 bytes) 000102 1761236024.367033 mapread 0x03b84f thru 0x03ffff (0x47b1 bytes) 000103 1761236024.367943 read 0x005220 thru 0x012bfa (0xd9db bytes) 000104 1761236024.368571 mapwrite 0x0362f3 thru 0x03fdc9 (0x9ad7 bytes) 000105 1761236024.384554 read 0x013d73 thru 0x020e56 (0xd0e4 bytes) 000106 1761236024.386262 write 0x034262 thru 0x039e87 (0x5c26 bytes) 000107 1761236024.387428 mapwrite 0x0206ab thru 0x0291b2 (0x8b08 bytes) 000108 1761236024.410718 punch from 0x00b8fe to 0x011596 (0x5c99 bytes) 000109 1761236024.417683 mapwrite 0x039e2e thru 0x03ffff (0x61d2 bytes) 000110 1761236024.434067 trunc from 0x01605e to 0x01c2e2 (0x6285 bytes) 000111 1761236024.438911 trunc from 0x01a100 to 0x020e9f (0x6da0 bytes) 000112 1761236024.444574 punch from 0x02fd6c to 0x03d31e (0xd5b3 bytes) 000113 1761236024.451076 trunc from 0x012069 to 0x013975 (0x190d bytes) 000114 1761236024.456543 trunc from 0x040000 to 0x037e4b (0x81b4 bytes) 000115 1761236024.476174 fallocate 0x008298 thru 0x013acc (0xb835 bytes) 000116 1761236024.488402 MIRROR_EXTEND to 2 mirrors 000117 1761236025.141730 read 0x0187d1 thru 0x0268df (0xe10f bytes) 000120 1761236025.178875 punch from 0x033f21 to 0x037e4b (0x3f2b bytes) 000121 1761236025.226804 mapread 0x00d9a6 thru 0x00f81e (0x1e79 bytes) 000122 1761236025.251086 read 0x027466 thru 0x029959 (0x24f4 bytes) 000124 1761236025.261265 trunc from 0x037e4c to 0x03f71f (0x78d4 bytes) 000125 1761236025.309913 read 0x01199b thru 0x014b9e (0x3204 bytes) 000126 1761236025.311197 trunc from 0x03f720 to 0x02d54c (0x121d3 bytes) 000130 1761236025.346142 trunc from 0x02d54d to 0x02c18c (0x13c0 bytes) 000131 1761236025.367508 trunc from 0x01198a to 0x01c427 (0xaa9e bytes) 000132 1761236025.385675 MIRROR_RESYNC 2 mirrors 000133 1761236026.065385 trunc from 0x02c18d to 0x00018f (0x2bffd bytes) 000134 1761236026.129955 trunc from 0x000190 to 0x03d48a (0x3d2fb bytes) 000135 1761236026.147446 trunc from 0x01a22c to 0x01cdfd (0x2bd2 bytes) 000136 1761236026.155066 trunc from 0x034f8d to 0x03d48a (0x84fe bytes) 000137 1761236026.170585 trunc from 0x00d8b3 to 0x013739 (0x5e87 bytes) 000138 1761236026.177132 trunc from 0x03d48b to 0x00ee4a (0x2e640 bytes) 000139 1761236026.192742 trunc from 0x00ee4b to 0x00b4a9 (0x39a1 bytes) 000140 1761236026.207178 mapread 0x003b20 thru 0x00b4a9 (0x798a bytes) 000141 1761236026.225016 punch from 0x006b4b to 0x00b4a9 (0x495f bytes) 000142 1761236026.234164 MIRROR_EXTEND to 3 mirrors 000143 1761236026.890540 read 0x0075b9 thru 0x0076c0 (0x108 bytes) 000144 1761236026.913615 punch from 0x006d15 to 0x00b4a9 (0x4795 bytes) 000145 1761236026.960420 MIRROR_EXTEND to 4 mirrors 000146 1761236027.666698 trunc from 0x003dce to 0x00a7d4 (0x6a07 bytes) 000149 1761236027.705258 mapwrite 0x022d68 thru 0x02fe02 (0xd09b bytes) 000150 1761236027.778846 MIRROR_EXTEND to 5 mirrors 000151 1761236028.501243 write 0x02ef29 thru 0x036b1b (0x7bf3 bytes) 000153 1761236028.552052 read 0x01f1ad thru 0x0260b8 (0x6f0c bytes) 000154 1761236028.565011 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000155 1761236028.777389 fallocate 0x01007c thru 0x015f55 (0x5eda bytes) 000156 1761236028.784078 punch from 0x019a2d to 0x020142 (0x6716 bytes) 000157 1761236028.792102 fallocate 0x00f0f7 thru 0x00fc99 (0xba3 bytes) 000158 1761236028.807215 trunc from 0x034b93 to 0x036b1b (0x1f89 bytes) 000159 1761236028.815911 trunc from 0x03664c to 0x036b1b (0x4d0 bytes) 000160 1761236028.823607 write 0x02e61e thru 0x035624 (0x7007 bytes) 000161 1761236028.836926 mapwrite 0x012652 thru 0x017893 (0x5242 bytes) 000163 1761236028.864188 trunc from 0x00b427 to 0x01a8d8 (0xf4b2 bytes) 000164 1761236028.871067 mapread 0x0189e2 thru 0x0289bf (0xffde bytes) 000165 1761236028.893176 trunc from 0x01089f to 0x016fd0 (0x6732 bytes) 000167 1761236028.900243 mapread 0x02bda2 thru 0x033700 (0x795f bytes) 000168 1761236028.906606 trunc from 0x036b1c to 0x012f11 (0x23c0a bytes) 000169 1761236028.929637 MIRROR_RESYNC 4 mirrors 000170 1761236029.702384 fallocate 0x022bd4 thru 0x0274b0 (0x48dd bytes) 000171 1761236029.750537 write 0x006d0d thru 0x010bb0 (0x9ea4 bytes) 000172 1761236029.762094 trunc from 0x00db2d to 0x00e2ea (0x7be bytes) 000174 1761236029.785402 trunc from 0x0060b4 to 0x0118c4 (0xb811 bytes) 000175 1761236029.838569 trunc from 0x012f12 to 0x03bd51 (0x28e40 bytes) 000176 1761236029.862721 trunc from 0x03bd52 to 0x022596 (0x197bb bytes) 000177 1761236029.894763 mapread 0x003bc0 thru 0x00eda2 (0xb1e3 bytes) 000178 1761236029.915844 write 0x036f3e thru 0x03ffff (0x90c2 bytes) 000179 1761236029.917297 mapread 0x00f7f9 thru 0x016b5a (0x7362 bytes) 000180 1761236029.925121 trunc from 0x034d1d to 0x03d91a (0x8bfe bytes) 000181 1761236029.941084 read 0x027578 thru 0x03435d (0xcde6 bytes) 000182 1761236029.955710 mapwrite 0x0301b4 thru 0x030485 (0x2d2 bytes) 000183 1761236029.970112 fallocate 0x00c9e5 thru 0x018b27 (0xc143 bytes) 000184 1761236029.974236 trunc from 0x040000 to 0x002c39 (0x3d3c6 bytes) 000185 1761236029.992902 write 0x01cbf7 thru 0x0284d6 (0xb8e0 bytes) 000186 1761236029.996531 read 0x00c351 thru 0x013b17 (0x77c7 bytes) 000187 1761236030.005498 mapread 0x00c9fe thru 0x01aef3 (0xe4f6 bytes) 000188 1761236030.006506 fallocate 0x029092 thru 0x02e413 (0x5382 bytes) 000189 1761236030.012166 fallocate 0x005432 thru 0x00ae14 (0x59e3 bytes) 000190 1761236030.018415 trunc from 0x02e414 to 0x014f34 (0x194df bytes) 000191 1761236030.033789 mapwrite 0x015779 thru 0x02098f (0xb217 bytes) 000192 1761236030.144351 write 0x02464d thru 0x0252db (0xc8f bytes) 000193 1761236030.145727 punch from 0x007028 to 0x009cb4 (0x2c8d bytes) 000194 1761236030.156151 mapread 0x0092bb thru 0x010cda (0x7a20 bytes) 000195 1761236030.164486 mapwrite 0x00c7a1 thru 0x01a20b (0xda6b bytes) 000196 1761236030.195846 fallocate 0x005174 thru 0x013747 (0xe5d4 bytes) 000197 1761236030.202679 read 0x015e4b thru 0x020d67 (0xaf1d bytes) 000198 1761236030.203181 fallocate 0x02bd48 thru 0x02f8e1 (0x3b9a bytes) 000199 1761236030.210130 read 0x013e7b thru 0x020f03 (0xd089 bytes) 000200 1761236030.210875 mapwrite 0x01ad28 thru 0x025228 (0xa501 bytes) 000201 1761236030.235051 MIRROR_RESYNC 4 mirrors 000202 1761236030.986301 trunc from 0x02f8e2 to 0x014605 (0x1b2dc bytes) 000203 1761236031.053415 mapwrite 0x0223b5 thru 0x02a8af (0x84fb bytes) 000204 1761236031.110195 punch from 0x004673 to 0x00b768 (0x70f6 bytes) 000205 1761236031.115541 punch from 0x00bd53 to 0x012990 (0x6c3e bytes) 000206 1761236031.121251 mapwrite 0x01f497 thru 0x02b462 (0xbfcc bytes) 000207 1761236031.221325 punch from 0x0167d3 to 0x01ab34 (0x4362 bytes) 000208 1761236031.241296 MIRROR_EXTEND to 5 mirrors 000209 1761236032.181816 mapread 0x025aa8 thru 0x02b462 (0x59bb bytes) 000211 1761236032.273282 trunc from 0x00e812 to 0x0171dd (0x89cc bytes) 000212 1761236032.357832 mapread 0x004f03 thru 0x012d4f (0xde4d bytes) 000213 1761236032.374790 punch from 0x000a9b to 0x00b043 (0xa5a9 bytes) 000214 1761236032.383803 write 0x0139ad thru 0x0234ef (0xfb43 bytes) 000215 1761236032.401399 trunc from 0x027d28 to 0x02b462 (0x373b bytes) 000216 1761236032.417673 write 0x0185dd thru 0x01bc54 (0x3678 bytes) 000218 1761236032.419696 mapread 0x015f0f thru 0x020daa (0xae9c bytes) 000219 1761236032.420642 trunc from 0x02b463 to 0x01b45f (0x10003 bytes) 000220 1761236032.466194 punch from 0x004e12 to 0x008943 (0x3b32 bytes) 000221 1761236032.471858 trunc from 0x01b460 to 0x03d9d9 (0x2257a bytes) 000223 1761236032.494682 MIRROR_EXTEND to 6 mirrors 000224 1761236033.299857 trunc from 0x017326 to 0x021033 (0x9d0e bytes) 000225 1761236033.334202 read 0x010742 thru 0x01de59 (0xd718 bytes) 000226 1761236033.343544 read 0x012780 thru 0x01d4d2 (0xad53 bytes) 000227 1761236033.344127 trunc from 0x01732e to 0x017b0f (0x7e2 bytes) 000228 1761236033.349951 trunc from 0x00112e to 0x007faa (0x6e7d bytes) 000229 1761236033.354412 mapread 0x022236 thru 0x023013 (0xdde bytes) 000230 1761236033.365054 punch from 0x003844 to 0x012e98 (0xf655 bytes) 000231 1761236033.369513 mapread 0x033df0 thru 0x036290 (0x24a1 bytes) 000232 1761236033.370335 read 0x032352 thru 0x034778 (0x2427 bytes) 000233 1761236033.374666 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000234 1761236033.569529 mapwrite 0x002574 thru 0x00be33 (0x98c0 bytes) 000236 1761236033.602687 trunc from 0x03d9da to 0x0029fe (0x3afdb bytes) 000237 1761236033.628588 fallocate 0x02af68 thru 0x030ea7 (0x5f40 bytes) 000238 1761236033.635790 mapwrite 0x008ecf thru 0x00f5fc (0x672e bytes) 000239 1761236033.677819 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000240 1761236033.927081 trunc from 0x00f5fd to 0x02aadf (0x1b4e3 bytes) 000241 1761236033.983208 mapwrite 0x022c4a thru 0x02892c (0x5ce3 bytes) 000242 1761236034.027944 trunc from 0x00354d to 0x00ddfd (0xa8b1 bytes) 000243 1761236034.036828 read 0x006316 thru 0x00a858 (0x4543 bytes) 000244 1761236034.046575 trunc from 0x0208d5 to 0x02aadf (0xa20b bytes) 000245 1761236034.059170 mapread 0x0180b6 thru 0x021472 (0x93bd bytes) 000246 1761236034.084931 punch from 0x00886d to 0x017e2f (0xf5c3 bytes) 000247 1761236034.089809 trunc from 0x02aae0 to 0x006d1c (0x23dc3 bytes) 000248 1761236034.113993 trunc from 0x0054ca to 0x006d1c (0x1853 bytes) 000249 1761236034.118734 trunc from 0x001e0f to 0x006d1c (0x4f0e bytes) 000250 1761236034.127128 mapread 0x0068d5 thru 0x006d1c (0x448 bytes) 000251 1761236034.143913 trunc from 0x000f14 to 0x006d1c (0x5e09 bytes) 000252 1761236034.161691 mapread 0x0046ce thru 0x006d1c (0x264f bytes) 000253 1761236034.187025 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000254 1761236034.411182 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000255 1761236034.711746 write 0x01a5ce thru 0x01c227 (0x1c5a bytes) 000257 1761236034.723786 read 0x01ae43 thru 0x01c227 (0x13e5 bytes) 000258 1761236034.724874 write 0x01a0b2 thru 0x028ffe (0xef4d bytes) 000259 1761236034.728595 fallocate 0x01cefc thru 0x02c0e5 (0xf1ea bytes) 000260 1761236034.744666 read 0x0139f5 thru 0x01dbd7 (0xa1e3 bytes) 000261 1761236034.770702 read 0x004c50 thru 0x00815a (0x350b bytes) 000262 1761236034.771959 punch from 0x01c84d to 0x028ffe (0xc7b2 bytes) 000263 1761236034.794411 trunc from 0x028fff to 0x01290b (0x166f3 bytes) 000264 1761236034.824035 write 0x02ade3 thru 0x02b6bd (0x8db bytes) 000265 1761236034.826332 mapread 0x006419 thru 0x007faf (0x1b97 bytes) 000266 1761236034.829571 trunc from 0x017f84 to 0x0189cf (0xa4c bytes) 000268 1761236034.836812 read 0x00feb8 thru 0x0181b0 (0x82f9 bytes) 000269 1761236034.856627 MIRROR_EXTEND to 3 mirrors 000270 1761236035.468441 punch from 0x01997a to 0x02428a (0xa911 bytes) 000271 1761236035.538746 write 0x01817d thru 0x02160a (0x948e bytes) 000272 1761236035.567129 mapread 0x000ec3 thru 0x008da6 (0x7ee4 bytes) 000273 1761236035.584674 mapwrite 0x039b09 thru 0x03ffff (0x64f7 bytes) 000274 1761236035.642007 fallocate 0x026ed1 thru 0x035e23 (0xef53 bytes) 000275 1761236035.649016 mapwrite 0x001b2b thru 0x011705 (0xfbdb bytes) 000276 1761236035.691972 punch from 0x012660 to 0x01a481 (0x7e22 bytes) 000277 1761236035.707923 mapread 0x002033 thru 0x00a706 (0x86d4 bytes) 000278 1761236035.715162 mapwrite 0x0224c6 thru 0x030671 (0xe1ac bytes) 000279 1761236035.798236 fallocate 0x00c9d8 thru 0x017f6c (0xb595 bytes) 000280 1761236035.817154 mapwrite 0x02114f thru 0x0231eb (0x209d bytes) 000281 1761236035.856780 read 0x0250da thru 0x02e98a (0x98b1 bytes) 000283 1761236035.859219 punch from 0x030a98 to 0x033db3 (0x331c bytes) 000284 1761236035.868606 read 0x00a964 thru 0x014fa3 (0xa640 bytes) 000285 1761236035.890875 fallocate 0x010351 thru 0x01aa58 (0xa708 bytes) 000286 1761236035.902357 mapwrite 0x002920 thru 0x00ff39 (0xd61a bytes) 000287 1761236035.963218 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000288 1761236036.252219 mapwrite 0x03bd92 thru 0x03ffff (0x426e bytes) 000289 1761236036.291138 MIRROR_RESYNC 2 mirrors 000290 1761236037.119649 fallocate 0x01b38c thru 0x01ceeb (0x1b60 bytes) 000292 1761236037.173348 punch from 0x00ea96 to 0x019a4d (0xafb8 bytes) 000293 1761236037.181633 mapread 0x00c5d8 thru 0x012ab1 (0x64da bytes) 000294 1761236037.208841 MIRROR_RESYNC 2 mirrors 000295 1761236037.893637 punch from 0x02dcf1 to 0x03b078 (0xd388 bytes) 000296 1761236037.953832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000297 1761236038.227082 trunc from 0x040000 to 0x006cff (0x39300 bytes) 000298 1761236038.280913 mapwrite 0x03034d thru 0x030ecb (0xb7f bytes) 000299 1761236038.386191 mapread 0x02345c thru 0x026cf5 (0x389a bytes) 000301 1761236038.413817 fallocate 0x034ab5 thru 0x03f713 (0xac5f bytes) 000302 1761236038.434309 write 0x011add thru 0x01dc5e (0xc182 bytes) 000303 1761236038.463031 mapwrite 0x010d72 thru 0x020306 (0xf595 bytes) 000304 1761236038.503445 trunc from 0x025c8b to 0x02c913 (0x6c89 bytes) 000305 1761236038.511920 trunc from 0x030ecc to 0x024810 (0xc6bb bytes) 000306 1761236038.537584 trunc from 0x00282f to 0x011677 (0xee49 bytes) 000307 1761236038.553230 read 0x01213a thru 0x018c66 (0x6b2d bytes) 000309 1761236038.558362 fallocate 0x017cc5 thru 0x026d5a (0xf096 bytes) 000310 1761236038.580774 MIRROR_EXTEND to 2 mirrors 000311 1761236039.399926 trunc from 0x00a0cc to 0x00c2b1 (0x21e6 bytes) 000312 1761236039.447047 read 0x01f718 thru 0x026d5a (0x7643 bytes) 000313 1761236039.475170 MIRROR_EXTEND to 3 mirrors 000314 1761236040.344109 trunc from 0x00c501 to 0x00ea25 (0x2525 bytes) 000315 1761236040.390594 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000316 1761236040.600544 write 0x0220af thru 0x02615d (0x40af bytes) 000317 1761236040.619908 mapwrite 0x01d511 thru 0x026b7f (0x966f bytes) 000318 1761236040.651243 read 0x026682 thru 0x026d5a (0x6d9 bytes) 000319 1761236040.651849 write 0x0314ab thru 0x03f554 (0xe0aa bytes) 000320 1761236040.653537 read 0x03c2b8 thru 0x03f554 (0x329d bytes) 000321 1761236040.654068 trunc from 0x03f555 to 0x037454 (0x8100 bytes) 000322 1761236040.681710 mapwrite 0x03366d thru 0x03c8c5 (0x9259 bytes) 000323 1761236040.720896 trunc from 0x039498 to 0x03c8c5 (0x342e bytes) 000324 1761236040.731156 mapread 0x0130f8 thru 0x01a606 (0x750f bytes) 000325 1761236040.732084 punch from 0x01eeac to 0x021f92 (0x30e7 bytes) 000326 1761236040.745137 MIRROR_RESYNC 2 mirrors 000327 1761236041.804273 write 0x00d116 thru 0x010f13 (0x3dfe bytes) 000330 1761236041.847937 MIRROR_EXTEND to 3 mirrors 000331 1761236042.584145 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000332 1761236042.829745 fallocate 0x004345 thru 0x00ffce (0xbc8a bytes) 000333 1761236042.847895 write 0x02cf95 thru 0x039eed (0xcf59 bytes) 000334 1761236042.862556 mapwrite 0x02c9e2 thru 0x035639 (0x8c58 bytes) 000335 1761236042.884434 mapwrite 0x037ccb thru 0x03ffff (0x8335 bytes) 000336 1761236042.932670 punch from 0x00b9c7 to 0x01311e (0x7758 bytes) 000338 1761236042.940028 fallocate 0x02097e thru 0x02d97c (0xcfff bytes) 000339 1761236042.952556 read 0x02a1c2 thru 0x033cc0 (0x9aff bytes) 000340 1761236042.964219 mapwrite 0x022f0c thru 0x02e6dd (0xb7d2 bytes) 000341 1761236043.016597 trunc from 0x0252ee to 0x028852 (0x3565 bytes) 000342 1761236043.023259 read 0x0314e9 thru 0x0318a6 (0x3be bytes) 000343 1761236043.034989 MIRROR_EXTEND to 3 mirrors 000344 1761236044.249245 trunc from 0x02bf18 to 0x03b014 (0xf0fd bytes) 000345 1761236044.322376 fallocate 0x027fc4 thru 0x02b71b (0x3758 bytes) 000346 1761236044.327722 trunc from 0x040000 to 0x03bfac (0x4053 bytes) 000347 1761236044.373385 fallocate 0x032ed7 thru 0x03f62e (0xc758 bytes) 000348 1761236044.392245 trunc from 0x03f62f to 0x02988a (0x15da4 bytes) 000349 1761236044.423567 fallocate 0x00e14d thru 0x0168f7 (0x87ab bytes) 000350 1761236044.439233 punch from 0x01619d to 0x0201b4 (0xa018 bytes) 000352 1761236044.444364 write 0x034b42 thru 0x03ffff (0xb4be bytes) 000353 1761236044.446442 mapwrite 0x02f3ec thru 0x03f2ee (0xff03 bytes) 000354 1761236044.538371 trunc from 0x01a066 to 0x01f20e (0x51a9 bytes) 000355 1761236044.560749 write 0x00f056 thru 0x019795 (0xa740 bytes) 000356 1761236044.604083 fallocate 0x0027ad thru 0x008067 (0x58bb bytes) 000357 1761236044.616701 fallocate 0x0193c8 thru 0x01d4de (0x4117 bytes) 000358 1761236044.635098 write 0x020b64 thru 0x024a35 (0x3ed2 bytes) 000359 1761236044.661571 read 0x03f2b2 thru 0x03ffff (0xd4e bytes) 000360 1761236044.662156 write 0x01db68 thru 0x02363f (0x5ad8 bytes) 000361 1761236044.663219 trunc from 0x00a90b to 0x00c3ae (0x1aa4 bytes) 000362 1761236044.698849 punch from 0x00a295 to 0x00c52c (0x2298 bytes) 000363 1761236044.713493 trunc from 0x040000 to 0x018631 (0x279ce bytes) 000365 1761236044.763185 punch from 0x00ec91 to 0x010334 (0x16a4 bytes) 000366 1761236044.806190 fallocate 0x01d56b thru 0x021934 (0x43ca bytes) 000367 1761236044.816023 write 0x017d5a thru 0x01d67d (0x5924 bytes) 000368 1761236044.845022 trunc from 0x018bc2 to 0x021934 (0x8d73 bytes) 000369 1761236044.897915 mapwrite 0x01df46 thru 0x0292d7 (0xb392 bytes) 000370 1761236045.092974 trunc from 0x0025e3 to 0x00c4d2 (0x9ef0 bytes) 000371 1761236045.128195 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000372 1761236045.599079 trunc from 0x019bdb to 0x027b29 (0xdf4f bytes) 000373 1761236045.615637 write 0x03f545 thru 0x03ffff (0xabb bytes) 000374 1761236045.624085 trunc from 0x040000 to 0x023966 (0x1c699 bytes) 000375 1761236045.657017 write 0x001bae thru 0x00ac49 (0x909c bytes) 000376 1761236045.677672 mapwrite 0x034ddf thru 0x03b674 (0x6896 bytes) 000377 1761236045.755689 mapwrite 0x029e29 thru 0x03572e (0xb906 bytes) 000378 1761236045.794678 read 0x034a73 thru 0x03b674 (0x6c02 bytes) 000379 1761236045.797030 mapwrite 0x03fb16 thru 0x03ffff (0x4ea bytes) 000380 1761236045.911983 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000381 1761236046.069566 fallocate 0x031dcd thru 0x03ba63 (0x9c97 bytes) 000382 1761236046.079210 read 0x02ba73 thru 0x039856 (0xdde4 bytes) 000383 1761236046.102254 MIRROR_EXTEND to 2 mirrors 000384 1761236047.448783 trunc from 0x040000 to 0x023356 (0x1cca9 bytes) 000385 1761236047.555916 trunc from 0x00fe57 to 0x017fc3 (0x816d bytes) 000386 1761236047.564872 trunc from 0x00f147 to 0x0112c3 (0x217d bytes) 000387 1761236047.573503 fallocate 0x02c434 thru 0x0322c0 (0x5e8d bytes) 000388 1761236047.582255 write 0x00b91c thru 0x013962 (0x8047 bytes) 000389 1761236047.596772 fallocate 0x01b66c thru 0x01bd05 (0x69a bytes) 000390 1761236047.626510 MIRROR_RESYNC 2 mirrors 000391 1761236048.572407 trunc from 0x014c06 to 0x020d30 (0xc12b bytes) 000392 1761236048.621562 write 0x02b41f thru 0x03780d (0xc3ef bytes) 000393 1761236048.639673 trunc from 0x035772 to 0x03780d (0x209c bytes) 000394 1761236048.654754 mapread 0x00efc6 thru 0x01d6b8 (0xe6f3 bytes) 000395 1761236048.702539 read 0x014d18 thru 0x01a9e3 (0x5ccc bytes) 000396 1761236048.703118 write 0x03c3c9 thru 0x03ffff (0x3c37 bytes) 000397 1761236048.707418 write 0x03c42c thru 0x03ee2d (0x2a02 bytes) 000398 1761236048.711999 mapread 0x011ce8 thru 0x0216ea (0xfa03 bytes) 000400 1761236048.732332 punch from 0x02c6c3 to 0x039bde (0xd51c bytes) 000401 1761236048.743778 fallocate 0x039e9d thru 0x03ffff (0x6163 bytes) 000402 1761236048.762234 mapwrite 0x012f6f thru 0x019e3e (0x6ed0 bytes) 000403 1761236048.830556 punch from 0x01f62b to 0x0207a5 (0x117b bytes) 000404 1761236048.844332 trunc from 0x040000 to 0x023291 (0x1cd6e bytes) 000405 1761236048.914845 mapwrite 0x013f87 thru 0x016976 (0x29f0 bytes) 000406 1761236048.950344 punch from 0x00bdf8 to 0x01743c (0xb645 bytes) 000407 1761236048.964419 punch from 0x00ebec to 0x01ced8 (0xe2ed bytes) 000408 1761236048.974915 trunc from 0x023292 to 0x006119 (0x1d178 bytes) 000409 1761236049.017196 write 0x026b8e thru 0x03573c (0xebaf bytes) 000410 1761236049.032650 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000411 1761236049.297952 trunc from 0x00ffab to 0x01c9e3 (0xca39 bytes) 000412 1761236049.312255 trunc from 0x02750d to 0x030534 (0x9028 bytes) 000413 1761236049.329321 trunc from 0x03573d to 0x0305cf (0x516d bytes) 000416 1761236049.412670 mapwrite 0x03af43 thru 0x03ffff (0x50bd bytes) 000417 1761236049.502582 read 0x02da29 thru 0x039d6b (0xc343 bytes) 000419 1761236049.518724 punch from 0x025fa8 to 0x02d225 (0x727e bytes) 000420 1761236049.534481 write 0x0161f3 thru 0x0254b4 (0xf2c2 bytes) 000421 1761236049.581400 punch from 0x015650 to 0x01b7cd (0x617e bytes) 000422 1761236049.623278 write 0x028005 thru 0x032630 (0xa62c bytes) 000423 1761236049.656798 fallocate 0x02f702 thru 0x030172 (0xa71 bytes) 000424 1761236049.670891 write 0x021551 thru 0x02319c (0x1c4c bytes) 000425 1761236049.684297 trunc from 0x040000 to 0x00a57a (0x35a85 bytes) 000426 1761236049.736975 write 0x0359a6 thru 0x039687 (0x3ce2 bytes) 000427 1761236049.747694 read 0x038360 thru 0x039687 (0x1328 bytes) 000428 1761236049.748596 mapwrite 0x01fbc6 thru 0x02819d (0x85d8 bytes) 000429 1761236049.841820 trunc from 0x00c2bd to 0x0105fc (0x4340 bytes) 000430 1761236049.850949 fallocate 0x03e858 thru 0x03ffff (0x17a8 bytes) 000431 1761236049.866650 mapwrite 0x0180a3 thru 0x01da97 (0x59f5 bytes) 000432 1761236049.922943 read 0x01d9d0 thru 0x02aab4 (0xd0e5 bytes) 000435 1761236049.930103 write 0x00bcfa thru 0x0196a0 (0xd9a7 bytes) 000436 1761236049.931643 punch from 0x0292cb to 0x02d2d1 (0x4007 bytes) 000437 1761236049.952725 MIRROR_EXTEND to 2 mirrors 000438 1761236051.395686 mapread 0x0331de thru 0x038ca4 (0x5ac7 bytes) 000439 1761236051.468400 read 0x001ddf thru 0x004c76 (0x2e98 bytes) 000441 1761236051.506174 write 0x01d2f8 thru 0x029eb9 (0xcbc2 bytes) 000442 1761236051.586021 mapread 0x007956 thru 0x008999 (0x1044 bytes) 000443 1761236051.611775 punch from 0x003446 to 0x00fcdf (0xc89a bytes) 000444 1761236051.623494 write 0x00313f thru 0x009385 (0x6247 bytes) 000446 1761236051.641143 MIRROR_RESYNC 2 mirrors 000447 1761236052.836246 fallocate 0x00b9ad thru 0x01555d (0x9bb1 bytes) 000448 1761236052.865331 read 0x036243 thru 0x039687 (0x3445 bytes) 000449 1761236052.873732 fallocate 0x01c5f7 thru 0x021e9c (0x58a6 bytes) 000450 1761236052.888303 mapread 0x034df1 thru 0x039687 (0x4897 bytes) 000451 1761236052.914858 trunc from 0x01692b to 0x01a2ac (0x3982 bytes) 000452 1761236052.924420 mapwrite 0x03bcb3 thru 0x03f3f8 (0x3746 bytes) 000453 1761236053.043330 mapwrite 0x001be4 thru 0x00ed95 (0xd1b2 bytes) 000454 1761236053.107814 trunc from 0x03dc46 to 0x03f3f8 (0x17b3 bytes) 000455 1761236053.122805 punch from 0x02b369 to 0x03593e (0xa5d6 bytes) 000456 1761236053.136495 mapread 0x033f80 thru 0x03bb2a (0x7bab bytes) 000457 1761236053.165274 fallocate 0x00f4b8 thru 0x016f1f (0x7a68 bytes) 000458 1761236053.180188 read 0x0135a5 thru 0x01ad29 (0x7785 bytes) 000459 1761236053.195298 trunc from 0x0012f1 to 0x00e2f0 (0xd000 bytes) 000460 1761236053.203019 read 0x03dd12 thru 0x03f3f8 (0x16e7 bytes) 000461 1761236053.203700 trunc from 0x03f3f9 to 0x02262f (0x1cdc9 bytes) 000463 1761236053.249018 mapread 0x018fc3 thru 0x01a244 (0x1282 bytes) 000464 1761236053.249949 fallocate 0x004e66 thru 0x00ab94 (0x5d2f bytes) 000465 1761236053.258185 mapwrite 0x00639d thru 0x007235 (0xe99 bytes) 000466 1761236053.296138 MIRROR_RESYNC 2 mirrors 000467 1761236054.343650 trunc from 0x0145d9 to 0x01e3c3 (0x9deb bytes) 000468 1761236054.410900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000469 1761236054.628796 trunc from 0x022630 to 0x003974 (0x1ecbb bytes) 000470 1761236054.664393 punch from 0x001550 to 0x002a9b (0x154c bytes) 000471 1761236054.686625 trunc from 0x000c55 to 0x003974 (0x2d20 bytes) 000472 1761236054.710604 MIRROR_EXTEND to 2 mirrors 000473 1761236055.476336 MIRROR_RESYNC 2 mirrors 000474 1761236055.701480 mapwrite 0x0195ff thru 0x023e58 (0xa85a bytes) 000475 1761236055.860840 mapwrite 0x0083f9 thru 0x015781 (0xd389 bytes) 000476 1761236055.941616 mapwrite 0x000f26 thru 0x00f036 (0xe111 bytes) 000477 1761236056.000487 mapread 0x01ef02 thru 0x023e58 (0x4f57 bytes) 000478 1761236056.004500 mapwrite 0x02c1ff thru 0x03124a (0x504c bytes) 000479 1761236056.087281 trunc from 0x01ddbb to 0x020ae0 (0x2d26 bytes) 000481 1761236056.094198 trunc from 0x03124b to 0x01da7f (0x137cb bytes) 000482 1761236056.111292 mapread 0x00bd0c thru 0x00f9ae (0x3ca3 bytes) 000483 1761236056.112549 mapwrite 0x00ffad thru 0x018b4e (0x8ba2 bytes) 000486 1761236056.132877 trunc from 0x01da80 to 0x01e5c1 (0xb42 bytes) 000487 1761236056.150897 trunc from 0x01e5c2 to 0x010eb5 (0xd70c bytes) 000488 1761236056.173626 mapread 0x0017b0 thru 0x007edb (0x672c bytes) 000489 1761236056.174611 read 0x00e4e5 thru 0x010eb5 (0x29d1 bytes) 000490 1761236056.175158 mapread 0x000d59 thru 0x003a77 (0x2d1f bytes) 000491 1761236056.183700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000492 1761236056.358982 trunc from 0x002dcd to 0x002fe9 (0x21d bytes) 000493 1761236056.367738 fallocate 0x00b23a thru 0x0123fb (0x71c2 bytes) 000494 1761236056.374581 write 0x00ecc4 thru 0x01e746 (0xfa83 bytes) 000495 1761236056.404252 mapread 0x009f3e thru 0x00efd2 (0x5095 bytes) 000496 1761236056.419592 mapwrite 0x02edac thru 0x0320e5 (0x333a bytes) 000497 1761236056.476806 fallocate 0x036a70 thru 0x03c430 (0x59c1 bytes) 000498 1761236056.488153 MIRROR_EXTEND to 2 mirrors 000499 1761236057.214974 mapread 0x0059ef thru 0x00b077 (0x5689 bytes) 000500 1761236057.278774 trunc from 0x01577b to 0x023a9a (0xe320 bytes) 000501 1761236057.319180 mapwrite 0x037620 thru 0x03ffff (0x89e0 bytes) 000502 1761236057.429208 punch from 0x03b25e to 0x03e2bb (0x305e bytes) 000503 1761236057.435894 trunc from 0x00b0b4 to 0x00bfd2 (0xf1f bytes) 000504 1761236057.451628 MIRROR_RESYNC 2 mirrors 000505 1761236058.205571 mapwrite 0x00e78a thru 0x0189d4 (0xa24b bytes) 000506 1761236058.330588 mapread 0x02e6e2 thru 0x02ffed (0x190c bytes) 000507 1761236058.346770 trunc from 0x02f79e to 0x03a2e2 (0xab45 bytes) 000508 1761236058.355287 write 0x012099 thru 0x019a1b (0x7983 bytes) 000510 1761236058.356783 punch from 0x00818d to 0x013247 (0xb0bb bytes) 000511 1761236058.376005 trunc from 0x01664b to 0x0202f1 (0x9ca7 bytes) 000513 1761236058.383360 punch from 0x02bfef to 0x037fde (0xbff0 bytes) 000514 1761236058.389694 read 0x0130e6 thru 0x021b0d (0xea28 bytes) 000515 1761236058.399222 trunc from 0x040000 to 0x00d5d4 (0x32a2b bytes) 000517 1761236058.503539 mapread 0x00a913 thru 0x00d5d4 (0x2cc2 bytes) 000518 1761236058.526134 write 0x000f1a thru 0x005bf1 (0x4cd8 bytes) 000519 1761236058.535024 write 0x014733 thru 0x0153db (0xca9 bytes) 000520 1761236058.539830 mapwrite 0x016700 thru 0x01d9c3 (0x72c4 bytes) 000521 1761236058.627220 trunc from 0x01d9c4 to 0x009324 (0x1469f bytes) 000523 1761236058.678354 mapwrite 0x010efe thru 0x01beac (0xafaf bytes) 000525 1761236058.792132 fallocate 0x00f965 thru 0x0107ac (0xe48 bytes) 000526 1761236058.804079 mapwrite 0x032d70 thru 0x033d97 (0x1028 bytes) 000527 1761236058.862759 mapread 0x000560 thru 0x00ceb3 (0xc954 bytes) 000528 1761236058.865825 trunc from 0x033d98 to 0x007d96 (0x2c001 bytes) 000529 1761236058.905035 write 0x0291a1 thru 0x032518 (0x9378 bytes) 000531 1761236058.906870 punch from 0x018e0c to 0x020603 (0x77f8 bytes) 000532 1761236058.914156 mapread 0x005d3b thru 0x008cf4 (0x2fba bytes) 000534 1761236058.933363 mapwrite 0x036ba5 thru 0x03cb57 (0x5fb3 bytes) 000535 1761236058.993184 trunc from 0x03cb58 to 0x02543f (0x17718 bytes) 000536 1761236059.029878 fallocate 0x012206 thru 0x01ce31 (0xac2c bytes) 000537 1761236059.048332 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000538 1761236059.263724 trunc from 0x01439a to 0x01fb5d (0xb7c4 bytes) 000539 1761236059.270089 write 0x037c31 thru 0x03ffff (0x83cf bytes) 000540 1761236059.281885 trunc from 0x02c67d to 0x02f861 (0x31e5 bytes) 000541 1761236059.289578 fallocate 0x03310a thru 0x035aa1 (0x2998 bytes) 000543 1761236059.299232 read 0x03f2a5 thru 0x03f6a2 (0x3fe bytes) 000544 1761236059.299751 write 0x031434 thru 0x031e51 (0xa1e bytes) 000545 1761236059.306391 trunc from 0x040000 to 0x02f3fc (0x10c03 bytes) 000547 1761236059.329028 trunc from 0x005311 to 0x00b454 (0x6144 bytes) 000548 1761236059.337442 trunc from 0x02f3fd to 0x02a21c (0x51e0 bytes) 000549 1761236059.357720 mapread 0x009406 thru 0x0106ea (0x72e5 bytes) 000550 1761236059.388588 read 0x014ac0 thru 0x01d57a (0x8abb bytes) 000552 1761236059.413080 trunc from 0x024bb6 to 0x02a21c (0x5667 bytes) 000553 1761236059.428307 mapwrite 0x027de5 thru 0x030b9c (0x8db8 bytes) 000554 1761236059.560604 fallocate 0x007de3 thru 0x0168f6 (0xeb14 bytes) 000555 1761236059.570583 fallocate 0x023721 thru 0x02aa3e (0x731e bytes) 000558 1761236059.585760 mapread 0x024c6c thru 0x02d3ff (0x8794 bytes) 000559 1761236059.627988 fallocate 0x0016cb thru 0x01103f (0xf975 bytes) 000560 1761236059.646316 mapwrite 0x001861 thru 0x00f09a (0xd83a bytes) 000561 1761236059.698275 mapwrite 0x02fa63 thru 0x03da85 (0xe023 bytes) 000563 1761236059.801431 trunc from 0x00ac2d to 0x00cd3b (0x210f bytes) 000565 1761236059.810297 mapread 0x0264ea thru 0x02d307 (0x6e1e bytes) 000566 1761236059.812789 read 0x03d46b thru 0x03da85 (0x61b bytes) 000567 1761236059.813831 fallocate 0x03c242 thru 0x03ffff (0x3dbe bytes) 000568 1761236059.830387 fallocate 0x01facf thru 0x02454d (0x4a7f bytes) 000569 1761236059.841999 read 0x0305d8 thru 0x03da85 (0xd4ae bytes) 000570 1761236059.843091 trunc from 0x02dbb9 to 0x02f6e3 (0x1b2b bytes) 000571 1761236059.859246 punch from 0x0223cf to 0x02d625 (0xb257 bytes) 000572 1761236059.878160 MIRROR_EXTEND to 2 mirrors 000573 1761236060.899780 mapread 0x0165d8 thru 0x01adc4 (0x47ed bytes) 000574 1761236060.938182 trunc from 0x03da86 to 0x0354e9 (0x859c bytes) 000576 1761236061.012152 mapwrite 0x02e8cf thru 0x030c82 (0x23b4 bytes) 000577 1761236061.064104 punch from 0x024eff to 0x03240e (0xd510 bytes) 000578 1761236061.075227 read 0x01b51f thru 0x01c106 (0xbe8 bytes) 000579 1761236061.091890 fallocate 0x006d5a thru 0x00c722 (0x59c9 bytes) 000580 1761236061.109881 write 0x005002 thru 0x014bf8 (0xfbf7 bytes) 000582 1761236061.134276 mapwrite 0x017999 thru 0x024da4 (0xd40c bytes) 000583 1761236061.191530 read 0x007fbf thru 0x014eb0 (0xcef2 bytes) 000584 1761236061.195027 punch from 0x00e9bc to 0x00f2b1 (0x8f6 bytes) 000585 1761236061.205172 mapread 0x0059a3 thru 0x00ba94 (0x60f2 bytes) 000586 1761236061.209844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000587 1761236061.368439 read 0x02f362 thru 0x0354e9 (0x6188 bytes) 000588 1761236061.384755 mapwrite 0x022231 thru 0x025448 (0x3218 bytes) 000589 1761236061.424889 fallocate 0x01e23e thru 0x02c129 (0xdeec bytes) 000590 1761236061.429893 punch from 0x01171f to 0x01d324 (0xbc06 bytes) 000592 1761236061.437022 write 0x0083e5 thru 0x00b31c (0x2f38 bytes) 000593 1761236061.451102 mapwrite 0x027cd3 thru 0x036495 (0xe7c3 bytes) 000595 1761236061.498781 read 0x017bc9 thru 0x023533 (0xb96b bytes) 000596 1761236061.506284 write 0x03ebd3 thru 0x03ffff (0x142d bytes) 000598 1761236061.508843 read 0x01be65 thru 0x02ab33 (0xeccf bytes) 000599 1761236061.510215 punch from 0x001053 to 0x00a2e4 (0x9292 bytes) 000600 1761236061.516957 mapread 0x00148d thru 0x00251d (0x1091 bytes) 000601 1761236061.534907 trunc from 0x040000 to 0x022083 (0x1df7c bytes) 000602 1761236061.554856 mapread 0x0192a2 thru 0x022083 (0x8de2 bytes) 000604 1761236061.559218 fallocate 0x028825 thru 0x028fa4 (0x780 bytes) 000605 1761236061.564180 mapread 0x00b70c thru 0x01a920 (0xf215 bytes) 000606 1761236061.565004 read 0x00d316 thru 0x01050e (0x31f9 bytes) 000607 1761236061.565534 mapwrite 0x029bca thru 0x0320c5 (0x84fc bytes) 000608 1761236061.611968 fallocate 0x0153c6 thru 0x016bd6 (0x1811 bytes) 000610 1761236061.617253 mapread 0x00de50 thru 0x01bbd7 (0xdd88 bytes) 000611 1761236061.618081 punch from 0x002d17 to 0x0127a8 (0xfa92 bytes) 000612 1761236061.622905 read 0x02caa3 thru 0x031c72 (0x51d0 bytes) 000613 1761236061.623494 mapwrite 0x027f73 thru 0x035785 (0xd813 bytes) 000614 1761236061.679965 MIRROR_EXTEND to 2 mirrors 000615 1761236062.427007 read 0x00aa71 thru 0x011732 (0x6cc2 bytes) 000616 1761236062.449554 mapread 0x008580 thru 0x0093b4 (0xe35 bytes) 000617 1761236062.493765 mapwrite 0x00e757 thru 0x011fbd (0x3867 bytes) 000618 1761236062.607374 fallocate 0x027255 thru 0x02daa2 (0x684e bytes) 000620 1761236062.614877 read 0x020106 thru 0x026311 (0x620c bytes) 000621 1761236062.636625 trunc from 0x014335 to 0x018e1f (0x4aeb bytes) 000622 1761236062.647624 trunc from 0x01bb5c to 0x029cb7 (0xe15c bytes) 000623 1761236062.664386 fallocate 0x0194b2 thru 0x023b83 (0xa6d2 bytes) 000624 1761236062.677184 fallocate 0x02e39e thru 0x03aa2d (0xc690 bytes) 000625 1761236062.686023 mapwrite 0x02331a thru 0x0270d0 (0x3db7 bytes) 000626 1761236062.812747 read 0x008ac8 thru 0x0151f5 (0xc72e bytes) 000629 1761236062.829099 mapwrite 0x03e1a5 thru 0x03ffff (0x1e5b bytes) 000630 1761236062.897487 trunc from 0x040000 to 0x013f7c (0x2c083 bytes) 000631 1761236062.935755 mapread 0x00fd53 thru 0x013f7c (0x422a bytes) 000632 1761236062.937441 mapwrite 0x025192 thru 0x02bb54 (0x69c3 bytes) 000634 1761236063.045941 trunc from 0x02bb55 to 0x01c628 (0xf52c bytes) 000635 1761236063.066363 mapwrite 0x03933f thru 0x039861 (0x523 bytes) 000636 1761236063.117438 punch from 0x00a25b to 0x00c40b (0x21b1 bytes) 000637 1761236063.124095 fallocate 0x006e9d thru 0x00df44 (0x70a8 bytes) 000638 1761236063.129289 trunc from 0x039862 to 0x038b8b (0xcd6 bytes) 000639 1761236063.145960 trunc from 0x01ee4e to 0x024695 (0x5848 bytes) 000640 1761236063.150813 read 0x037461 thru 0x038b8b (0x172b bytes) 000641 1761236063.166696 read 0x0243dc thru 0x02a7b3 (0x63d8 bytes) 000643 1761236063.167834 read 0x005eb6 thru 0x011b46 (0xbc91 bytes) 000644 1761236063.170324 trunc from 0x026956 to 0x02c319 (0x59c4 bytes) 000645 1761236063.180491 trunc from 0x02627d to 0x035c5e (0xf9e2 bytes) 000646 1761236063.186372 mapwrite 0x004e88 thru 0x00c8f8 (0x7a71 bytes) 000647 1761236063.222637 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000648 1761236063.403833 fallocate 0x03a05c thru 0x03ffff (0x5fa4 bytes) 000650 1761236063.411765 trunc from 0x0084c9 to 0x00f943 (0x747b bytes) 000651 1761236063.418547 fallocate 0x0376d9 thru 0x03ffff (0x8927 bytes) 000652 1761236063.425068 trunc from 0x033990 to 0x03ffff (0xc670 bytes) 000653 1761236063.429940 mapwrite 0x030ebf thru 0x0368f3 (0x5a35 bytes) 000654 1761236063.472323 mapread 0x031c77 thru 0x039b9f (0x7f29 bytes) 000655 1761236063.473403 trunc from 0x032090 to 0x03ffff (0xdf70 bytes) 000656 1761236063.482502 trunc from 0x040000 to 0x03faa4 (0x55b bytes) 000657 1761236063.512188 punch from 0x008c9b to 0x00a034 (0x139a bytes) 000658 1761236063.523284 fallocate 0x00f460 thru 0x017d41 (0x88e2 bytes) 000659 1761236063.537722 mapread 0x03c7a8 thru 0x03cc98 (0x4f1 bytes) 000661 1761236063.556535 MIRROR_EXTEND to 2 mirrors 000662 1761236064.346277 mapread 0x0026df thru 0x010147 (0xda69 bytes) 000663 1761236064.397322 read 0x03c690 thru 0x03f2c1 (0x2c32 bytes) 000664 1761236064.410632 read 0x00b405 thru 0x0173fc (0xbff8 bytes) 000665 1761236064.451289 fallocate 0x034179 thru 0x039129 (0x4fb1 bytes) 000666 1761236064.510421 trunc from 0x03faa5 to 0x00601f (0x39a85 bytes) 000667 1761236064.559399 trunc from 0x006020 to 0x03c4ed (0x364ce bytes) 000668 1761236064.594310 trunc from 0x03c4ee to 0x025c73 (0x1687a bytes) 000669 1761236064.620146 punch from 0x0119a0 to 0x011f28 (0x589 bytes) 000670 1761236064.631635 trunc from 0x025c74 to 0x019bd4 (0xc09f bytes) 000672 1761236064.664768 write 0x023985 thru 0x026304 (0x2980 bytes) 000673 1761236064.671697 trunc from 0x001100 to 0x00af9a (0x9e9b bytes) 000674 1761236064.678318 trunc from 0x019a69 to 0x01eb20 (0x50b8 bytes) 000675 1761236064.683934 write 0x001bc6 thru 0x010024 (0xe45f bytes) 000677 1761236064.703896 trunc from 0x007691 to 0x00cc16 (0x5586 bytes) 000678 1761236064.722720 mapwrite 0x0381cb thru 0x03d19c (0x4fd2 bytes) 000679 1761236064.800068 fallocate 0x029617 thru 0x03722c (0xdc16 bytes) 000680 1761236064.806199 punch from 0x013984 to 0x018998 (0x5015 bytes) 000681 1761236064.814808 trunc from 0x01a2ec to 0x01c7af (0x24c4 bytes) 000682 1761236064.826636 fallocate 0x02c592 thru 0x02eba1 (0x2610 bytes) 000683 1761236064.832826 write 0x02b5d5 thru 0x031239 (0x5c65 bytes) 000684 1761236064.844145 punch from 0x00a14a to 0x016c13 (0xcaca bytes) 000685 1761236064.852423 fallocate 0x01f9aa thru 0x029923 (0x9f7a bytes) 000686 1761236064.860064 write 0x02d8c2 thru 0x0359f5 (0x8134 bytes) 000687 1761236064.872316 read 0x0016a3 thru 0x00f4f7 (0xde55 bytes) 000688 1761236064.894284 read 0x015d7b thru 0x01efc0 (0x9246 bytes) 000689 1761236064.895019 trunc from 0x009a6d to 0x00e8c2 (0x4e56 bytes) 000690 1761236064.904128 trunc from 0x03773c to 0x03a02d (0x28f2 bytes) 000691 1761236064.910037 trunc from 0x03d19d to 0x027bd6 (0x155c6 bytes) 000692 1761236064.932939 trunc from 0x0185a8 to 0x01bdac (0x3805 bytes) 000693 1761236064.938245 mapwrite 0x032f5b thru 0x0373e0 (0x4486 bytes) 000695 1761236065.004003 fallocate 0x00b630 thru 0x00ff08 (0x48d9 bytes) 000696 1761236065.011444 read 0x019b8f thru 0x01b316 (0x1788 bytes) 000697 1761236065.013378 mapwrite 0x012d08 thru 0x015750 (0x2a49 bytes) 000699 1761236065.047178 MIRROR_EXTEND to 3 mirrors 000700 1761236065.795618 mapwrite 0x00592a thru 0x011807 (0xbede bytes) 000701 1761236065.913113 read 0x01ffa0 thru 0x025988 (0x59e9 bytes) 000702 1761236065.925714 read 0x0221f7 thru 0x031bf0 (0xf9fa bytes) 000703 1761236065.940785 MIRROR_RESYNC 3 mirrors 000704 1761236067.069354 mapwrite 0x0316d4 thru 0x03c9ca (0xb2f7 bytes) 000705 1761236067.176357 write 0x01933f thru 0x01eb52 (0x5814 bytes) 000706 1761236067.186751 trunc from 0x0078b6 to 0x010c2d (0x9378 bytes) 000707 1761236067.192688 punch from 0x0282b7 to 0x02d4c7 (0x5211 bytes) 000709 1761236067.207398 trunc from 0x03c9cb to 0x0075ad (0x3541d bytes) 000711 1761236067.242767 trunc from 0x004dd5 to 0x0075ad (0x27d9 bytes) 000712 1761236067.250082 trunc from 0x0075ae to 0x02cc8b (0x256de bytes) 000713 1761236067.269853 mapwrite 0x02d43d thru 0x039af2 (0xc6b6 bytes) 000714 1761236067.318863 trunc from 0x039af3 to 0x016150 (0x239a2 bytes) 000715 1761236067.337616 fallocate 0x00ecbf thru 0x0155ab (0x68ed bytes) 000717 1761236067.349695 trunc from 0x014b70 to 0x016150 (0x15e1 bytes) 000718 1761236067.356561 write 0x039f1a thru 0x03ffff (0x60e6 bytes) 000719 1761236067.358000 mapwrite 0x02bef5 thru 0x037702 (0xb80e bytes) 000720 1761236067.391856 mapread 0x0296ce thru 0x032bd5 (0x9508 bytes) 000721 1761236067.393882 trunc from 0x040000 to 0x02f307 (0x10cf8 bytes) 000722 1761236067.414783 write 0x03404d thru 0x038d77 (0x4d2b bytes) 000723 1761236067.416596 punch from 0x014141 to 0x018540 (0x4400 bytes) 000724 1761236067.426156 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000725 1761236067.668981 mapwrite 0x0296e9 thru 0x02b783 (0x209b bytes) 000726 1761236067.698671 read 0x00fb26 thru 0x01b8de (0xbdb9 bytes) 000727 1761236067.707826 mapwrite 0x023ec9 thru 0x02f8cb (0xba03 bytes) 000728 1761236067.738493 mapread 0x017616 thru 0x023b06 (0xc4f1 bytes) 000729 1761236067.740302 mapread 0x0065eb thru 0x0074fa (0xf10 bytes) 000730 1761236067.752428 write 0x0011d2 thru 0x004fb0 (0x3ddf bytes) 000732 1761236067.758314 punch from 0x009fb2 to 0x016619 (0xc668 bytes) 000733 1761236067.766070 mapwrite 0x00763e thru 0x013619 (0xbfdc bytes) 000736 1761236067.802930 punch from 0x006579 to 0x007c1a (0x16a2 bytes) 000737 1761236067.809791 trunc from 0x02d15f to 0x038d77 (0xbc19 bytes) 000738 1761236067.821305 MIRROR_RESYNC 2 mirrors 000739 1761236068.500112 mapwrite 0x00b82d thru 0x00ecc1 (0x3495 bytes) 000740 1761236068.592924 fallocate 0x034cc7 thru 0x035a24 (0xd5e bytes) 000741 1761236068.602231 mapread 0x0286c2 thru 0x02f7ae (0x70ed bytes) 000742 1761236068.611955 trunc from 0x038d78 to 0x001fa9 (0x36dce bytes) 000743 1761236068.632602 mapwrite 0x0305cc thru 0x038fd6 (0x8a0b bytes) 000745 1761236068.685051 MIRROR_EXTEND to 3 mirrors 000746 1761236069.472810 MIRROR_RESYNC 3 mirrors 000747 1761236070.083826 mapread 0x013dea thru 0x014f23 (0x113a bytes) 000748 1761236070.109425 read 0x021497 thru 0x021dac (0x916 bytes) 000749 1761236070.116492 trunc from 0x038fd7 to 0x025a8b (0x1354b bytes) 000752 1761236070.178839 trunc from 0x025a8c to 0x028c25 (0x319a bytes) 000753 1761236070.201124 punch from 0x025964 to 0x028c25 (0x32c2 bytes) 000754 1761236070.205986 fallocate 0x024bfd thru 0x02b4ab (0x68af bytes) 000755 1761236070.211850 mapread 0x01174d thru 0x015475 (0x3d29 bytes) 000757 1761236070.223357 mapread 0x02192f thru 0x028c25 (0x72f7 bytes) 000758 1761236070.232053 mapread 0x006956 thru 0x00eb06 (0x81b1 bytes) 000759 1761236070.240757 mapread 0x015296 thru 0x016ceb (0x1a56 bytes) 000760 1761236070.241626 mapwrite 0x004e5f thru 0x012532 (0xd6d4 bytes) 000761 1761236070.261401 mapread 0x013a46 thru 0x022d16 (0xf2d1 bytes) 000762 1761236070.267288 MIRROR_EXTEND to 4 mirrors 000763 1761236070.972203 MIRROR_RESYNC 4 mirrors 000765 1761236071.908886 mapread 0x00ade2 thru 0x017e12 (0xd031 bytes) 000766 1761236071.932107 write 0x004f52 thru 0x008221 (0x32d0 bytes) 000767 1761236071.991701 MIRROR_EXTEND to 5 mirrors 000768 1761236072.708730 mapwrite 0x02a68f thru 0x036e43 (0xc7b5 bytes) 000769 1761236072.786177 trunc from 0x010f01 to 0x0110d1 (0x1d1 bytes) 000770 1761236072.796943 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000771 1761236072.996321 fallocate 0x01f2e5 thru 0x024aea (0x5806 bytes) 000772 1761236073.003992 mapwrite 0x02b035 thru 0x033b3a (0x8b06 bytes) 000773 1761236073.051774 write 0x009f16 thru 0x00cab7 (0x2ba2 bytes) 000774 1761236073.071561 read 0x00bb7d thru 0x015c36 (0xa0ba bytes) 000775 1761236073.082025 mapread 0x02fa90 thru 0x036e43 (0x73b4 bytes) 000776 1761236073.109574 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000777 1761236073.337859 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000778 1761236073.570203 read 0x0036dc thru 0x0108a9 (0xd1ce bytes) 000779 1761236073.582673 fallocate 0x001507 thru 0x010661 (0xf15b bytes) 000780 1761236073.588683 punch from 0x01d1ab to 0x0224cf (0x5325 bytes) 000781 1761236073.595535 trunc from 0x013ff5 to 0x01a1fe (0x620a bytes) 000782 1761236073.600781 trunc from 0x025edf to 0x03148d (0xb5af bytes) 000783 1761236073.607848 write 0x02fb31 thru 0x0324f6 (0x29c6 bytes) 000785 1761236073.624809 mapwrite 0x03cfac thru 0x03ffff (0x3054 bytes) 000786 1761236073.664421 read 0x035869 thru 0x03ffff (0xa797 bytes) 000788 1761236073.665596 mapwrite 0x00f552 thru 0x01b738 (0xc1e7 bytes) 000789 1761236073.698514 mapwrite 0x0323f6 thru 0x0331eb (0xdf6 bytes) 000790 1761236073.717230 mapread 0x0047b8 thru 0x014286 (0xfacf bytes) 000791 1761236073.720380 punch from 0x02fba9 to 0x0377ff (0x7c57 bytes) 000792 1761236073.725425 read 0x00de3e thru 0x0165c8 (0x878b bytes) 000793 1761236073.726119 trunc from 0x031218 to 0x03ffff (0xede8 bytes) 000794 1761236073.732279 mapwrite 0x00abad thru 0x00dd28 (0x317c bytes) 000795 1761236073.749477 fallocate 0x033721 thru 0x03ffff (0xc8df bytes) 000796 1761236073.754270 read 0x010a34 thru 0x012841 (0x1e0e bytes) 000797 1761236073.759936 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000798 1761236073.894539 read 0x03ff8e thru 0x03ffff (0x72 bytes) 000799 1761236073.902113 fallocate 0x00d102 thru 0x01a11c (0xd01b bytes) 000800 1761236073.909353 read 0x02c439 thru 0x03694a (0xa512 bytes) 000801 1761236073.921622 punch from 0x016a45 to 0x024aaf (0xe06b bytes) 000802 1761236073.926248 read 0x02d2e4 thru 0x034093 (0x6db0 bytes) 000803 1761236073.926901 read 0x0001fe thru 0x002d9f (0x2ba2 bytes) 000804 1761236073.927380 punch from 0x0160c7 to 0x0223d9 (0xc313 bytes) 000805 1761236073.932807 read 0x0360bb thru 0x03dafc (0x7a42 bytes) 000806 1761236073.933384 mapwrite 0x02fcd5 thru 0x03a76b (0xaa97 bytes) 000807 1761236073.953875 punch from 0x030ace to 0x03ffff (0xf532 bytes) 000808 1761236073.960912 punch from 0x0283f9 to 0x030613 (0x821b bytes) 000809 1761236073.966274 fallocate 0x015d6e thru 0x018cfa (0x2f8d bytes) 000810 1761236073.972932 read 0x01ef87 thru 0x021c76 (0x2cf0 bytes) 000812 1761236073.984338 punch from 0x03f1b5 to 0x03ffff (0xe4b bytes) 000813 1761236073.993242 MIRROR_EXTEND to 2 mirrors 000814 1761236074.684394 trunc from 0x019355 to 0x023518 (0xa1c4 bytes) 000815 1761236074.720082 write 0x02c901 thru 0x03147e (0x4b7e bytes) 000816 1761236074.736277 mapread 0x037995 thru 0x03e46b (0x6ad7 bytes) 000817 1761236074.744926 read 0x004c70 thru 0x006e49 (0x21da bytes) 000818 1761236074.766150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000819 1761236074.936293 write 0x0008ff thru 0x00cb33 (0xc235 bytes) 000820 1761236074.958935 write 0x002d9a thru 0x003fd2 (0x1239 bytes) 000821 1761236074.963217 write 0x017573 thru 0x02525d (0xdceb bytes) 000822 1761236074.982339 fallocate 0x036d0d thru 0x03ffff (0x92f3 bytes) 000824 1761236074.990948 trunc from 0x040000 to 0x02f541 (0x10abe bytes) 000825 1761236075.014805 punch from 0x0119c7 to 0x01666d (0x4ca7 bytes) 000826 1761236075.020993 write 0x018d83 thru 0x022280 (0x94fe bytes) 000827 1761236075.023248 mapwrite 0x007838 thru 0x01156c (0x9d35 bytes) 000830 1761236075.059514 mapread 0x00e25c thru 0x01237f (0x4124 bytes) 000832 1761236075.062013 punch from 0x00d931 to 0x00f807 (0x1ed7 bytes) 000833 1761236075.067253 read 0x0100a8 thru 0x01f1d3 (0xf12c bytes) 000834 1761236075.072744 trunc from 0x004a94 to 0x011e33 (0xd3a0 bytes) 000835 1761236075.080788 trunc from 0x02f542 to 0x013082 (0x1c4bf bytes) 000836 1761236075.098022 trunc from 0x00386b to 0x009781 (0x5f17 bytes) 000837 1761236075.105772 write 0x0233b6 thru 0x032eef (0xfb3a bytes) 000838 1761236075.107766 mapwrite 0x017cb9 thru 0x025e5b (0xe1a3 bytes) 000840 1761236075.165087 trunc from 0x032ef0 to 0x011b56 (0x21399 bytes) 000841 1761236075.187239 mapread 0x007986 thru 0x00e9f4 (0x706f bytes) 000842 1761236075.198860 read 0x0107c5 thru 0x011b56 (0x1392 bytes) 000843 1761236075.200212 mapread 0x007fc4 thru 0x010afe (0x8b3b bytes) 000844 1761236075.201171 mapread 0x00adc4 thru 0x011b56 (0x6d93 bytes) 000845 1761236075.201976 mapread 0x00885c thru 0x00a124 (0x18c9 bytes) 000846 1761236075.202685 trunc from 0x011b57 to 0x037c38 (0x260e2 bytes) 000847 1761236075.218884 mapwrite 0x0031e4 thru 0x003e3e (0xc5b bytes) 000848 1761236075.234019 trunc from 0x02315c to 0x0235ce (0x473 bytes) 000849 1761236075.241410 punch from 0x0016ed to 0x004e31 (0x3745 bytes) 000850 1761236075.249316 mapwrite 0x0078a4 thru 0x015262 (0xd9bf bytes) 000851 1761236075.298995 write 0x01de7c thru 0x028c01 (0xad86 bytes) 000852 1761236075.303907 punch from 0x036d1f to 0x037c38 (0xf1a bytes) 000853 1761236075.317928 trunc from 0x037254 to 0x037c38 (0x9e5 bytes) 000854 1761236075.323793 fallocate 0x02a082 thru 0x036983 (0xc902 bytes) 000855 1761236075.334991 read 0x0037f9 thru 0x006199 (0x29a1 bytes) 000856 1761236075.337896 mapwrite 0x016ff3 thru 0x01ef05 (0x7f13 bytes) 000857 1761236075.361966 read 0x0052ec thru 0x01268c (0xd3a1 bytes) 000858 1761236075.362676 mapwrite 0x017d8f thru 0x022873 (0xaae5 bytes) 000859 1761236075.390618 punch from 0x02404f to 0x02c192 (0x8144 bytes) 000860 1761236075.401711 mapwrite 0x00374c thru 0x0135eb (0xfea0 bytes) 000861 1761236075.438267 punch from 0x01923f to 0x019ec2 (0xc84 bytes) 000862 1761236075.442580 trunc from 0x037c39 to 0x0075c0 (0x30678 bytes) 000863 1761236075.467610 mapread 0x0014d6 thru 0x0075c0 (0x60eb bytes) 000864 1761236075.470803 punch from 0x00185e to 0x004c67 (0x340a bytes) 000865 1761236075.487726 MIRROR_EXTEND to 2 mirrors 000866 1761236076.069357 mapwrite 0x018827 thru 0x020ccc (0x84a6 bytes) 000867 1761236076.172124 mapread 0x000735 thru 0x00b2e8 (0xabb4 bytes) 000868 1761236076.188700 read 0x01c2f1 thru 0x020ccc (0x49dc bytes) 000869 1761236076.189272 punch from 0x01ed43 to 0x020ccc (0x1f8a bytes) 000870 1761236076.194738 mapread 0x00d064 thru 0x012743 (0x56e0 bytes) 000871 1761236076.201924 read 0x004427 thru 0x004559 (0x133 bytes) 000872 1761236076.202628 read 0x0085a4 thru 0x017786 (0xf1e3 bytes) 000874 1761236076.203699 trunc from 0x00be97 to 0x019a09 (0xdb73 bytes) 000876 1761236076.213684 write 0x03ea44 thru 0x03ffff (0x15bc bytes) 000879 1761236076.216166 write 0x02bf07 thru 0x033d1a (0x7e14 bytes) 000880 1761236076.226084 mapwrite 0x001161 thru 0x003ed8 (0x2d78 bytes) 000881 1761236076.241893 fallocate 0x012690 thru 0x022303 (0xfc74 bytes) 000883 1761236076.250547 write 0x00327c thru 0x00adcc (0x7b51 bytes) 000884 1761236076.260959 MIRROR_EXTEND to 3 mirrors 000885 1761236077.044297 fallocate 0x026295 thru 0x02e136 (0x7ea2 bytes) 000887 1761236077.086969 fallocate 0x01307a thru 0x018095 (0x501c bytes) 000889 1761236077.103205 fallocate 0x030dfd thru 0x03d4f0 (0xc6f4 bytes) 000890 1761236077.110847 read 0x0124d4 thru 0x017231 (0x4d5e bytes) 000891 1761236077.131981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000892 1761236077.331647 write 0x03c4c2 thru 0x03ffff (0x3b3e bytes) 000893 1761236077.344559 write 0x02289d thru 0x030efc (0xe660 bytes) 000894 1761236077.355332 mapwrite 0x03518b thru 0x03bb3d (0x69b3 bytes) 000895 1761236077.376980 trunc from 0x040000 to 0x02e8b4 (0x1174b bytes) 000896 1761236077.416652 mapread 0x01505f thru 0x01595c (0x8fe bytes) 000897 1761236077.426127 mapread 0x00adc8 thru 0x012fc3 (0x81fc bytes) 000898 1761236077.435652 punch from 0x0181b8 to 0x020848 (0x8691 bytes) 000899 1761236077.441390 mapwrite 0x01cd0a thru 0x02877d (0xba74 bytes) 000900 1761236077.472305 mapwrite 0x035599 thru 0x03b3e1 (0x5e49 bytes) 000901 1761236077.527314 mapwrite 0x026f50 thru 0x035891 (0xe942 bytes) 000902 1761236077.551127 trunc from 0x03b3e2 to 0x02daff (0xd8e2 bytes) 000903 1761236077.574927 trunc from 0x004a5a to 0x0098cf (0x4e76 bytes) 000904 1761236077.582038 trunc from 0x02db00 to 0x031cb2 (0x41b3 bytes) 000905 1761236077.604876 punch from 0x01d0c7 to 0x0263b1 (0x92eb bytes) 000906 1761236077.612715 trunc from 0x031cb3 to 0x01821d (0x19a95 bytes) 000907 1761236077.632201 MIRROR_RESYNC 2 mirrors 000908 1761236078.259665 punch from 0x01746a to 0x01821d (0xdb4 bytes) 000909 1761236078.324524 trunc from 0x00f1cc to 0x01821d (0x9052 bytes) 000910 1761236078.335789 mapread 0x0000d3 thru 0x003ce5 (0x3c13 bytes) 000911 1761236078.379590 trunc from 0x01821e to 0x001816 (0x16a07 bytes) 000912 1761236078.455237 mapread 0x000b06 thru 0x001816 (0xd11 bytes) 000913 1761236078.456249 read 0x001711 thru 0x001816 (0x106 bytes) 000914 1761236078.456799 trunc from 0x001817 to 0x0291e8 (0x279d2 bytes) 000915 1761236078.504246 trunc from 0x0291e9 to 0x01136e (0x17e7a bytes) 000916 1761236078.538972 fallocate 0x006b68 thru 0x013510 (0xc9a9 bytes) 000917 1761236078.546899 fallocate 0x01cda5 thru 0x028733 (0xb98f bytes) 000918 1761236078.555859 read 0x0075dc thru 0x009cd8 (0x26fd bytes) 000919 1761236078.575555 trunc from 0x028734 to 0x0187d6 (0xff5d bytes) 000920 1761236078.605784 mapwrite 0x02229c thru 0x024ae1 (0x2846 bytes) 000921 1761236078.682426 punch from 0x0036b1 to 0x011ea0 (0xe7f0 bytes) 000922 1761236078.689136 write 0x0299f5 thru 0x0341ab (0xa7b7 bytes) 000923 1761236078.692797 punch from 0x030be4 to 0x03382c (0x2c49 bytes) 000924 1761236078.722915 trunc from 0x0341ac to 0x00cd02 (0x274a9 bytes) 000925 1761236078.749861 write 0x039334 thru 0x03ef85 (0x5c52 bytes) 000926 1761236078.751994 mapwrite 0x03f7ad thru 0x03ffff (0x853 bytes) 000927 1761236078.805819 punch from 0x035189 to 0x038ecf (0x3d47 bytes) 000928 1761236078.815873 trunc from 0x040000 to 0x01669a (0x29965 bytes) 000929 1761236078.833123 mapread 0x000f61 thru 0x010158 (0xf1f8 bytes) 000930 1761236078.913167 punch from 0x003902 to 0x0077a6 (0x3ea5 bytes) 000931 1761236078.918132 trunc from 0x01669b to 0x01cdf9 (0x675f bytes) 000932 1761236078.939054 write 0x00d3a4 thru 0x018731 (0xb38e bytes) 000933 1761236078.945899 read 0x01cb1a thru 0x01cdf9 (0x2e0 bytes) 000934 1761236078.956654 write 0x0016f4 thru 0x010b23 (0xf430 bytes) 000935 1761236078.958308 write 0x02682e thru 0x02d47d (0x6c50 bytes) 000936 1761236078.960217 punch from 0x01d5f7 to 0x028ad4 (0xb4de bytes) 000937 1761236078.977369 mapread 0x00552c thru 0x00fbcf (0xa6a4 bytes) 000938 1761236078.979215 mapwrite 0x02127d thru 0x027eda (0x6c5e bytes) 000940 1761236079.004398 write 0x007a0f thru 0x015d4f (0xe341 bytes) 000941 1761236079.007173 trunc from 0x01b3ba to 0x0225a8 (0x71ef bytes) 000942 1761236079.014939 write 0x012afd thru 0x02255a (0xfa5e bytes) 000943 1761236079.023679 write 0x032f47 thru 0x03fe9a (0xcf54 bytes) 000944 1761236079.025528 punch from 0x0360d7 to 0x03ee1f (0x8d49 bytes) 000945 1761236079.043881 read 0x031607 thru 0x03fe9a (0xe894 bytes) 000946 1761236079.062195 MIRROR_RESYNC 2 mirrors 000947 1761236079.707149 fallocate 0x006348 thru 0x00e38a (0x8043 bytes) 000949 1761236079.744571 trunc from 0x03fe9b to 0x020d75 (0x1f125 bytes) 000950 1761236079.821710 fallocate 0x034a3b thru 0x0379b5 (0x2f7b bytes) 000952 1761236079.826909 trunc from 0x0379b6 to 0x029330 (0xe685 bytes) 000953 1761236079.849422 read 0x016310 thru 0x020c42 (0xa933 bytes) 000954 1761236079.859213 mapread 0x020d8e thru 0x029330 (0x85a3 bytes) 000956 1761236079.876776 fallocate 0x01fc1b thru 0x02d557 (0xd93d bytes) 000957 1761236079.881124 fallocate 0x033de3 thru 0x0393d6 (0x55f4 bytes) 000958 1761236079.887105 write 0x005e56 thru 0x00be0f (0x5fba bytes) 000959 1761236079.895628 read 0x00d450 thru 0x017b45 (0xa6f6 bytes) 000961 1761236079.904609 read 0x005c76 thru 0x00c8de (0x6c69 bytes) 000962 1761236079.905949 trunc from 0x00469b to 0x0087ce (0x4134 bytes) 000963 1761236079.927230 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c2:0x0]: Invalid argument (22) 000965 1761236080.077482 trunc from 0x0393d7 to 0x0130ce (0x26308 bytes) 000966 1761236080.105575 trunc from 0x00efc1 to 0x0127c5 (0x3805 bytes) 000967 1761236080.112116 trunc from 0x0130cf to 0x03a34d (0x2727f bytes) 000968 1761236080.136884 fallocate 0x038d50 thru 0x03ffff (0x72b0 bytes) 000969 1761236080.142248 fallocate 0x0200f7 thru 0x0243a9 (0x42b3 bytes) 000970 1761236080.148098 punch from 0x0007c9 to 0x0023bd (0x1bf5 bytes) 000971 1761236080.159893 trunc from 0x03a34e to 0x02d935 (0xca18 bytes) 000972 1761236080.173533 mapwrite 0x03dbdb thru 0x03ffff (0x2425 bytes) 000974 1761236080.263083 trunc from 0x020d19 to 0x024d8a (0x4072 bytes) 000975 1761236080.271898 punch from 0x01b23e to 0x024990 (0x9753 bytes) 000977 1761236080.277358 read 0x01414a thru 0x017cda (0x3b91 bytes) 000978 1761236080.285292 write 0x0361af thru 0x03e40a (0x825c bytes) 000979 1761236080.294317 mapwrite 0x014e13 thru 0x0241f2 (0xf3e0 bytes) 000980 1761236080.339928 mapwrite 0x0339ee thru 0x034a81 (0x1094 bytes) 000982 1761236080.370959 fallocate 0x037924 thru 0x03dfe9 (0x66c6 bytes) 000983 1761236080.376838 mapwrite 0x01043d thru 0x01c457 (0xc01b bytes) 000985 1761236080.414661 mapwrite 0x03b853 thru 0x03ffff (0x47ad bytes) 000987 1761236080.433235 mapread 0x034650 thru 0x03fdc7 (0xb778 bytes) 000988 1761236080.434157 fallocate 0x00cd24 thru 0x013449 (0x6726 bytes) 000989 1761236080.440858 write 0x011eca thru 0x01ad7c (0x8eb3 bytes) 000990 1761236080.442868 mapwrite 0x03f0dd thru 0x03ffff (0xf23 bytes) 000991 1761236080.460345 mapread 0x002b32 thru 0x009855 (0x6d24 bytes) 000992 1761236080.470183 write 0x0223d0 thru 0x023901 (0x1532 bytes) 000993 1761236080.471333 read 0x00b6fb thru 0x00bdd5 (0x6db bytes) 000995 1761236080.472614 fallocate 0x023d6c thru 0x02c960 (0x8bf5 bytes) 000996 1761236080.478359 fallocate 0x02d830 thru 0x039679 (0xbe4a bytes) 000997 1761236080.484504 mapread 0x0094c7 thru 0x016f09 (0xda43 bytes) 000998 1761236080.485390 read 0x026ce9 thru 0x02ec01 (0x7f19 bytes) 000999 1761236080.498948 read 0x00d927 thru 0x016c90 (0x936a bytes) All operations completed A-OK! PASS 70a (64s) == sanity-flr test 71: check mirror extend parent fid ==== 12:14:45 (1761236085) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x1fb:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 200a: stress test ===================== 12:14:53 (1761236093) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7222336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2397504 .. Extending file size to 3405728 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5652672 .. Extending file size to 5889024 .. Extending file size to 8816864 .. Extending file size to 7689280 .. Extending file size to 9098368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8944352 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59079 59080 59081 59086 59087 done 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:16:08 (1761236168) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5890560 .. Extending file size to 1278528 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6930592 .. Extending file size to 7305984 .. Extending file size to 8272416 .. Extending file size to 8800096 .. Extending file size to 7589536 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9205024 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8848608 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9295552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61320 61321 61324 61326 61327 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:17:21 (1761236241) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15884 3591136 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21976 3585044 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37860 7176180 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15884 3591136 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21976 3585044 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37860 7176180 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0660613 s, 31.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236046 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:17:36 (1761236256) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0768194 s, 27.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 12:17:42 (1761236262) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:17:48 (1761236268) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0885761 s, 23.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:17:55 (1761236275) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0922781 s, 22.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:18:01 (1761236281) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118975 s, 17.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:18:31 (1761236311) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635375 s, 16.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:19:17 (1761236357) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123174 s, 17.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:20:04 (1761236404) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg316-server: Pool lustre.test_204f created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0870429 s, 24.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg316-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:21:07 (1761236467) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:21:14 (1761236474) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:21:19 (1761236479) oleg316-server: Pool lustre.test_206 created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg316-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:21:43 (1761236503) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689372 s, 15.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356797 s, 29.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xeeed04fc Mirror 3: 0xeeed04fc PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:21:52 (1761236512) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:21:55 (1761236515) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:21:58 (1761236518) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0609864 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1410000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb00000 Read this (4 bytes): BBBB 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:22:35 (1761236555) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.033354 s, 246 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:22:41 (1761236561) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372179 s, 28.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611262 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-23 12:22:42.000000000 -0400 Modify: 2025-10-23 12:22:42.000000000 -0400 Change: 2025-10-23 12:22:43.000000000 -0400 Birth: 2025-10-23 12:22:43.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:22:46 (1761236566) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300832 s, 34.9 MB/s oleg316-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg316-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg316-server: O/280000401/d29/bd: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:23:08 (1761236588) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.230314 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.026444 s, 155 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 23 12:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x83:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 23 12:23 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test complete, duration 1426 sec =========== 12:23:15 (1761236595) === sanity-flr: start cleanup 12:23:16 (1761236596) === === sanity-flr: finish cleanup 12:23:23 (1761236603) ===