-----============= acceptance-small: sanity-flr ============----- Fri Oct 24 01:21:56 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg314-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:22:04 (1761283324) === oleg314-client.virtnet: executing check_config_client /mnt/lustre oleg314-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg314-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf704fde800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf704fde800.idle_timeout=debug disable quota as required oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:22:19 (1761283339) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8208] == sanity-flr test 0a: lfs mirror create with -N option == 01:22:20 (1761283340) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:22:25 (1761283345) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:22:27 (1761283347) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:22:28 (1761283348) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:22:37 (1761283357) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:22:39 (1761283359) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:22:40 (1761283360) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9cf704fde800.stats=clear osc.lustre-OST0001-osc-ffff9cf704fde800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:22:43 (1761283363) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:22:47 (1761283367) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:22:50 (1761283370) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:22:52 (1761283372) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 01:22:55 (1761283375) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:22:58 (1761283378) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:23:01 (1761283381) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:23:05 (1761283385) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:23:09 (1761283389) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:23:11 (1761283391) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.282026 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034396 s, 30.5 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:23:14 (1761283394) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181784 s, 92.3 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:23:18 (1761283398) Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:23:52 (1761283432) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.111672 s, 46.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:24:08 (1761283448) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (31s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:24:39 (1761283479) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08932 s, 48.1 MB/s -rw-r--r-- 1 root root 52428800 Oct 24 01:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.338s user 0m0.000s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.447s user 0m0.000s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.355s user 0m0.000s sys 0m0.168s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf704fde800.ost_server_uuid in FULL state after 0 sec PASS 33b (32s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:25:11 (1761283511) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:25:13 (1761283513) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:25:14 (1761283514) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:25:15 (1761283515) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 01:25:18 (1761283518) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.158279 s, 79.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:25:22 (1761283522) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012237 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133249 s, 78.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:25:24 (1761283524) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:25:28 (1761283528) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000801288 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000550808 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000892617 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000820241 s, 1.2 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 01:25:31 (1761283531) create test files with size 15731815 .. f6914cf35a05cf1ee46e761f489fa207 - ebb558b3d093fe1960a40379ba5db63b - acc62f3ff80ba556b195240741e57364 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 01:25:38 (1761283538) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:25:42 (1761283542) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:25:44 (1761283544) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301779 s, 34.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137098 s, 76.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 01:25:47 (1761283547) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0228097 s, 91.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0913791 s, 45.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.041647 s, 50.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 01:25:50 (1761283550) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:25:51 (1761283551) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:25:52 (1761283552) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125159 s, 83.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 01:25:55 (1761283555) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0813203 s, 38.7 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:25:58 (1761283558) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x43:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x43:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:26:01 (1761283561) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118316 s, 88.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:26:21 (1761283581) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9cf710e6f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112455 s, 93.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9cf713372800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:26:26 (1761283586) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:26:38 (1761283598) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:26:51 (1761283611) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:26:52 (1761283612) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0946299 s, 33.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0677863 s, 46.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 7537cce6b8b45ed959a9857f80c03df5 - mirror 1 checksum: da256e7f3582ffc9010b9c60965c43ad - mirror 2 checksum: 7537cce6b8b45ed959a9857f80c03df5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 24 01:26:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 24 01:26:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7537cce6b8b45ed959a9857f80c03df5 - mirror 1 checksum: da256e7f3582ffc9010b9c60965c43ad - mirror 2 checksum: da256e7f3582ffc9010b9c60965c43ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 24 01:26:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 24 01:26:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:26:56 (1761283616) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158525 s, 66.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:26:59 (1761283619) Failing mds1 on oleg314-server Stopping /mnt/lustre-mds1 (opts:) on oleg314-server 01:27:06 (1761283626) shut down facet: mds1 facet_host: oleg314-server facet_failover_host: oleg314-server Failover mds1 to oleg314-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 01:27:21 (1761283641) targets are mounted 01:27:21 (1761283641) facet_failover done oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134186 s, 78.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (32s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:27:30 (1761283650) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0183474 s, 1.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00642266 s, 3.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:27:34 (1761283654) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0176061 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00418582 s, 4.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:27:37 (1761283657) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00352148 s, 4.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:27:40 (1761283660) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0233467 s, 877 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:27:43 (1761283663) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00403868 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157677 s, 260 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:27:47 (1761283667) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:27:55 (1761283675) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:28:15 (1761283695) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x49:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:28:33 (1761283713) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:28:51 (1761283731) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 01:28:54 (1761283734) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55857 mirror split pid 55858 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 55857 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 55858 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 01:29:57 (1761283797) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 3797 skipping zero length zero range skipping zero size read fallocating to largest ever: 0x3f9a2 000003 1761283797.985428 fallocate 0x03cf16 thru 0x03f9a1 (0x2a8c bytes) skipping zero size read skipping zero length punch hole 000006 1761283797.989917 write 0x03b10b thru 0x03c07b (0xf71 bytes) 000007 1761283797.991032 mapread 0x030d84 thru 0x0391e5 (0x8462 bytes) 000008 1761283798.000225 mapread 0x0293a2 thru 0x0343a4 (0xb003 bytes) 000009 1761283798.004825 mapwrite 0x03d6ab thru 0x03ffff (0x2955 bytes) 000010 1761283798.023510 trunc from 0x028520 to 0x034c95 (0xc776 bytes) 000011 1761283798.026473 read 0x03a294 thru 0x03ffff (0x5d6c bytes) 000012 1761283798.027036 mapread 0x007a64 thru 0x00a48c (0x2a29 bytes) 000013 1761283798.033478 MIRROR_EXTEND to 2 mirrors 000014 1761283798.266306 read 0x016f4d thru 0x01c711 (0x57c5 bytes) 000015 1761283798.274638 trunc from 0x040000 to 0x015e19 (0x2a1e6 bytes) 000016 1761283798.300625 trunc from 0x0109c0 to 0x015e19 (0x545a bytes) 000017 1761283798.302999 read 0x006b30 thru 0x009009 (0x24da bytes) 000018 1761283798.307467 mapread 0x00fb57 thru 0x010730 (0xbda bytes) 000019 1761283798.308304 punch from 0x006861 to 0x015051 (0xe7f1 bytes) 000020 1761283798.311026 fallocate 0x00de93 thru 0x011395 (0x3503 bytes) 000022 1761283798.314081 punch from 0x0102cc to 0x01422d (0x3f62 bytes) 000023 1761283798.317044 write 0x01dd43 thru 0x02d2fd (0xf5bb bytes) 000024 1761283798.318893 punch from 0x009347 to 0x01250c (0x91c6 bytes) 000025 1761283798.322115 write 0x03e847 thru 0x03ffff (0x17b9 bytes) 000026 1761283798.323019 fallocate 0x02ff13 thru 0x038a43 (0x8b31 bytes) 000027 1761283798.325843 trunc from 0x040000 to 0x02d8da (0x12725 bytes) 000029 1761283798.334177 mapwrite 0x02d9e6 thru 0x0377ad (0x9dc8 bytes) 000030 1761283798.356973 trunc from 0x0377ae to 0x02396c (0x13e41 bytes) 000031 1761283798.364617 fallocate 0x02de56 thru 0x03d99a (0xfb45 bytes) 000032 1761283798.367505 fallocate 0x000853 thru 0x00597f (0x512d bytes) 000033 1761283798.369974 trunc from 0x02738d to 0x02bd53 (0x49c7 bytes) 000034 1761283798.373532 fallocate 0x01fade thru 0x02049d (0x9c0 bytes) 000035 1761283798.376682 fallocate 0x020c87 thru 0x02c931 (0xbcab bytes) 000036 1761283798.379536 trunc from 0x03d99b to 0x038c00 (0x4d9a bytes) 000037 1761283798.389058 read 0x005e54 thru 0x014710 (0xe8bd bytes) 000039 1761283798.394299 punch from 0x021615 to 0x02843e (0x6e2a bytes) 000040 1761283798.397977 fallocate 0x019ff7 thru 0x01a2a5 (0x2af bytes) 000041 1761283798.400800 punch from 0x02185b to 0x02a984 (0x912a bytes) 000042 1761283798.403953 fallocate 0x008437 thru 0x00c1f7 (0x3dc1 bytes) 000043 1761283798.406781 mapread 0x0113f0 thru 0x01c2ea (0xaefb bytes) 000044 1761283798.407786 read 0x013a4d thru 0x022ff0 (0xf5a4 bytes) 000045 1761283798.412291 punch from 0x02f957 to 0x038c00 (0x92aa bytes) 000046 1761283798.414864 read 0x00add7 thru 0x012c31 (0x7e5b bytes) 000047 1761283798.415320 fallocate 0x027189 thru 0x029f3b (0x2db3 bytes) 000048 1761283798.417880 fallocate 0x0028c7 thru 0x004890 (0x1fca bytes) 000049 1761283798.420782 mapread 0x00ac61 thru 0x012fef (0x838f bytes) 000050 1761283798.424034 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000051 1761283798.484911 mapread 0x036ba5 thru 0x038c00 (0x205c bytes) 000052 1761283798.495554 mapwrite 0x00b7ba thru 0x01681b (0xb062 bytes) 000053 1761283798.515643 trunc from 0x0282b2 to 0x02936b (0x10ba bytes) 000054 1761283798.518803 trunc from 0x03702b to 0x038c00 (0x1bd6 bytes) 000055 1761283798.521892 fallocate 0x007ced thru 0x01549b (0xd7af bytes) 000056 1761283798.525520 trunc from 0x038c01 to 0x028909 (0x102f7 bytes) 000057 1761283798.535228 read 0x01c707 thru 0x023f5e (0x7858 bytes) 000058 1761283798.539689 mapwrite 0x0276bf thru 0x02bf4c (0x488e bytes) 000060 1761283798.560594 mapwrite 0x002012 thru 0x0044e8 (0x24d7 bytes) 000062 1761283798.572063 trunc from 0x02bf4d to 0x007f6d (0x23fdf bytes) 000063 1761283798.579902 write 0x0385d0 thru 0x03ffff (0x7a30 bytes) 000065 1761283798.582105 trunc from 0x040000 to 0x02b63a (0x149c5 bytes) 000066 1761283798.589270 mapwrite 0x022e75 thru 0x02ea39 (0xbbc5 bytes) 000067 1761283798.615202 mapread 0x006e8a thru 0x010b4a (0x9cc1 bytes) 000068 1761283798.620923 trunc from 0x00443b to 0x00dd5b (0x9921 bytes) 000069 1761283798.624870 read 0x00fe7a thru 0x015da8 (0x5f2f bytes) 000070 1761283798.630888 punch from 0x01735a to 0x01f2ac (0x7f53 bytes) 000071 1761283798.634500 read 0x01e7bc thru 0x0299c7 (0xb20c bytes) 000072 1761283798.639760 trunc from 0x02ea3a to 0x039fff (0xb5c6 bytes) 000073 1761283798.650503 punch from 0x023fdd to 0x032119 (0xe13d bytes) 000074 1761283798.656093 MIRROR_EXTEND to 2 mirrors 000075 1761283798.893426 trunc from 0x03a000 to 0x007750 (0x328af bytes) 000076 1761283798.921043 mapwrite 0x03be4c thru 0x03ffff (0x41b4 bytes) 000077 1761283798.947576 mapwrite 0x00da68 thru 0x0157f0 (0x7d89 bytes) 000078 1761283798.961904 mapread 0x015d47 thru 0x017b8a (0x1e44 bytes) 000079 1761283798.966846 mapwrite 0x011026 thru 0x01b094 (0xa06f bytes) 000080 1761283798.976488 trunc from 0x040000 to 0x031ff9 (0xe006 bytes) 000082 1761283798.985808 write 0x022ada thru 0x02b78d (0x8cb4 bytes) 000083 1761283798.993181 read 0x013ef8 thru 0x016996 (0x2a9f bytes) 000084 1761283798.993852 punch from 0x021dc7 to 0x0249aa (0x2be4 bytes) 000085 1761283799.000294 mapwrite 0x021dd9 thru 0x022e5a (0x1082 bytes) 000086 1761283799.009880 read 0x008a30 thru 0x0124b6 (0x9a87 bytes) 000087 1761283799.010576 trunc from 0x017c22 to 0x01d45e (0x583d bytes) 000088 1761283799.013809 punch from 0x008c2f to 0x00c992 (0x3d64 bytes) 000089 1761283799.016786 write 0x03abb1 thru 0x03ffff (0x544f bytes) 000091 1761283799.018162 punch from 0x00b659 to 0x01b066 (0xfa0e bytes) 000092 1761283799.021151 write 0x03b6d5 thru 0x03d3f5 (0x1d21 bytes) 000093 1761283799.021978 trunc from 0x040000 to 0x02ed1d (0x112e2 bytes) 000094 1761283799.030794 fallocate 0x021f7e thru 0x028dca (0x6e4d bytes) 000095 1761283799.034349 punch from 0x0042cb to 0x0049fc (0x732 bytes) 000097 1761283799.037606 fallocate 0x016b7b thru 0x01899a (0x1e20 bytes) 000099 1761283799.041661 punch from 0x01f336 to 0x02b8d2 (0xc59d bytes) 000100 1761283799.045458 mapread 0x00057a thru 0x005101 (0x4b88 bytes) truncating to largest ever: 0x3fec7 000101 1761283799.052520 trunc from 0x02ed1e to 0x03fec6 (0x111a9 bytes) 000102 1761283799.060978 read 0x0100fc thru 0x019bc9 (0x9ace bytes) 000103 1761283799.061666 mapwrite 0x01bf41 thru 0x02032b (0x43eb bytes) 000104 1761283799.071085 mapwrite 0x00d594 thru 0x015006 (0x7a73 bytes) 000105 1761283799.080663 mapread 0x008013 thru 0x0088ec (0x8da bytes) 000106 1761283799.081490 read 0x025690 thru 0x02b755 (0x60c6 bytes) 000107 1761283799.082070 read 0x012497 thru 0x01bbf6 (0x9760 bytes) 000108 1761283799.082617 trunc from 0x03fec7 to 0x011fc9 (0x2defd bytes) 000109 1761283799.091102 mapread 0x00b45a thru 0x011fc9 (0x6b70 bytes) 000110 1761283799.092038 punch from 0x00863f to 0x00d19f (0x4b61 bytes) 000111 1761283799.096442 mapread 0x0087a0 thru 0x011fc9 (0x982a bytes) fallocating to largest ever: 0x40000 000112 1761283799.105145 fallocate 0x037f61 thru 0x03ffff (0x809f bytes) 000113 1761283799.110225 write 0x029932 thru 0x02d615 (0x3ce4 bytes) 000114 1761283799.111344 mapread 0x0273a4 thru 0x02d615 (0x6272 bytes) 000115 1761283799.116866 write 0x01952f thru 0x02852c (0xeffe bytes) 000116 1761283799.118094 mapread 0x01c13f thru 0x02542c (0x92ee bytes) 000117 1761283799.118953 mapread 0x02b934 thru 0x02d615 (0x1ce2 bytes) 000118 1761283799.119768 punch from 0x02d3e7 to 0x02d615 (0x22f bytes) 000119 1761283799.127465 fallocate 0x022885 thru 0x02d804 (0xaf80 bytes) 000120 1761283799.130642 fallocate 0x0345a3 thru 0x03c41d (0x7e7b bytes) 000121 1761283799.134052 fallocate 0x033450 thru 0x03d647 (0xa1f8 bytes) 000122 1761283799.137680 write 0x0220a4 thru 0x029c35 (0x7b92 bytes) 000123 1761283799.138899 punch from 0x004225 to 0x00e8d4 (0xa6b0 bytes) 000124 1761283799.150104 MIRROR_RESYNC 2 mirrors 000125 1761283799.455657 trunc from 0x03d648 to 0x003424 (0x3a223 bytes) 000127 1761283799.486295 fallocate 0x006cba thru 0x00f279 (0x85c0 bytes) 000128 1761283799.489507 fallocate 0x01cb61 thru 0x024b1b (0x7fbb bytes) 000129 1761283799.492389 trunc from 0x024b1c to 0x03a47b (0x15960 bytes) 000130 1761283799.501317 mapwrite 0x02729f thru 0x0275f1 (0x353 bytes) 000131 1761283799.511804 trunc from 0x03a47c to 0x012ebd (0x275be bytes) 000132 1761283799.520560 fallocate 0x0203db thru 0x02936a (0x8f90 bytes) 000133 1761283799.524136 trunc from 0x00edd7 to 0x012ebd (0x40e7 bytes) 000134 1761283799.529679 MIRROR_RESYNC 2 mirrors 000135 1761283799.748325 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000137 1761283799.801577 mapwrite 0x016316 thru 0x01e8de (0x85c9 bytes) 000138 1761283799.835653 punch from 0x001ee4 to 0x0112db (0xf3f8 bytes) 000140 1761283799.838663 mapread 0x01784b thru 0x01a941 (0x30f7 bytes) 000141 1761283799.839470 write 0x039924 thru 0x03ffff (0x66dc bytes) 000142 1761283799.840520 mapread 0x011819 thru 0x017bff (0x63e7 bytes) 000143 1761283799.844970 trunc from 0x0296d9 to 0x0313e0 (0x7d08 bytes) 000146 1761283799.848862 write 0x028bda thru 0x037968 (0xed8f bytes) 000147 1761283799.856262 read 0x019762 thru 0x01df53 (0x47f2 bytes) 000149 1761283799.857355 trunc from 0x03a6fd to 0x03e30a (0x3c0e bytes) 000151 1761283799.864456 trunc from 0x040000 to 0x02535c (0x1aca3 bytes) 000153 1761283799.871577 mapread 0x024195 thru 0x02535c (0x11c8 bytes) 000154 1761283799.872401 punch from 0x00c68a to 0x00eeac (0x2823 bytes) 000155 1761283799.875579 trunc from 0x02535d to 0x010c33 (0x14729 bytes) 000156 1761283799.881653 read 0x008a44 thru 0x010c33 (0x81f0 bytes) 000157 1761283799.885409 fallocate 0x01879a thru 0x0222ae (0x9b15 bytes) 000158 1761283799.887758 trunc from 0x0222af to 0x00c308 (0x15fa6 bytes) 000159 1761283799.893526 fallocate 0x00c2e3 thru 0x01bf5c (0xfc7a bytes) 000160 1761283799.896028 trunc from 0x00456e to 0x00bc1a (0x76ad bytes) 000161 1761283799.898759 write 0x01a918 thru 0x01c7da (0x1ec3 bytes) 000162 1761283799.902614 trunc from 0x009a1e to 0x012699 (0x8c7c bytes) 000163 1761283799.904898 mapwrite 0x0044ea thru 0x013a43 (0xf55a bytes) 000164 1761283799.946384 write 0x005b25 thru 0x00cd5f (0x723b bytes) 000165 1761283799.947455 mapread 0x018b36 thru 0x01c7da (0x3ca5 bytes) 000166 1761283799.948106 mapwrite 0x03a730 thru 0x03ffff (0x58d0 bytes) 000167 1761283799.967419 mapread 0x03aab1 thru 0x03ffff (0x554f bytes) 000169 1761283799.968984 mapread 0x004712 thru 0x0092d4 (0x4bc3 bytes) 000170 1761283799.969797 punch from 0x030eca to 0x03d84a (0xc981 bytes) 000172 1761283799.973797 mapread 0x03a61c thru 0x03ffff (0x59e4 bytes) 000174 1761283799.982823 mapwrite 0x033c69 thru 0x03c751 (0x8ae9 bytes) 000176 1761283800.004955 punch from 0x03cbd9 to 0x03fef1 (0x3319 bytes) 000177 1761283800.007795 read 0x03c1c7 thru 0x03d928 (0x1762 bytes) 000178 1761283800.012613 fallocate 0x036bc4 thru 0x038e02 (0x223f bytes) 000180 1761283800.016261 mapread 0x00db73 thru 0x010784 (0x2c12 bytes) 000181 1761283800.016930 trunc from 0x02b334 to 0x038362 (0xd02f bytes) 000182 1761283800.019608 punch from 0x03c7fd to 0x03ffff (0x3803 bytes) 000183 1761283800.022146 fallocate 0x02eb04 thru 0x03e0d2 (0xf5cf bytes) 000184 1761283800.024737 trunc from 0x040000 to 0x03d15e (0x2ea1 bytes) 000185 1761283800.030259 mapwrite 0x0014b1 thru 0x00e232 (0xcd82 bytes) 000186 1761283800.040793 read 0x00a1ef thru 0x0104d8 (0x62ea bytes) 000187 1761283800.041292 write 0x009d97 thru 0x019d25 (0xff8f bytes) 000189 1761283800.042757 mapwrite 0x022e65 thru 0x02634b (0x34e7 bytes) 000190 1761283800.051025 write 0x01fd27 thru 0x026e06 (0x70e0 bytes) 000191 1761283800.052183 trunc from 0x03d15f to 0x039296 (0x3ec8 bytes) 000192 1761283800.060564 write 0x02f240 thru 0x034058 (0x4e19 bytes) 000193 1761283800.065965 write 0x002b2a thru 0x011cd4 (0xf1ab bytes) 000194 1761283800.067407 punch from 0x00d9c2 to 0x014a28 (0x7067 bytes) 000195 1761283800.076447 trunc from 0x00a060 to 0x0170e5 (0xd086 bytes) 000196 1761283800.079519 trunc from 0x024d35 to 0x032408 (0xd6d4 bytes) 000197 1761283800.082407 punch from 0x003c23 to 0x006553 (0x2931 bytes) 000198 1761283800.085469 fallocate 0x03e34b thru 0x03ffff (0x1cb5 bytes) 000199 1761283800.088029 mapread 0x0091a2 thru 0x010353 (0x71b2 bytes) 000201 1761283800.101878 fallocate 0x02b6da thru 0x0359f2 (0xa319 bytes) 000202 1761283800.104680 trunc from 0x040000 to 0x00d16d (0x32e92 bytes) 000204 1761283800.115290 MIRROR_EXTEND to 2 mirrors 000205 1761283800.343858 trunc from 0x003810 to 0x004994 (0x1185 bytes) 000206 1761283800.355605 punch from 0x00cafd to 0x00d16d (0x671 bytes) 000207 1761283800.357670 trunc from 0x00d16e to 0x010ddd (0x3c70 bytes) 000208 1761283800.371882 fallocate 0x009e76 thru 0x01903a (0xf1c5 bytes) 000210 1761283800.374995 mapwrite 0x000502 thru 0x009070 (0x8b6f bytes) 000211 1761283800.392551 punch from 0x00fdf0 to 0x010ddd (0xfee bytes) 000212 1761283800.394989 read 0x008977 thru 0x010ddd (0x8467 bytes) 000213 1761283800.397916 punch from 0x00bc09 to 0x010ddd (0x51d5 bytes) 000214 1761283800.401138 read 0x00d86f thru 0x010ddd (0x356f bytes) 000215 1761283800.408350 MIRROR_EXTEND to 3 mirrors 000216 1761283800.620626 mapwrite 0x02d323 thru 0x02f175 (0x1e53 bytes) 000217 1761283800.652766 write 0x017a49 thru 0x022aba (0xb072 bytes) 000218 1761283800.658094 mapread 0x027f14 thru 0x02f175 (0x7262 bytes) 000219 1761283800.661639 punch from 0x016596 to 0x01e283 (0x7cee bytes) 000220 1761283800.670077 MIRROR_RESYNC 3 mirrors 000223 1761283800.917415 trunc from 0x002426 to 0x00bb23 (0x96fe bytes) 000224 1761283800.929525 fallocate 0x01fc93 thru 0x024174 (0x44e2 bytes) 000225 1761283800.932115 trunc from 0x002832 to 0x003d07 (0x14d6 bytes) 000226 1761283800.934971 write 0x03b2ba thru 0x03ffff (0x4d46 bytes) 000227 1761283800.936104 trunc from 0x006a14 to 0x00754f (0xb3c bytes) 000228 1761283800.938465 trunc from 0x040000 to 0x02d1fc (0x12e03 bytes) 000229 1761283800.953079 trunc from 0x006062 to 0x010ea6 (0xae45 bytes) 000230 1761283800.955173 read 0x005bc1 thru 0x00836d (0x27ad bytes) 000231 1761283800.958216 fallocate 0x0023a5 thru 0x00e9aa (0xc606 bytes) 000232 1761283800.961404 punch from 0x029a9b to 0x02d1fc (0x3762 bytes) 000233 1761283800.964125 fallocate 0x01d919 thru 0x02bcf3 (0xe3db bytes) 000234 1761283800.966854 trunc from 0x02a69d to 0x02d1fc (0x2b60 bytes) 000235 1761283800.969474 read 0x020e05 thru 0x026309 (0x5505 bytes) 000236 1761283800.973800 read 0x01167d thru 0x018709 (0x708d bytes) 000238 1761283800.974895 trunc from 0x02d1fd to 0x0082bf (0x24f3d bytes) 000239 1761283800.980979 mapwrite 0x00cb57 thru 0x011870 (0x4d1a bytes) 000240 1761283800.998628 trunc from 0x00fea2 to 0x011870 (0x19cf bytes) 000241 1761283801.001323 fallocate 0x01ef79 thru 0x02b53f (0xc5c7 bytes) 000242 1761283801.004021 fallocate 0x00ccbe thru 0x01b942 (0xec85 bytes) 000243 1761283801.006464 read 0x00b8b8 thru 0x011870 (0x5fb9 bytes) 000244 1761283801.009032 punch from 0x00062a to 0x00e8b9 (0xe290 bytes) 000245 1761283801.011826 read 0x0025c5 thru 0x00aa76 (0x84b2 bytes) 000246 1761283801.016271 trunc from 0x011871 to 0x0308fc (0x1f08c bytes) 000247 1761283801.023550 mapread 0x01519f thru 0x01e2ec (0x914e bytes) 000248 1761283801.029201 mapread 0x016fd4 thru 0x022c26 (0xbc53 bytes) 000249 1761283801.030113 mapread 0x01d66b thru 0x023a2a (0x63c0 bytes) 000250 1761283801.030898 read 0x005aa0 thru 0x010e9c (0xb3fd bytes) 000251 1761283801.031353 fallocate 0x007c4d thru 0x0130b2 (0xb466 bytes) 000252 1761283801.034542 read 0x01b816 thru 0x029d72 (0xe55d bytes) 000253 1761283801.035174 read 0x022a8f thru 0x02be9d (0x940f bytes) 000254 1761283801.035757 mapwrite 0x019b4f thru 0x01b9b4 (0x1e66 bytes) 000255 1761283801.043260 punch from 0x019b98 to 0x0225c8 (0x8a31 bytes) 000256 1761283801.046397 mapwrite 0x02afe4 thru 0x03967a (0xe697 bytes) 000257 1761283801.065100 mapread 0x03091a thru 0x039032 (0x8719 bytes) 000258 1761283801.066053 fallocate 0x036142 thru 0x03ffff (0x9ebe bytes) 000259 1761283801.069908 MIRROR_RESYNC 3 mirrors 000260 1761283801.374388 mapread 0x03da09 thru 0x03ffff (0x25f7 bytes) 000261 1761283801.379442 write 0x02797b thru 0x02a3a4 (0x2a2a bytes) 000262 1761283801.392232 read 0x03d7ff thru 0x03da5d (0x25f bytes) 000263 1761283801.394753 read 0x036228 thru 0x03ffff (0x9dd8 bytes) 000264 1761283801.399019 write 0x00834c thru 0x012da2 (0xaa57 bytes) 000265 1761283801.400276 trunc from 0x040000 to 0x020fb1 (0x1f04e bytes) 000266 1761283801.414436 mapread 0x00a615 thru 0x00b99d (0x1389 bytes) 000267 1761283801.415249 fallocate 0x036edc thru 0x03ffff (0x9124 bytes) 000268 1761283801.417972 fallocate 0x00d490 thru 0x01a91e (0xd48f bytes) 000269 1761283801.420539 trunc from 0x040000 to 0x02f317 (0x10ce8 bytes) 000270 1761283801.426239 punch from 0x00fd9f to 0x011b8a (0x1dec bytes) 000271 1761283801.431993 trunc from 0x026ea9 to 0x02f317 (0x846f bytes) 000272 1761283801.434192 mapread 0x02ee44 thru 0x02f317 (0x4d4 bytes) 000273 1761283801.439050 punch from 0x0163aa to 0x01dfae (0x7c05 bytes) 000274 1761283801.441889 read 0x025714 thru 0x028ab6 (0x33a3 bytes) 000275 1761283801.446331 read 0x02ab37 thru 0x02f317 (0x47e1 bytes) 000276 1761283801.446796 mapread 0x01d631 thru 0x02b4dd (0xdead bytes) 000277 1761283801.447579 trunc from 0x014db3 to 0x023208 (0xe456 bytes) 000278 1761283801.450957 fallocate 0x00c0fb thru 0x01bbf4 (0xfafa bytes) 000279 1761283801.453556 trunc from 0x021429 to 0x029345 (0x7f1d bytes) 000280 1761283801.457854 MIRROR_EXTEND to 4 mirrors 000281 1761283801.664799 MIRROR_EXTEND to 5 mirrors 000282 1761283801.879107 fallocate 0x033de8 thru 0x034bd1 (0xdea bytes) 000283 1761283801.893299 mapread 0x009361 thru 0x016312 (0xcfb2 bytes) 000284 1761283801.902351 write 0x02907f thru 0x02d6dd (0x465f bytes) 000285 1761283801.908047 read 0x0187a8 thru 0x01f8c7 (0x7120 bytes) 000287 1761283801.911078 mapread 0x0009d4 thru 0x0074f3 (0x6b20 bytes) 000288 1761283801.914020 fallocate 0x03e2de thru 0x03ffff (0x1d22 bytes) 000289 1761283801.916571 mapread 0x001775 thru 0x011682 (0xff0e bytes) 000291 1761283801.917949 mapwrite 0x002fcf thru 0x010aba (0xdaec bytes) 000292 1761283801.928474 trunc from 0x027d1b to 0x032980 (0xac66 bytes) 000293 1761283801.930977 fallocate 0x01171f thru 0x011980 (0x262 bytes) 000294 1761283801.932860 punch from 0x0265d9 to 0x02da61 (0x7489 bytes) 000295 1761283801.936289 MIRROR_RESYNC 5 mirrors 000297 1761283802.300394 mapread 0x006931 thru 0x00ed04 (0x83d4 bytes) 000298 1761283802.310909 mapread 0x031146 thru 0x034bd1 (0x3a8c bytes) 000299 1761283802.315478 read 0x02885c thru 0x034bd1 (0xc376 bytes) 000300 1761283802.318756 write 0x0037aa thru 0x005d66 (0x25bd bytes) 000302 1761283802.337285 trunc from 0x02bff9 to 0x034bd1 (0x8bd9 bytes) 000303 1761283802.340461 mapread 0x0026bf thru 0x008f9e (0x68e0 bytes) 000304 1761283802.344063 MIRROR_EXTEND to 6 mirrors 000305 1761283802.591229 mapwrite 0x001148 thru 0x006ccf (0x5b88 bytes) 000306 1761283802.639616 read 0x003cf8 thru 0x007935 (0x3c3e bytes) 000307 1761283802.640149 punch from 0x015256 to 0x017fa2 (0x2d4d bytes) 000308 1761283802.643038 trunc from 0x025e41 to 0x031212 (0xb3d2 bytes) 000309 1761283802.645684 trunc from 0x01f0da to 0x02165d (0x2584 bytes) 000310 1761283802.648393 mapwrite 0x01c63f thru 0x01fc65 (0x3627 bytes) 000311 1761283802.660160 trunc from 0x034bd2 to 0x0086da (0x2c4f7 bytes) 000312 1761283802.678943 mapread 0x0074ff thru 0x0086da (0x11dc bytes) 000313 1761283802.679898 read 0x00274c thru 0x0086da (0x5f8f bytes) 000314 1761283802.680500 write 0x010841 thru 0x0138da (0x309a bytes) 000315 1761283802.681566 mapwrite 0x03e05e thru 0x03ffff (0x1fa2 bytes) 000316 1761283802.704219 punch from 0x021f28 to 0x02d59c (0xb675 bytes) 000318 1761283802.708193 write 0x01b223 thru 0x029331 (0xe10f bytes) 000319 1761283802.713039 mapwrite 0x01325c thru 0x014968 (0x170d bytes) 000320 1761283802.721774 write 0x031225 thru 0x0353b8 (0x4194 bytes) 000321 1761283802.722670 mapread 0x01e099 thru 0x02a94d (0xc8b5 bytes) 000322 1761283802.725778 write 0x02fd66 thru 0x0363b4 (0x664f bytes) 000324 1761283802.727337 fallocate 0x023ad5 thru 0x032f0b (0xf437 bytes) 000325 1761283802.730240 mapwrite 0x0275f2 thru 0x02c23a (0x4c49 bytes) 000326 1761283802.745321 trunc from 0x03c914 to 0x03ffff (0x36ec bytes) 000327 1761283802.748767 mapwrite 0x0205a3 thru 0x02df6c (0xd9ca bytes) 000328 1761283802.762577 trunc from 0x0017e9 to 0x006b75 (0x538d bytes) 000329 1761283802.765480 mapwrite 0x034226 thru 0x038846 (0x4621 bytes) 000330 1761283802.774764 punch from 0x03e9bd to 0x03ffff (0x1643 bytes) 000331 1761283802.777308 read 0x003292 thru 0x009b6a (0x68d9 bytes) 000332 1761283802.783536 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000333 1761283802.870264 MIRROR_EXTEND to 6 mirrors 000334 1761283803.196840 mapwrite 0x020433 thru 0x02fac5 (0xf693 bytes) 000335 1761283803.267767 write 0x01a8a9 thru 0x0273d8 (0xcb30 bytes) 000336 1761283803.271467 fallocate 0x03f835 thru 0x03ffff (0x7cb bytes) 000337 1761283803.274599 trunc from 0x009f10 to 0x01621c (0xc30d bytes) 000338 1761283803.277471 punch from 0x026d71 to 0x0293c1 (0x2651 bytes) 000340 1761283803.286217 fallocate 0x02075e thru 0x02f601 (0xeea4 bytes) 000341 1761283803.289056 mapread 0x03ef9d thru 0x03ffff (0x1063 bytes) 000342 1761283803.297088 MIRROR_RESYNC 6 mirrors 000343 1761283803.853433 mapwrite 0x0371f7 thru 0x03ffff (0x8e09 bytes) 000344 1761283803.900215 fallocate 0x014bb1 thru 0x01607e (0x14ce bytes) 000346 1761283803.903002 mapread 0x0018b8 thru 0x00375c (0x1ea5 bytes) 000347 1761283803.907049 trunc from 0x03f424 to 0x03ffff (0xbdc bytes) 000348 1761283803.909919 mapread 0x025a6d thru 0x027a69 (0x1ffd bytes) 000349 1761283803.914474 fallocate 0x033593 thru 0x03bfbb (0x8a29 bytes) 000350 1761283803.917130 trunc from 0x040000 to 0x0114bd (0x2eb42 bytes) 000352 1761283803.933669 write 0x02236c thru 0x02ec3b (0xc8d0 bytes) 000353 1761283803.935051 read 0x005987 thru 0x00e8cc (0x8f46 bytes) 000354 1761283803.935609 mapwrite 0x0178ce thru 0x019b67 (0x229a bytes) 000355 1761283803.946897 mapwrite 0x016c43 thru 0x020c7e (0xa03c bytes) 000356 1761283803.958095 punch from 0x004624 to 0x0096fe (0x50db bytes) 000357 1761283803.960782 trunc from 0x01207e to 0x01db1c (0xba9f bytes) 000358 1761283803.963169 write 0x012f87 thru 0x01bc08 (0x8c82 bytes) 000359 1761283803.968640 trunc from 0x02ec3c to 0x01f6c6 (0xf575 bytes) 000360 1761283803.974675 mapread 0x005933 thru 0x005c7e (0x34c bytes) 000361 1761283803.980137 MIRROR_RESYNC 6 mirrors 000362 1761283804.292311 read 0x00e8e0 thru 0x01b911 (0xd032 bytes) 000363 1761283804.296369 read 0x01ba16 thru 0x01f6c6 (0x3cb1 bytes) 000364 1761283804.299448 punch from 0x00e69f to 0x00f721 (0x1083 bytes) 000365 1761283804.312258 trunc from 0x01e699 to 0x01eb76 (0x4de bytes) 000366 1761283804.314785 fallocate 0x016024 thru 0x019142 (0x311f bytes) 000367 1761283804.317371 read 0x0161fe thru 0x01900c (0x2e0f bytes) 000368 1761283804.321547 trunc from 0x01f6c7 to 0x03855f (0x18e99 bytes) 000370 1761283804.328950 mapwrite 0x01f67a thru 0x026cd9 (0x7660 bytes) 000371 1761283804.339296 write 0x0317b0 thru 0x03ffff (0xe850 bytes) 000372 1761283804.340477 read 0x01df05 thru 0x022306 (0x4402 bytes) 000373 1761283804.340865 fallocate 0x019fbd thru 0x0240ba (0xa0fe bytes) 000374 1761283804.343320 trunc from 0x0107ad to 0x012523 (0x1d77 bytes) 000375 1761283804.345580 mapwrite 0x010362 thru 0x013ec5 (0x3b64 bytes) 000376 1761283804.354277 trunc from 0x00dbf2 to 0x00f826 (0x1c35 bytes) 000377 1761283804.356292 punch from 0x00a46e to 0x0142d0 (0x9e63 bytes) 000378 1761283804.358575 punch from 0x01a633 to 0x01c97f (0x234d bytes) 000379 1761283804.360508 write 0x02ec73 thru 0x03a5ae (0xb93c bytes) 000380 1761283804.361872 read 0x0370f7 thru 0x03ffff (0x8f09 bytes) 000381 1761283804.362400 mapwrite 0x011bad thru 0x01bf27 (0xa37b bytes) 000382 1761283804.375176 write 0x02c5bd thru 0x0355b4 (0x8ff8 bytes) 000383 1761283804.376271 mapread 0x008ae4 thru 0x012f22 (0xa43f bytes) 000384 1761283804.377162 trunc from 0x040000 to 0x00a9f0 (0x3560f bytes) 000385 1761283804.384963 mapread 0x005c5e thru 0x00968e (0x3a31 bytes) 000386 1761283804.385911 write 0x016143 thru 0x01b09d (0x4f5b bytes) 000387 1761283804.386990 mapwrite 0x00cfa5 thru 0x00f5f4 (0x2650 bytes) 000389 1761283804.403632 fallocate 0x002095 thru 0x008ccd (0x6c39 bytes) 000390 1761283804.406451 mapread 0x0180fc thru 0x01b09d (0x2fa2 bytes) 000391 1761283804.407247 fallocate 0x00c0b5 thru 0x00e6e1 (0x262d bytes) 000392 1761283804.410438 punch from 0x019989 to 0x01b09d (0x1715 bytes) 000393 1761283804.413772 trunc from 0x009976 to 0x0174d7 (0xdb62 bytes) 000395 1761283804.417469 fallocate 0x01f826 thru 0x02b6f0 (0xbecb bytes) 000396 1761283804.420276 mapread 0x007e48 thru 0x0123c8 (0xa581 bytes) 000398 1761283804.442393 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000399 1761283804.523399 read 0x005214 thru 0x00d81c (0x8609 bytes) 000400 1761283804.527544 write 0x031336 thru 0x03242f (0x10fa bytes) 000401 1761283804.533033 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000402 1761283804.603779 mapwrite 0x007071 thru 0x00f2b7 (0x8247 bytes) 000404 1761283804.617795 trunc from 0x00828a to 0x013766 (0xb4dd bytes) 000405 1761283804.620723 read 0x023497 thru 0x02a950 (0x74ba bytes) 000406 1761283804.625920 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000407 1761283804.690692 mapwrite 0x02080e thru 0x0291ad (0x89a0 bytes) 000408 1761283804.704853 punch from 0x00d84e to 0x01600b (0x87be bytes) 000409 1761283804.706826 trunc from 0x0292da to 0x031c61 (0x8988 bytes) 000410 1761283804.708934 read 0x00a1f6 thru 0x0166ec (0xc4f7 bytes) 000411 1761283804.712732 punch from 0x032355 to 0x03242f (0xdb bytes) 000412 1761283804.714980 fallocate 0x02be30 thru 0x02f45b (0x362c bytes) 000413 1761283804.717289 trunc from 0x032430 to 0x00c101 (0x2632e bytes) 000414 1761283804.725590 write 0x0209e6 thru 0x029c1c (0x9237 bytes) 000415 1761283804.726848 read 0x014077 thru 0x0176c9 (0x3653 bytes) 000416 1761283804.730801 mapwrite 0x03db91 thru 0x03ffff (0x246f bytes) 000419 1761283804.746967 trunc from 0x02d21a to 0x038ead (0xbc94 bytes) 000420 1761283804.749445 trunc from 0x040000 to 0x039535 (0x6aca bytes) 000421 1761283804.755343 fallocate 0x020881 thru 0x023a82 (0x3202 bytes) 000422 1761283804.757813 punch from 0x006701 to 0x0136c0 (0xcfc0 bytes) 000423 1761283804.760275 mapwrite 0x038157 thru 0x03ffff (0x7ea9 bytes) 000424 1761283804.790153 punch from 0x030b08 to 0x039743 (0x8c3c bytes) 000425 1761283804.792900 read 0x0283ce thru 0x0288b3 (0x4e6 bytes) 000426 1761283804.793477 write 0x036f5d thru 0x03ffff (0x90a3 bytes) 000428 1761283804.798998 trunc from 0x040000 to 0x01398a (0x2c675 bytes) 000430 1761283804.806392 trunc from 0x00615c to 0x00bb6d (0x5a12 bytes) 000431 1761283804.809065 write 0x00f409 thru 0x019f88 (0xab80 bytes) 000432 1761283804.810283 fallocate 0x00cb40 thru 0x01756b (0xaa2c bytes) 000433 1761283804.812781 trunc from 0x010b24 to 0x019f88 (0x9465 bytes) 000434 1761283804.818218 trunc from 0x008e7a to 0x018da1 (0xff28 bytes) 000435 1761283804.820632 mapwrite 0x02fd5f thru 0x031a64 (0x1d06 bytes) 000436 1761283804.841454 fallocate 0x03cc3d thru 0x03ffff (0x33c3 bytes) 000437 1761283804.845572 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000438 1761283804.912950 mapwrite 0x020ad1 thru 0x024c32 (0x4162 bytes) 000439 1761283804.928991 MIRROR_RESYNC 2 mirrors 000441 1761283805.174942 MIRROR_EXTEND to 3 mirrors 000443 1761283805.398025 fallocate 0x023d16 thru 0x02e01f (0xa30a bytes) 000444 1761283805.411241 read 0x03ab0c thru 0x03b264 (0x759 bytes) 000445 1761283805.413983 write 0x01e55d thru 0x023b45 (0x55e9 bytes) 000446 1761283805.419840 trunc from 0x03f7b9 to 0x03ffff (0x847 bytes) 000447 1761283805.422433 read 0x03421e thru 0x03f13d (0xaf20 bytes) 000448 1761283805.427687 mapread 0x03bbff thru 0x03ffff (0x4401 bytes) 000449 1761283805.428490 mapwrite 0x016754 thru 0x0175c0 (0xe6d bytes) 000450 1761283805.437098 mapwrite 0x01fa19 thru 0x0221f9 (0x27e1 bytes) 000451 1761283805.444502 trunc from 0x00d9df to 0x00ede3 (0x1405 bytes) 000452 1761283805.447263 fallocate 0x00a6d7 thru 0x010914 (0x623e bytes) 000453 1761283805.449842 punch from 0x0075e0 to 0x00982d (0x224e bytes) 000454 1761283805.452153 write 0x03dba0 thru 0x03ffff (0x2460 bytes) 000455 1761283805.452943 read 0x03ff7f thru 0x03ffff (0x81 bytes) 000456 1761283805.453420 trunc from 0x03c422 to 0x03ffff (0x3bde bytes) 000457 1761283805.459529 trunc from 0x040000 to 0x008a70 (0x3758f bytes) 000458 1761283805.468606 fallocate 0x00a65f thru 0x01a007 (0xf9a9 bytes) 000459 1761283805.470876 write 0x0209ce thru 0x028758 (0x7d8b bytes) 000460 1761283805.472205 read 0x013b65 thru 0x01a73a (0x6bd6 bytes) 000461 1761283805.476819 punch from 0x01cb2b to 0x028758 (0xbc2e bytes) 000462 1761283805.482850 punch from 0x017c7c to 0x0240f6 (0xc47b bytes) 000463 1761283805.485848 trunc from 0x028759 to 0x03d4b7 (0x14d5f bytes) 000464 1761283805.492868 trunc from 0x028432 to 0x0305cd (0x819c bytes) 000465 1761283805.495529 punch from 0x006671 to 0x00b8cc (0x525c bytes) 000466 1761283805.498595 write 0x02c438 thru 0x031c92 (0x585b bytes) 000468 1761283805.504070 mapread 0x0393bf thru 0x03d4b7 (0x40f9 bytes) 000469 1761283805.505021 trunc from 0x03d4b8 to 0x03a0ec (0x33cb bytes) 000470 1761283805.512535 fallocate 0x01a025 thru 0x024c67 (0xac43 bytes) 000471 1761283805.515591 punch from 0x0319a8 to 0x03a0ec (0x8745 bytes) 000472 1761283805.521763 read 0x027a61 thru 0x02d6d1 (0x5c71 bytes) 000473 1761283805.529150 MIRROR_EXTEND to 4 mirrors 000474 1761283805.738507 mapwrite 0x0378e9 thru 0x03ffff (0x8717 bytes) 000475 1761283805.775614 write 0x019d82 thru 0x01ae63 (0x10e2 bytes) 000476 1761283805.779871 mapwrite 0x0301b2 thru 0x03273d (0x258c bytes) 000477 1761283805.792148 read 0x034a49 thru 0x03e14e (0x9706 bytes) 000479 1761283805.795415 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000480 1761283805.880045 punch from 0x03a6c8 to 0x03ffff (0x5938 bytes) 000481 1761283805.882376 mapwrite 0x024421 thru 0x02bb9b (0x777b bytes) 000482 1761283805.894994 punch from 0x00cf5f to 0x014099 (0x713b bytes) 000483 1761283805.896982 punch from 0x009bf7 to 0x00f8ec (0x5cf6 bytes) 000484 1761283805.898729 mapread 0x00b25a thru 0x01ae3c (0xfbe3 bytes) 000485 1761283805.905164 mapread 0x0394bc thru 0x03d22d (0x3d72 bytes) 000486 1761283805.909154 trunc from 0x0317c9 to 0x037184 (0x59bc bytes) 000487 1761283805.911184 mapwrite 0x03672a thru 0x0370b2 (0x989 bytes) 000488 1761283805.917835 mapread 0x0027a0 thru 0x0114f7 (0xed58 bytes) 000489 1761283805.919863 write 0x0170c5 thru 0x021a4c (0xa988 bytes) 000490 1761283805.922304 fallocate 0x03ae96 thru 0x03ffff (0x516a bytes) 000491 1761283805.924581 mapread 0x02ccca thru 0x02e79e (0x1ad5 bytes) 000492 1761283805.925266 mapwrite 0x02a0b8 thru 0x037416 (0xd35f bytes) 000494 1761283805.935140 trunc from 0x01ab79 to 0x01de56 (0x32de bytes) 000495 1761283805.937240 trunc from 0x040000 to 0x03b651 (0x49ae bytes) 000496 1761283805.941643 trunc from 0x034290 to 0x03b651 (0x73c2 bytes) 000497 1761283805.943242 write 0x033e38 thru 0x03ffff (0xc1c8 bytes) 000498 1761283805.944153 trunc from 0x015140 to 0x01946e (0x432f bytes) 000499 1761283805.945864 write 0x02f367 thru 0x03e471 (0xf10b bytes) 000500 1761283805.947047 mapwrite 0x006a4b thru 0x014c2c (0xe1e2 bytes) 000501 1761283805.956418 write 0x0178df thru 0x023c5f (0xc381 bytes) 000502 1761283805.960488 mapwrite 0x01171a thru 0x014e40 (0x3727 bytes) 000503 1761283805.968000 mapwrite 0x01ed03 thru 0x021399 (0x2697 bytes) 000504 1761283805.974199 punch from 0x01b602 to 0x025691 (0xa090 bytes) 000505 1761283805.977147 trunc from 0x040000 to 0x03c0e9 (0x3f16 bytes) 000506 1761283805.983099 trunc from 0x00827c to 0x017c94 (0xfa19 bytes) 000507 1761283805.985833 trunc from 0x03c0ea to 0x03c5ea (0x501 bytes) 000508 1761283805.991949 mapwrite 0x01e32a thru 0x0211e9 (0x2ec0 bytes) 000509 1761283806.001088 fallocate 0x03d27b thru 0x03faed (0x2873 bytes) 000510 1761283806.003228 write 0x00e45c thru 0x00fb45 (0x16ea bytes) 000512 1761283806.007090 trunc from 0x03c5eb to 0x0362f0 (0x62fa bytes) 000513 1761283806.012239 mapwrite 0x0234fd thru 0x03334c (0xfe50 bytes) 000514 1761283806.020520 read 0x032622 thru 0x0362f0 (0x3ccf bytes) 000515 1761283806.021115 mapread 0x011c55 thru 0x01acc4 (0x9070 bytes) 000516 1761283806.021913 read 0x00e379 thru 0x010e08 (0x2a90 bytes) 000517 1761283806.024342 MIRROR_RESYNC 3 mirrors 000518 1761283806.369716 trunc from 0x0138de to 0x021a6e (0xe191 bytes) 000519 1761283806.380883 read 0x0229ed thru 0x027aa2 (0x50b6 bytes) 000520 1761283806.384374 trunc from 0x01364a to 0x019b9c (0x6553 bytes) 000521 1761283806.386856 trunc from 0x0362f1 to 0x02aa5b (0xb895 bytes) 000522 1761283806.399788 mapread 0x020071 thru 0x02560e (0x559e bytes) 000523 1761283806.404546 punch from 0x022a1a to 0x02aa5b (0x8042 bytes) 000524 1761283806.407358 punch from 0x001822 to 0x00f099 (0xd878 bytes) 000525 1761283806.410218 fallocate 0x00bdf7 thru 0x011388 (0x5592 bytes) 000526 1761283806.414748 MIRROR_EXTEND to 4 mirrors 000527 1761283806.635159 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000528 1761283806.744038 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000529 1761283806.872174 mapread 0x01d30e thru 0x024014 (0x6d07 bytes) 000530 1761283806.887795 write 0x03b554 thru 0x03c7e7 (0x1294 bytes) 000531 1761283806.901694 read 0x02d9e1 thru 0x030b0d (0x312d bytes) 000532 1761283806.904302 trunc from 0x001cc9 to 0x005263 (0x359b bytes) 000534 1761283806.907603 punch from 0x01a59d to 0x01eb6b (0x45cf bytes) 000535 1761283806.909964 read 0x038ec8 thru 0x03b993 (0x2acc bytes) 000536 1761283806.917187 MIRROR_RESYNC 2 mirrors 000537 1761283807.165662 trunc from 0x03c7e8 to 0x03a147 (0x26a0 bytes) 000538 1761283807.180782 MIRROR_RESYNC 2 mirrors 000539 1761283807.437794 MIRROR_RESYNC 2 mirrors 000540 1761283807.524469 write 0x00add5 thru 0x016d85 (0xbfb1 bytes) 000541 1761283807.537494 mapread 0x01a2de thru 0x01f151 (0x4e74 bytes) 000543 1761283807.541199 trunc from 0x03a148 to 0x0294fe (0x10c49 bytes) 000544 1761283807.548002 trunc from 0x0102d3 to 0x0191f5 (0x8f23 bytes) 000545 1761283807.554820 MIRROR_EXTEND to 3 mirrors 000546 1761283807.789599 mapwrite 0x02aaad thru 0x032a52 (0x7fa6 bytes) 000547 1761283807.821982 punch from 0x00907b to 0x00d146 (0x40cc bytes) 000548 1761283807.826560 MIRROR_EXTEND to 4 mirrors 000549 1761283808.054849 MIRROR_RESYNC 4 mirrors 000550 1761283808.386384 trunc from 0x032a53 to 0x01a926 (0x1812c bytes) 000551 1761283808.412158 mapread 0x013d05 thru 0x0183b4 (0x46b0 bytes) 000552 1761283808.420343 write 0x03909d thru 0x03ffff (0x6f63 bytes) 000554 1761283808.421908 fallocate 0x004959 thru 0x00d189 (0x8831 bytes) 000555 1761283808.425210 mapread 0x00e2b6 thru 0x00f448 (0x1193 bytes) 000556 1761283808.426060 trunc from 0x040000 to 0x035886 (0xa779 bytes) 000557 1761283808.434805 trunc from 0x035887 to 0x021f12 (0x13974 bytes) 000558 1761283808.441169 trunc from 0x021f13 to 0x039cfc (0x17dea bytes) 000560 1761283808.448131 trunc from 0x00ef4d to 0x01cc8a (0xdd3e bytes) 000562 1761283808.452061 fallocate 0x02d8cf thru 0x02e7b7 (0xee9 bytes) 000563 1761283808.455218 fallocate 0x022644 thru 0x02aa5f (0x841c bytes) 000564 1761283808.459419 punch from 0x004cfe to 0x013bd0 (0xeed3 bytes) 000565 1761283808.462559 trunc from 0x020d6e to 0x022743 (0x19d6 bytes) 000566 1761283808.465134 read 0x014c45 thru 0x019847 (0x4c03 bytes) 000567 1761283808.469627 mapwrite 0x02e299 thru 0x039c51 (0xb9b9 bytes) 000569 1761283808.490218 trunc from 0x036721 to 0x039cfc (0x35dc bytes) 000570 1761283808.493756 read 0x005323 thru 0x00a77c (0x545a bytes) 000571 1761283808.501828 MIRROR_EXTEND to 5 mirrors 000572 1761283808.733395 read 0x03609b thru 0x039cfc (0x3c62 bytes) 000574 1761283808.736898 mapread 0x00e9cf thru 0x018f37 (0xa569 bytes) 000575 1761283808.746782 fallocate 0x037c54 thru 0x03ed33 (0x70e0 bytes) 000576 1761283808.763730 mapwrite 0x03c453 thru 0x03ffff (0x3bad bytes) 000577 1761283808.793760 punch from 0x038bcf to 0x03c004 (0x3436 bytes) 000578 1761283808.796952 punch from 0x032107 to 0x0344dd (0x23d7 bytes) 000579 1761283808.799598 trunc from 0x040000 to 0x012637 (0x2d9c8 bytes) 000580 1761283808.805920 trunc from 0x012638 to 0x011127 (0x1510 bytes) 000581 1761283808.812254 fallocate 0x00a07a thru 0x018af5 (0xea7c bytes) 000582 1761283808.816989 MIRROR_EXTEND to 6 mirrors 000583 1761283809.030162 mapwrite 0x01e896 thru 0x01f6c5 (0xe30 bytes) 000584 1761283809.065131 write 0x034984 thru 0x03ffff (0xb67c bytes) 000585 1761283809.066693 punch from 0x038e11 to 0x03927f (0x46f bytes) 000586 1761283809.073846 mapwrite 0x01e97b thru 0x027ce8 (0x936e bytes) 000587 1761283809.087636 trunc from 0x040000 to 0x0039a0 (0x3c65f bytes) 000588 1761283809.100155 MIRROR_EXTEND to 7 mirrors 000590 1761283809.339151 read 0x0010cd thru 0x0039a0 (0x28d4 bytes) 000591 1761283809.344274 write 0x00eb49 thru 0x01a731 (0xbbe9 bytes) 000593 1761283809.358001 trunc from 0x01a732 to 0x01ccd9 (0x25a8 bytes) 000595 1761283809.366309 trunc from 0x01ccda to 0x006b11 (0x161c8 bytes) 000596 1761283809.373380 trunc from 0x006b12 to 0x033870 (0x2cd5f bytes) 000597 1761283809.382583 MIRROR_EXTEND to 8 mirrors 000599 1761283809.626749 write 0x017798 thru 0x0255f8 (0xde61 bytes) 000600 1761283809.647041 trunc from 0x033871 to 0x01e593 (0x152dd bytes) 000601 1761283809.656072 mapwrite 0x02e397 thru 0x035800 (0x746a bytes) 000602 1761283809.676159 MIRROR_RESYNC 8 mirrors 000603 1761283810.186472 MIRROR_RESYNC 8 mirrors 000604 1761283810.309491 punch from 0x0183f2 to 0x01b59c (0x31ab bytes) 000605 1761283810.323814 trunc from 0x035801 to 0x028371 (0xd48f bytes) 000606 1761283810.332436 write 0x03a725 thru 0x03ffff (0x58db bytes) 000607 1761283810.333516 mapwrite 0x00d4b7 thru 0x019ea0 (0xc9ea bytes) 000608 1761283810.352958 write 0x00fba6 thru 0x01711d (0x7578 bytes) 000610 1761283810.354669 fallocate 0x016013 thru 0x018262 (0x2250 bytes) 000611 1761283810.357241 mapread 0x0262e3 thru 0x02f9f0 (0x970e bytes) 000612 1761283810.364696 mapread 0x03239d thru 0x03ffff (0xdc63 bytes) 000613 1761283810.369792 MIRROR_RESYNC 8 mirrors 000614 1761283810.950389 trunc from 0x040000 to 0x035300 (0xacff bytes) 000615 1761283810.969369 mapread 0x03428d thru 0x035300 (0x1074 bytes) 000616 1761283810.973538 read 0x01ad88 thru 0x023b3f (0x8db8 bytes) 000617 1761283810.976661 mapwrite 0x0034e8 thru 0x0071b2 (0x3ccb bytes) 000620 1761283810.989134 trunc from 0x035301 to 0x00f75c (0x25ba4 bytes) 000621 1761283810.997674 write 0x009466 thru 0x00f5db (0x6176 bytes) 000623 1761283810.999028 trunc from 0x00211f to 0x007839 (0x571b bytes) 000624 1761283811.001748 mapwrite 0x037c58 thru 0x03ffff (0x83a8 bytes) 000625 1761283811.018480 trunc from 0x040000 to 0x026d76 (0x19289 bytes) 000626 1761283811.025140 mapread 0x0267ad thru 0x026d76 (0x5ca bytes) 000627 1761283811.029421 punch from 0x026cff to 0x026d76 (0x78 bytes) 000628 1761283811.031558 read 0x020ed8 thru 0x026d76 (0x5e9f bytes) 000629 1761283811.035792 trunc from 0x01e487 to 0x023a38 (0x55b2 bytes) 000630 1761283811.041078 MIRROR_EXTEND to 9 mirrors 000631 1761283811.251505 punch from 0x00ed11 to 0x01a692 (0xb982 bytes) 000632 1761283811.262838 trunc from 0x026d77 to 0x02d8f5 (0x6b7f bytes) 000633 1761283811.274911 punch from 0x003900 to 0x00eaca (0xb1cb bytes) 000634 1761283811.277445 write 0x035efc thru 0x0397d1 (0x38d6 bytes) 000635 1761283811.278489 trunc from 0x0397d2 to 0x018d4f (0x20a82 bytes) 000636 1761283811.285041 write 0x024d7c thru 0x027697 (0x291c bytes) 000637 1761283811.286217 fallocate 0x00fa19 thru 0x017607 (0x7bef bytes) 000638 1761283811.289120 mapread 0x004034 thru 0x00dae8 (0x9ab5 bytes) 000639 1761283811.296837 mapread 0x02443b thru 0x027697 (0x325d bytes) 000641 1761283811.297906 fallocate 0x013b04 thru 0x017c82 (0x417f bytes) 000642 1761283811.300692 fallocate 0x000bcf thru 0x005e59 (0x528b bytes) 000643 1761283811.303827 mapwrite 0x03022f thru 0x039427 (0x91f9 bytes) 000644 1761283811.326214 write 0x00bf3c thru 0x01b226 (0xf2eb bytes) 000645 1761283811.331217 MIRROR_EXTEND to 10 mirrors 000646 1761283811.557961 mapread 0x004534 thru 0x00dc5d (0x972a bytes) 000647 1761283811.570892 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000650 1761283811.647925 trunc from 0x039428 to 0x00701d (0x3240a bytes) 000652 1761283811.662390 mapwrite 0x0081c4 thru 0x013a5c (0xb899 bytes) 000653 1761283811.685638 trunc from 0x013a5d to 0x033890 (0x1fe34 bytes) 000654 1761283811.692066 punch from 0x00a972 to 0x011e58 (0x74e7 bytes) 000655 1761283811.694871 punch from 0x0254e7 to 0x02bdac (0x68c6 bytes) 000656 1761283811.697054 read 0x0334d2 thru 0x033890 (0x3bf bytes) 000657 1761283811.699407 mapwrite 0x03738d thru 0x0382e4 (0xf58 bytes) 000658 1761283811.714041 trunc from 0x0382e5 to 0x03276c (0x5b78 bytes) 000659 1761283811.719951 trunc from 0x03276d to 0x017f8c (0x1a7e0 bytes) 000660 1761283811.727762 MIRROR_RESYNC 9 mirrors 000661 1761283812.289853 trunc from 0x017f8d to 0x02885b (0x108cf bytes) 000662 1761283812.301003 fallocate 0x01995d thru 0x024bb6 (0xb25a bytes) 000663 1761283812.303339 trunc from 0x0153f3 to 0x0182a8 (0x2eb6 bytes) 000664 1761283812.305207 mapwrite 0x037f0e thru 0x03efb2 (0x70a5 bytes) 000665 1761283812.323174 trunc from 0x03ae99 to 0x03efb2 (0x411a bytes) 000666 1761283812.325173 mapwrite 0x0186b1 thru 0x021a95 (0x93e5 bytes) 000667 1761283812.338035 mapread 0x019d01 thru 0x01f91c (0x5c1c bytes) 000669 1761283812.339152 mapwrite 0x012ccf thru 0x014600 (0x1932 bytes) 000670 1761283812.345321 trunc from 0x03efb3 to 0x026856 (0x1875c bytes) 000671 1761283812.351581 write 0x027572 thru 0x031852 (0xa2e1 bytes) 000672 1761283812.352775 read 0x030e51 thru 0x031852 (0xa02 bytes) 000673 1761283812.353245 write 0x01927f thru 0x028202 (0xef84 bytes) 000674 1761283812.354274 punch from 0x025590 to 0x0310cd (0xbb3e bytes) 000675 1761283812.362846 MIRROR_RESYNC 9 mirrors 000676 1761283813.101316 punch from 0x0298cf to 0x031852 (0x7f84 bytes) 000677 1761283813.117608 write 0x03e0f9 thru 0x03ffff (0x1f07 bytes) 000678 1761283813.118725 write 0x022637 thru 0x029a83 (0x744d bytes) 000679 1761283813.124523 mapread 0x02ce70 thru 0x037b4c (0xacdd bytes) 000680 1761283813.134230 mapread 0x027a2a thru 0x036ef3 (0xf4ca bytes) 000681 1761283813.135148 trunc from 0x018e9d to 0x025d48 (0xceac bytes) 000682 1761283813.143059 write 0x01c792 thru 0x02b586 (0xedf5 bytes) 000683 1761283813.149417 trunc from 0x040000 to 0x0353bc (0xac43 bytes) 000685 1761283813.168572 fallocate 0x03b808 thru 0x03ffff (0x47f8 bytes) 000686 1761283813.171462 trunc from 0x002440 to 0x0047ee (0x23af bytes) 000687 1761283813.173989 mapread 0x0328ef thru 0x03539f (0x2ab1 bytes) 000689 1761283813.175377 trunc from 0x0353bd to 0x03bd8a (0x69ce bytes) 000690 1761283813.181814 mapwrite 0x02c839 thru 0x03730f (0xaad7 bytes) 000691 1761283813.199900 MIRROR_RESYNC 9 mirrors 000692 1761283814.075359 trunc from 0x000ea9 to 0x0067e4 (0x593c bytes) 000693 1761283814.089112 punch from 0x0319ac to 0x0336f9 (0x1d4e bytes) 000694 1761283814.091094 trunc from 0x03bd8b to 0x01a9a7 (0x213e3 bytes) 000696 1761283814.102767 trunc from 0x01a9a8 to 0x01473a (0x626d bytes) 000697 1761283814.107808 mapwrite 0x03556b thru 0x03ffff (0xaa95 bytes) 000698 1761283814.128239 mapread 0x036d2e thru 0x0389ed (0x1cc0 bytes) 000699 1761283814.128920 trunc from 0x040000 to 0x03ee81 (0x117e bytes) 000700 1761283814.134863 write 0x00c96d thru 0x018941 (0xbfd5 bytes) 000701 1761283814.140845 read 0x03d2c1 thru 0x03ee81 (0x1bc1 bytes) 000702 1761283814.141281 mapread 0x00a159 thru 0x016a4b (0xc8f3 bytes) 000703 1761283814.142003 read 0x00502c thru 0x00eb62 (0x9b37 bytes) 000704 1761283814.142459 read 0x02669b thru 0x02a6ed (0x4053 bytes) 000705 1761283814.147090 mapread 0x028513 thru 0x033524 (0xb012 bytes) 000707 1761283814.148250 fallocate 0x0249df thru 0x028910 (0x3f32 bytes) 000708 1761283814.150929 read 0x02deb7 thru 0x03b2c2 (0xd40c bytes) 000709 1761283814.151570 trunc from 0x004a13 to 0x00f70a (0xacf8 bytes) 000710 1761283814.158287 read 0x02ec21 thru 0x03a4a5 (0xb885 bytes) 000711 1761283814.159003 punch from 0x02c947 to 0x02ed69 (0x2423 bytes) 000712 1761283814.161576 punch from 0x02630b to 0x034665 (0xe35b bytes) 000713 1761283814.164378 trunc from 0x03ee82 to 0x025df4 (0x1908d bytes) 000715 1761283814.170450 fallocate 0x01eeab thru 0x02cb89 (0xdcdf bytes) 000716 1761283814.173310 fallocate 0x00ab61 thru 0x01643d (0xb8dd bytes) 000717 1761283814.175839 mapread 0x012e6a thru 0x013687 (0x81e bytes) 000718 1761283814.178835 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000719 1761283814.249366 trunc from 0x01b905 to 0x023a57 (0x8153 bytes) 000720 1761283814.252011 trunc from 0x008748 to 0x00c06b (0x3924 bytes) 000722 1761283814.254617 mapwrite 0x0297e0 thru 0x0339fc (0xa21d bytes) 000723 1761283814.277034 MIRROR_EXTEND to 9 mirrors 000724 1761283814.488400 mapread 0x0338d9 thru 0x0339fc (0x124 bytes) 000725 1761283814.500983 punch from 0x031cd3 to 0x0339fc (0x1d2a bytes) 000726 1761283814.518602 fallocate 0x02a0b6 thru 0x02d8ed (0x3838 bytes) 000727 1761283814.521197 write 0x00c2d3 thru 0x016c89 (0xa9b7 bytes) 000728 1761283814.527893 write 0x03fb7c thru 0x03ffff (0x484 bytes) 000729 1761283814.528825 fallocate 0x0106b4 thru 0x014d46 (0x4693 bytes) 000730 1761283814.531829 trunc from 0x040000 to 0x024bae (0x1b451 bytes) 000731 1761283814.545530 mapread 0x00098a thru 0x004420 (0x3a97 bytes) 000732 1761283814.549731 punch from 0x012635 to 0x01a94e (0x831a bytes) 000733 1761283814.555793 mapwrite 0x02f1d8 thru 0x03c0c9 (0xcef2 bytes) 000734 1761283814.578459 mapread 0x01e219 thru 0x022cb8 (0x4aa0 bytes) 000735 1761283814.582749 punch from 0x01a116 to 0x020a5c (0x6947 bytes) 000736 1761283814.585135 mapwrite 0x0202ae thru 0x02df18 (0xdc6b bytes) 000737 1761283814.598916 punch from 0x01db91 to 0x01ef7e (0x13ee bytes) 000738 1761283814.601376 trunc from 0x03c0ca to 0x004f85 (0x37144 bytes) 000740 1761283814.608618 trunc from 0x002b6f to 0x004f85 (0x2417 bytes) 000741 1761283814.611127 mapread 0x0037c6 thru 0x004f85 (0x17c0 bytes) 000742 1761283814.614101 trunc from 0x002c85 to 0x004f85 (0x2301 bytes) 000743 1761283814.616138 mapread 0x000654 thru 0x004f85 (0x4932 bytes) 000744 1761283814.620239 MIRROR_EXTEND to 10 mirrors 000745 1761283814.815964 read 0x004c57 thru 0x004f85 (0x32f bytes) 000746 1761283814.820815 fallocate 0x023512 thru 0x032645 (0xf134 bytes) 000747 1761283814.835812 trunc from 0x004f86 to 0x024f24 (0x1ff9f bytes) 000748 1761283814.849900 punch from 0x001ec8 to 0x002baf (0xce8 bytes) 000749 1761283814.853502 MIRROR_RESYNC 10 mirrors 000750 1761283815.264683 read 0x00229b thru 0x0112ce (0xf034 bytes) 000751 1761283815.267456 mapwrite 0x0065b9 thru 0x015aed (0xf535 bytes) 000752 1761283815.305690 mapread 0x020fb0 thru 0x024f24 (0x3f75 bytes) 000753 1761283815.309448 fallocate 0x01f765 thru 0x028614 (0x8eb0 bytes) 000754 1761283815.311750 fallocate 0x01a6e4 thru 0x024b83 (0xa4a0 bytes) 000755 1761283815.313803 write 0x027bd6 thru 0x02aed4 (0x32ff bytes) 000756 1761283815.314648 punch from 0x01143e to 0x01fd06 (0xe8c9 bytes) 000757 1761283815.317292 mapwrite 0x0201f7 thru 0x02b0f9 (0xaf03 bytes) 000758 1761283815.337683 punch from 0x022489 to 0x0233a2 (0xf1a bytes) 000759 1761283815.340002 punch from 0x01f34d to 0x024a62 (0x5716 bytes) 000760 1761283815.342381 fallocate 0x0302ba thru 0x03aaa4 (0xa7eb bytes) 000761 1761283815.344611 mapread 0x028a5c thru 0x02b0f9 (0x269e bytes) 000762 1761283815.345399 write 0x032200 thru 0x03de42 (0xbc43 bytes) 000763 1761283815.347079 trunc from 0x03de43 to 0x019b44 (0x242fe bytes) 000764 1761283815.356181 MIRROR_RESYNC 10 mirrors 000765 1761283815.860497 mapwrite 0x018e02 thru 0x01c721 (0x3920 bytes) 000767 1761283815.888830 write 0x02324f thru 0x028533 (0x52e5 bytes) 000768 1761283815.889955 fallocate 0x01f4fb thru 0x024f6c (0x5a72 bytes) 000769 1761283815.892292 read 0x01e3fd thru 0x027c1e (0x9822 bytes) 000770 1761283815.895414 mapwrite 0x03f744 thru 0x03ffff (0x8bc bytes) 000771 1761283815.911393 fallocate 0x01b83e thru 0x0217c3 (0x5f86 bytes) 000772 1761283815.913456 trunc from 0x0380e1 to 0x03ffff (0x7f1f bytes) 000774 1761283815.916058 write 0x019be7 thru 0x02678f (0xcba9 bytes) 000775 1761283815.917302 trunc from 0x040000 to 0x03069c (0xf963 bytes) 000776 1761283815.923025 mapread 0x00f774 thru 0x019e4c (0xa6d9 bytes) 000777 1761283815.927641 trunc from 0x03069d to 0x02fa20 (0xc7c bytes) 000778 1761283815.933880 trunc from 0x02fa21 to 0x01cf86 (0x12a9a bytes) 000779 1761283815.939769 trunc from 0x01cf87 to 0x020b79 (0x3bf3 bytes) 000780 1761283815.946445 write 0x027a1f thru 0x02e5ad (0x6b8f bytes) 000781 1761283815.947773 read 0x0152dd thru 0x01ee44 (0x9b68 bytes) 000782 1761283815.952022 mapread 0x02c621 thru 0x02e5ad (0x1f8d bytes) 000784 1761283815.953248 mapwrite 0x033407 thru 0x03a85e (0x7458 bytes) 000785 1761283815.968803 punch from 0x008e1c to 0x009cec (0xed1 bytes) 000786 1761283815.971513 mapwrite 0x03e0da thru 0x03ffff (0x1f26 bytes) 000787 1761283815.987074 fallocate 0x000b7c thru 0x006c9b (0x6120 bytes) 000788 1761283815.989989 read 0x02e7e8 thru 0x02fef0 (0x1709 bytes) 000790 1761283815.991010 mapread 0x0388e2 thru 0x03c85e (0x3f7d bytes) 000791 1761283815.991842 trunc from 0x040000 to 0x01f5ae (0x20a51 bytes) 000792 1761283815.999331 mapread 0x01321e thru 0x01f5ae (0xc391 bytes) 000793 1761283816.000274 trunc from 0x01f5af to 0x01e89c (0xd12 bytes) 000794 1761283816.006487 mapwrite 0x007078 thru 0x016d06 (0xfc8f bytes) 000795 1761283816.018778 mapread 0x00f401 thru 0x016538 (0x7138 bytes) 000796 1761283816.021807 MIRROR_RESYNC 10 mirrors 000797 1761283816.666022 mapwrite 0x002e43 thru 0x01145d (0xe61b bytes) 000798 1761283816.712084 mapwrite 0x032b0b thru 0x03be08 (0x92fe bytes) 000799 1761283816.736781 trunc from 0x0228c6 to 0x030f81 (0xe6bc bytes) 000800 1761283816.739355 mapwrite 0x028b9f thru 0x02e556 (0x59b8 bytes) 000801 1761283816.748351 fallocate 0x00da07 thru 0x011b2e (0x4128 bytes) 000802 1761283816.750465 write 0x01a636 thru 0x0267f2 (0xc1bd bytes) 000803 1761283816.754072 mapread 0x014298 thru 0x01b8a4 (0x760d bytes) 000804 1761283816.754964 fallocate 0x006360 thru 0x00b38c (0x502d bytes) 000805 1761283816.757499 punch from 0x019548 to 0x028ee5 (0xf99e bytes) 000806 1761283816.762693 write 0x0215bf thru 0x0240b9 (0x2afb bytes) 000807 1761283816.766398 trunc from 0x03be09 to 0x035941 (0x64c7 bytes) 000808 1761283816.771751 trunc from 0x035942 to 0x03c5df (0x6c9e bytes) 000809 1761283816.777663 write 0x022a47 thru 0x02a1d4 (0x778e bytes) 000810 1761283816.780901 MIRROR_RESYNC 10 mirrors 000811 1761283817.678178 punch from 0x0102c2 to 0x01cd43 (0xca82 bytes) 000812 1761283817.694210 read 0x0008d3 thru 0x00ae9d (0xa5cb bytes) 000813 1761283817.698438 fallocate 0x00fa28 thru 0x014454 (0x4a2d bytes) 000814 1761283817.703473 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000815 1761283817.786539 write 0x009321 thru 0x01130f (0x7fef bytes) 000816 1761283817.792168 mapwrite 0x03f64b thru 0x03ffff (0x9b5 bytes) 000817 1761283817.808173 read 0x002a97 thru 0x004628 (0x1b92 bytes) 000818 1761283817.810977 punch from 0x03c1fb to 0x03c946 (0x74c bytes) 000819 1761283817.814917 MIRROR_RESYNC 9 mirrors 000820 1761283818.567405 write 0x0140a0 thru 0x01e4dc (0xa43d bytes) 000821 1761283818.583368 trunc from 0x040000 to 0x0055f1 (0x3aa0e bytes) 000822 1761283818.590489 punch from 0x004f52 to 0x0055f1 (0x6a0 bytes) 000823 1761283818.592501 punch from 0x00473c to 0x0055f1 (0xeb6 bytes) 000824 1761283818.594282 punch from 0x0051b5 to 0x0055f1 (0x43d bytes) 000825 1761283818.596037 mapwrite 0x0150ec thru 0x01b376 (0x628b bytes) 000826 1761283818.611783 mapread 0x011b2a thru 0x0198c5 (0x7d9c bytes) 000827 1761283818.612608 mapwrite 0x032a2d thru 0x03dbd7 (0xb1ab bytes) 000828 1761283818.631058 punch from 0x0239ab to 0x02c3c9 (0x8a1f bytes) 000829 1761283818.633169 write 0x03d4ba thru 0x03ffff (0x2b46 bytes) 000830 1761283818.633947 fallocate 0x0157cb thru 0x024489 (0xecbf bytes) 000831 1761283818.636390 punch from 0x0368dc to 0x037d52 (0x1477 bytes) 000832 1761283818.638574 mapread 0x02f8b2 thru 0x035736 (0x5e85 bytes) 000833 1761283818.639481 write 0x0188ae thru 0x01c346 (0x3a99 bytes) 000834 1761283818.642649 trunc from 0x007780 to 0x00bf11 (0x4792 bytes) 000835 1761283818.644908 mapwrite 0x0170d3 thru 0x01a6e3 (0x3611 bytes) 000836 1761283818.651074 read 0x011d75 thru 0x019d28 (0x7fb4 bytes) 000837 1761283818.651576 mapread 0x00f559 thru 0x01770e (0x81b6 bytes) 000838 1761283818.652150 read 0x03a4dc thru 0x03ffff (0x5b24 bytes) 000839 1761283818.652626 write 0x01f719 thru 0x02b251 (0xbb39 bytes) 000841 1761283818.657114 write 0x012491 thru 0x0206c3 (0xe233 bytes) 000842 1761283818.658085 mapwrite 0x03f45d thru 0x03ffff (0xba3 bytes) 000843 1761283818.665324 fallocate 0x02fb05 thru 0x03950e (0x9a0a bytes) 000844 1761283818.667964 trunc from 0x040000 to 0x0155d3 (0x2aa2c bytes) 000845 1761283818.673816 punch from 0x0001a7 to 0x000ab2 (0x90c bytes) 000846 1761283818.676047 trunc from 0x0155d4 to 0x00f048 (0x658b bytes) 000847 1761283818.681599 mapread 0x0064ee thru 0x00f048 (0x8b5b bytes) 000848 1761283818.682403 punch from 0x00ea0e to 0x00f048 (0x63b bytes) 000850 1761283818.685140 write 0x00aa5f thru 0x014978 (0x9f1a bytes) 000851 1761283818.686062 trunc from 0x014979 to 0x03e256 (0x298de bytes) 000853 1761283818.693335 punch from 0x036e46 to 0x03e256 (0x7411 bytes) 000854 1761283818.695570 read 0x02d9ec thru 0x0328c3 (0x4ed8 bytes) 000855 1761283818.699890 write 0x010678 thru 0x010cda (0x663 bytes) 000856 1761283818.700393 mapwrite 0x03f8da thru 0x03ffff (0x726 bytes) 000857 1761283818.713526 write 0x0250a0 thru 0x02b4aa (0x640b bytes) 000858 1761283818.714423 punch from 0x03135e to 0x034423 (0x30c6 bytes) 000859 1761283818.716809 fallocate 0x017def thru 0x0202c4 (0x84d6 bytes) 000860 1761283818.718832 write 0x0365cb thru 0x038e82 (0x28b8 bytes) 000861 1761283818.721534 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000862 1761283818.795611 mapwrite 0x01654d thru 0x01cece (0x6982 bytes) 000863 1761283818.808006 fallocate 0x03fa27 thru 0x03ffff (0x5d9 bytes) 000864 1761283818.810284 trunc from 0x01e996 to 0x02a24b (0xb8b6 bytes) 000865 1761283818.812926 write 0x00a90d thru 0x012185 (0x7879 bytes) 000866 1761283818.818318 trunc from 0x040000 to 0x028677 (0x17988 bytes) 000867 1761283818.824619 fallocate 0x033e54 thru 0x03b812 (0x79bf bytes) 000868 1761283818.826887 mapread 0x00e543 thru 0x01e1dd (0xfc9b bytes) 000869 1761283818.832053 MIRROR_EXTEND to 9 mirrors 000870 1761283819.060393 trunc from 0x028678 to 0x019903 (0xed74 bytes) 000872 1761283819.081086 punch from 0x000c3e to 0x004181 (0x3544 bytes) 000873 1761283819.083202 mapwrite 0x001849 thru 0x0064dc (0x4c94 bytes) 000875 1761283819.096311 fallocate 0x02723d thru 0x032ea6 (0xbc6a bytes) 000877 1761283819.098728 trunc from 0x019904 to 0x031b82 (0x1827f bytes) 000879 1761283819.105662 mapwrite 0x000be7 thru 0x006865 (0x5c7f bytes) 000880 1761283819.116164 MIRROR_EXTEND to 10 mirrors 000882 1761283819.351068 trunc from 0x013351 to 0x01e3a1 (0xb051 bytes) 000883 1761283819.366044 fallocate 0x00f1a4 thru 0x014393 (0x51f0 bytes) 000884 1761283819.369669 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000887 1761283819.441042 trunc from 0x031045 to 0x031b82 (0xb3e bytes) 000888 1761283819.443885 read 0x00f1fb thru 0x013766 (0x456c bytes) 000889 1761283819.447348 fallocate 0x0057a7 thru 0x01389f (0xe0f9 bytes) 000892 1761283819.452627 punch from 0x001b55 to 0x001e71 (0x31d bytes) 000893 1761283819.455469 fallocate 0x01e6be thru 0x02a7db (0xc11e bytes) 000894 1761283819.458477 punch from 0x0133f6 to 0x0168f6 (0x3501 bytes) 000897 1761283819.461895 fallocate 0x013146 thru 0x01f7aa (0xc665 bytes) 000898 1761283819.466655 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000899 1761283819.538423 mapwrite 0x022771 thru 0x02919b (0x6a2b bytes) 000900 1761283819.552509 trunc from 0x031b83 to 0x02082c (0x11356 bytes) 000901 1761283819.564874 mapwrite 0x009482 thru 0x00f0cf (0x5c4e bytes) 000902 1761283819.576030 write 0x0285d8 thru 0x034775 (0xc19e bytes) 000903 1761283819.577452 mapread 0x02e54c thru 0x02f7e8 (0x129d bytes) 000904 1761283819.578235 punch from 0x021d9d to 0x02cd82 (0xafe6 bytes) 000905 1761283819.583691 fallocate 0x008c1f thru 0x013784 (0xab66 bytes) 000906 1761283819.585960 read 0x014817 thru 0x01ea06 (0xa1f0 bytes) 000907 1761283819.589626 mapread 0x004f2e thru 0x01187f (0xc952 bytes) 000908 1761283819.590439 write 0x01a07a thru 0x01f07e (0x5005 bytes) 000909 1761283819.591129 mapwrite 0x002dfc thru 0x00664a (0x384f bytes) 000910 1761283819.602439 MIRROR_RESYNC 8 mirrors 000911 1761283820.236243 trunc from 0x034776 to 0x027322 (0xd453 bytes) 000912 1761283820.249666 trunc from 0x027323 to 0x019f1a (0xd408 bytes) 000913 1761283820.257062 trunc from 0x0016a4 to 0x0029dd (0x133a bytes) 000914 1761283820.259618 mapread 0x00d0fa thru 0x00e070 (0xf77 bytes) 000915 1761283820.264199 mapread 0x00b72e thru 0x00fca7 (0x457a bytes) 000916 1761283820.267819 trunc from 0x014f1d to 0x019f1a (0x4ffe bytes) 000917 1761283820.270098 mapread 0x00d0ce thru 0x019f1a (0xce4d bytes) 000918 1761283820.274696 mapread 0x007f15 thru 0x0140fe (0xc1ea bytes) 000919 1761283820.275550 punch from 0x00ad8a to 0x019f1a (0xf191 bytes) 000920 1761283820.278964 mapwrite 0x029a70 thru 0x02c02e (0x25bf bytes) 000921 1761283820.295748 trunc from 0x02c02f to 0x0194f0 (0x12b3e bytes) 000922 1761283820.302088 trunc from 0x006df6 to 0x01051d (0x9728 bytes) 000923 1761283820.305016 trunc from 0x0194f1 to 0x00a0b7 (0xf439 bytes) 000924 1761283820.311144 trunc from 0x00138d to 0x004562 (0x31d6 bytes) 000925 1761283820.315389 MIRROR_EXTEND to 9 mirrors 000926 1761283820.555112 mapwrite 0x03a5c6 thru 0x03ffff (0x5a3a bytes) 000927 1761283820.585762 MIRROR_EXTEND to 10 mirrors 000928 1761283820.808030 write 0x0055db thru 0x00f7e4 (0xa20a bytes) 000929 1761283820.826375 MIRROR_EXTEND to 11 mirrors 000930 1761283821.043539 write 0x023fb2 thru 0x02dc9f (0x9cee bytes) 000931 1761283821.063986 fallocate 0x022e43 thru 0x023315 (0x4d3 bytes) 000932 1761283821.066649 mapread 0x034c8b thru 0x03fcd1 (0xb047 bytes) 000933 1761283821.073797 trunc from 0x017985 to 0x025636 (0xdcb2 bytes) 000934 1761283821.080810 trunc from 0x01453d to 0x021026 (0xcaea bytes) 000935 1761283821.084263 mapwrite 0x02e002 thru 0x0321f6 (0x41f5 bytes) 000936 1761283821.093880 mapread 0x008826 thru 0x00c026 (0x3801 bytes) 000937 1761283821.099179 mapwrite 0x01a2a5 thru 0x01eaf7 (0x4853 bytes) 000938 1761283821.113955 read 0x00a5fc thru 0x012432 (0x7e37 bytes) 000939 1761283821.117678 mapwrite 0x024f10 thru 0x030c9a (0xbd8b bytes) 000940 1761283821.133276 write 0x02e247 thru 0x02ef76 (0xd30 bytes) 000941 1761283821.134189 trunc from 0x03b922 to 0x03ffff (0x46de bytes) 000943 1761283821.138197 punch from 0x011843 to 0x017f55 (0x6713 bytes) 000944 1761283821.144063 MIRROR_RESYNC 11 mirrors 000945 1761283822.198132 read 0x00797d thru 0x00efb4 (0x7638 bytes) 000947 1761283822.201833 write 0x02df46 thru 0x03d6a2 (0xf75d bytes) 000948 1761283822.223295 read 0x03ec04 thru 0x03ffff (0x13fc bytes) 000949 1761283822.228887 trunc from 0x040000 to 0x016513 (0x29aec bytes) 000950 1761283822.238941 mapwrite 0x026e6e thru 0x03035c (0x94ef bytes) 000951 1761283822.279816 fallocate 0x0099d4 thru 0x01251f (0x8b4c bytes) 000953 1761283822.282775 read 0x018262 thru 0x027277 (0xf016 bytes) 000954 1761283822.286628 trunc from 0x03035d to 0x02ac6b (0x56f1 bytes) 000956 1761283822.293389 fallocate 0x00160c thru 0x006653 (0x5048 bytes) 000957 1761283822.295675 punch from 0x011a0a to 0x017a22 (0x6019 bytes) 000958 1761283822.298310 mapwrite 0x02f6c6 thru 0x0376a7 (0x7fe2 bytes) 000959 1761283822.322035 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x5c8:0x0]: Invalid argument (22) 000960 1761283822.412010 punch from 0x016d78 to 0x01a061 (0x32ea bytes) 000961 1761283822.414749 fallocate 0x0023ea thru 0x004b68 (0x277f bytes) 000962 1761283822.417550 punch from 0x006ed9 to 0x00f532 (0x865a bytes) 000963 1761283822.420779 fallocate 0x0241ae thru 0x030977 (0xc7ca bytes) 000964 1761283822.424158 mapread 0x01ef7f thru 0x029359 (0xa3db bytes) 000965 1761283822.437850 fallocate 0x009c0c thru 0x00dd9b (0x4190 bytes) 000966 1761283822.440922 trunc from 0x01ae61 to 0x024aa7 (0x9c47 bytes) 000967 1761283822.444877 write 0x023e12 thru 0x02dc68 (0x9e57 bytes) 000968 1761283822.452645 fallocate 0x0251bf thru 0x0333a2 (0xe1e4 bytes) 000969 1761283822.455552 fallocate 0x001978 thru 0x009c46 (0x82cf bytes) 000970 1761283822.458806 trunc from 0x0376a8 to 0x03e8c1 (0x721a bytes) 000971 1761283822.465695 read 0x03d8cc thru 0x03e8c1 (0xff6 bytes) 000972 1761283822.471852 MIRROR_EXTEND to 11 mirrors 000973 1761283822.745692 trunc from 0x00ee6a to 0x019abc (0xac53 bytes) 000974 1761283822.764572 MIRROR_RESYNC 11 mirrors 000975 1761283823.590284 mapwrite 0x0288fb thru 0x02b720 (0x2e26 bytes) 000976 1761283823.626390 read 0x0127ef thru 0x020069 (0xd87b bytes) 000977 1761283823.629794 mapread 0x024505 thru 0x029b8c (0x5688 bytes) 000978 1761283823.630684 write 0x03390e thru 0x039304 (0x59f7 bytes) 000979 1761283823.635317 mapread 0x022be3 thru 0x02501b (0x2439 bytes) 000980 1761283823.636222 trunc from 0x0087a4 to 0x017635 (0xee92 bytes) 000981 1761283823.639401 read 0x029f14 thru 0x032899 (0x8986 bytes) 000982 1761283823.640062 write 0x018ade thru 0x01ae74 (0x2397 bytes) 000983 1761283823.641039 fallocate 0x02403c thru 0x02a7c5 (0x678a bytes) 000984 1761283823.644167 mapread 0x00d026 thru 0x00d6ca (0x6a5 bytes) 000985 1761283823.649005 trunc from 0x03e8c2 to 0x021891 (0x1d030 bytes) 000986 1761283823.657366 trunc from 0x01c65e to 0x020be2 (0x4585 bytes) 000987 1761283823.660168 fallocate 0x036e27 thru 0x038b1c (0x1cf6 bytes) 000988 1761283823.663016 mapread 0x038100 thru 0x038b1c (0xa1d bytes) 000989 1761283823.666286 mapwrite 0x017e39 thru 0x025ff3 (0xe1bb bytes) 000990 1761283823.695040 trunc from 0x038b1d to 0x02e2ec (0xa830 bytes) 000991 1761283823.701566 mapread 0x00ab1f thru 0x016919 (0xbdfb bytes) 000992 1761283823.702494 mapwrite 0x01e854 thru 0x02e46e (0xfc1b bytes) 000993 1761283823.733605 mapread 0x005fde thru 0x013e8b (0xdeae bytes) 000994 1761283823.734323 trunc from 0x02e46f to 0x026227 (0x8247 bytes) 000995 1761283823.739353 trunc from 0x01f5a3 to 0x026227 (0x6c85 bytes) 000996 1761283823.741293 mapread 0x01fad9 thru 0x0218a4 (0x1dcc bytes) 000997 1761283823.745863 write 0x01daad thru 0x0297f8 (0xbd4c bytes) 000998 1761283823.746974 write 0x007aa4 thru 0x013d5f (0xc2bc bytes) 000999 1761283823.747814 write 0x004bdf thru 0x012d9f (0xe1c1 bytes) 001000 1761283823.748742 mapwrite 0x01434e thru 0x01b7d7 (0x748a bytes) All operations completed A-OK! PASS 70a (28s) == sanity-flr test 71: check mirror extend parent fid ==== 01:30:25 (1761283825) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x5f8:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 01:30:29 (1761283829) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8460800 .. Extending file size to 8453536 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8722208 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8791040 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8960160 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9058944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8751744 .. failed Extending file size to 8892000 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8900032 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9128992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61706 61707 61709 61711 61713 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) 192.168.203.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:31:34 (1761283894) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1163264 .. Extending file size to 4260384 .. done Extending file size to 5867392 .. Extending file size to 5349952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5989056 .. Extending file size to 7020128 .. failed Extending file size to 7612256 .. Extending file size to 6700736 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8251616 .. Extending file size to 7490176 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7637056 .. Extending file size to 8816800 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9166144 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8819712 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8861280 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 65842 65843 65845 65847 65849 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) 192.168.203.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:32:39 (1761283959) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2548 1268756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2796 1268508 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21672 3585348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15768 3591252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37440 7176600 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2548 1268756 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2796 1268508 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21672 3585348 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15768 3591252 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37440 7176600 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.042543 s, 49.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0569401 s, 36.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:32:45 (1761283965) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0353934 s, 59.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 01:32:48 (1761283968) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:32:50 (1761283970) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.024411 s, 85.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:32:52 (1761283972) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0258308 s, 81.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:32:55 (1761283975) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0201232 s, 104 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:33:19 (1761283999) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018643 s, 56.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:33:55 (1761284035) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0308261 s, 68.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:34:31 (1761284071) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg314-server: Pool lustre.test_204f created Adding targets to pool oleg314-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg314-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0236367 s, 88.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x80b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x80b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg314-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg314-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg314-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:35:22 (1761284122) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:35:24 (1761284124) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:35:27 (1761284127) oleg314-server: Pool lustre.test_206 created Adding targets to pool oleg314-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg314-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg314-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg314-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg314-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:35:42 (1761284142) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346745 s, 30.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117066 s, 89.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x53a54078 Mirror 3: 0x53a54078 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:35:46 (1761284146) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:35:47 (1761284147) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:35:48 (1761284148) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0185169 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1250000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:36:05 (1761284165) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00323289 s, 2.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:36:07 (1761284167) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172277 s, 60.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:36:08.000000000 -0400 Modify: 2025-10-24 01:36:08.000000000 -0400 Change: 2025-10-24 01:36:08.000000000 -0400 Birth: 2025-10-24 01:36:08.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:36:10 (1761284170) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155952 s, 67.2 MB/s oleg314-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg314-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg314-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:36:20 (1761284180) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0290176 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00578213 s, 708 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 24 01:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 24 01:36 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 866 sec ============ 01:36:23 (1761284183) === sanity-flr: start cleanup 01:36:23 (1761284183) === === sanity-flr: finish cleanup 01:36:26 (1761284186) ===