-----============= acceptance-small: sanity-flr ============----- Mon Jul 14 03:05:41 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg356-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:05:51 (1752476751) === oleg356-client.virtnet: executing check_config_client /mnt/lustre oleg356-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg356-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99198642d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99198642d800.idle_timeout=debug disable quota as required oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:06:08 (1752476768) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7564] == sanity-flr test 0a: lfs mirror create with -N option == 03:06:09 (1752476769) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:06:18 (1752476778) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:06:19 (1752476779) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:06:21 (1752476781) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:06:34 (1752476794) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:06:35 (1752476795) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 03:06:37 (1752476797) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99198642d800.stats=clear osc.lustre-OST0001-osc-ffff99198642d800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:06:41 (1752476801) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:06:46 (1752476806) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:06:49 (1752476809) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:06:50 (1752476810) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 03:06:53 (1752476813) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:06:56 (1752476816) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:06:57 (1752476817) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:07:00 (1752476820) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:07:03 (1752476823) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:07:06 (1752476826) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.363001 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459116 s, 22.8 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:07:10 (1752476830) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.0855602 s, 110 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:07:13 (1752476833) Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:07:39 (1752476859) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.118019 s, 62.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:07:51 (1752476871) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:08:14 (1752476894) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.79747 s, 65.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 14 03:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.214s user 0m0.000s sys 0m0.132s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.231s user 0m0.000s sys 0m0.140s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.306s user 0m0.001s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99198642d800.ost_server_uuid in FULL state after 0 sec PASS 33b (25s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:08:39 (1752476919) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:08:40 (1752476920) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:08:41 (1752476921) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 03:08:42 (1752476922) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 03:08:44 (1752476924) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.152326 s, 82.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:08:48 (1752476928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145667 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143361 s, 73.1 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:08:51 (1752476931) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:08:54 (1752476934) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000587793 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000437604 s, 2.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000584375 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000554651 s, 1.8 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 03:08:57 (1752476937) create test files with size 15755796 .. beeba81e7fc6c1769853a83ef7c39580 - 050223c2ab00057e3d141a8d28dcc26c - b14e3bec7033343124854d014991f97b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 03:09:03 (1752476943) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:09:07 (1752476947) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:09:09 (1752476949) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364928 s, 28.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925157 s, 113 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 03:09:12 (1752476952) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0380685 s, 55.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.101359 s, 41.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0560665 s, 37.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 03:09:16 (1752476956) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 03:09:17 (1752476957) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:09:18 (1752476958) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116011 s, 90.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 03:09:20 (1752476960) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:09:21 (1752476961) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:09:24 (1752476964) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0970612 s, 108 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:09:46 (1752476986) llite.lustre-ffff99198a3de800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0788203 s, 133 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff991999111000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:09:50 (1752476990) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:10:01 (1752477001) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:10:12 (1752477012) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 03:10:13 (1752477013) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0620471 s, 50.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0570051 s, 55.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 8af834b6ee4f617f6c22a7326770a3e7 - mirror 1 checksum: 733d4e2fc1e8483562ca04b99621bd0e - mirror 2 checksum: 8af834b6ee4f617f6c22a7326770a3e7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jul 14 03:10:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jul 14 03:10:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8af834b6ee4f617f6c22a7326770a3e7 - mirror 1 checksum: 733d4e2fc1e8483562ca04b99621bd0e - mirror 2 checksum: 733d4e2fc1e8483562ca04b99621bd0e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jul 14 03:10:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jul 14 03:10:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:10:16 (1752477016) Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 03:10:18 (1752477018) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 03:10:32 (1752477032) targets are mounted 03:10:32 (1752477032) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129381 s, 81.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:10:38 (1752477038) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:10:39 (1752477039) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:10:40 (1752477040) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00279002 s, 5.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:10:43 (1752477043) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:10:44 (1752477044) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:10:45 (1752477045) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:10:50 (1752477050) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:11:08 (1752477068) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:11:29 (1752477089) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:11:52 (1752477112) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 03:11:59 (1752477119) mirror create pid 52227 mirror split pid 52229 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7670: 52227 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 52229 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 03:13:06 (1752477186) Seed set to 7188 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000017 1752477188.947036 MIRROR_EXTEND to 2 mirrors 000018 1752477189.346278 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) skipping zero size read truncating to largest ever: 0x2c9cd 000024 1752477189.476539 trunc from 00000000 to 0x02c9cc (0x2c9cd bytes) 000025 1752477189.504421 read 0x004981 thru 0x006529 (0x1ba9 bytes) 000026 1752477189.511644 mapread 0x011b24 thru 0x012124 (0x601 bytes) 000029 1752477189.524814 trunc from 0x02c9cd to 0x00095c (0x2c070 bytes) 000030 1752477189.538686 mapread 0x000077 thru 0x00095c (0x8e6 bytes) 000036 1752477189.547946 write 0x00a1f4 thru 0x00acf6 (0xb03 bytes) truncating to largest ever: 0x374dc 000038 1752477189.550601 trunc from 0x00acf7 to 0x0374db (0x2c7e5 bytes) 000040 1752477189.570357 mapread 0x02c539 thru 0x037015 (0xaadd bytes) 000041 1752477189.584399 trunc from 0x0374dc to 0x019d8f (0x1d74c bytes) 000046 1752477189.605167 MIRROR_EXTEND to 2 mirrors 000047 1752477190.481265 read 0x011004 thru 0x0141fc (0x31f9 bytes) 000048 1752477190.488814 MIRROR_RESYNC 2 mirrors 000049 1752477190.633130 MIRROR_RESYNC 2 mirrors 000050 1752477190.813554 read 0x017ed6 thru 0x019d8f (0x1eba bytes) 000053 1752477190.817899 mapwrite 0x012967 thru 0x021fb6 (0xf650 bytes) 000054 1752477190.961925 write 0x01ef4a thru 0x02ae43 (0xbefa bytes) 000055 1752477190.963183 read 0x00af54 thru 0x00eca8 (0x3d55 bytes) 000056 1752477190.973256 mapread 0x020498 thru 0x0255a4 (0x510d bytes) 000057 1752477190.974305 mapread 0x005c52 thru 0x00a5cf (0x497e bytes) 000061 1752477190.976801 mapwrite 0x01a5fb thru 0x01a983 (0x389 bytes) 000062 1752477191.032282 mapread 0x000fc6 thru 0x007902 (0x693d bytes) 000064 1752477191.033562 mapread 0x023112 thru 0x027ca4 (0x4b93 bytes) 000066 1752477191.034718 mapread 0x00754f thru 0x00ad90 (0x3842 bytes) truncating to largest ever: 0x3e9c2 000068 1752477191.041130 trunc from 0x02ae44 to 0x03e9c1 (0x13b7e bytes) 000070 1752477191.067169 write 0x037794 thru 0x037b26 (0x393 bytes) 000072 1752477191.087145 mapwrite 0x028452 thru 0x035394 (0xcf43 bytes) 000073 1752477191.162653 write 0x014268 thru 0x02064e (0xc3e7 bytes) 000074 1752477191.163921 write 0x00545f thru 0x0092ac (0x3e4e bytes) 000075 1752477191.164776 mapwrite 0x034677 thru 0x036987 (0x2311 bytes) 000076 1752477191.226190 read 0x0054f2 thru 0x014d74 (0xf883 bytes) 000077 1752477191.228412 read 0x00f44b thru 0x019ba4 (0xa75a bytes) 000078 1752477191.229001 mapwrite 0x019177 thru 0x025dc1 (0xcc4b bytes) 000079 1752477191.279424 read 0x000e39 thru 0x008e21 (0x7fe9 bytes) 000080 1752477191.283904 write 0x030522 thru 0x03ff5a (0xfa39 bytes) 000081 1752477191.285667 write 0x00ffaf thru 0x01394c (0x399e bytes) 000083 1752477191.288252 mapwrite 0x014388 thru 0x01f756 (0xb3cf bytes) 000085 1752477191.364411 mapwrite 0x039241 thru 0x03ffff (0x6dbf bytes) 000086 1752477191.493846 read 0x017d3f thru 0x01d8f4 (0x5bb6 bytes) 000087 1752477191.499279 write 0x03862a thru 0x03ffff (0x79d6 bytes) 000088 1752477191.508980 MIRROR_RESYNC 2 mirrors 000089 1752477192.047805 mapwrite 0x02ef1b thru 0x03b397 (0xc47d bytes) 000091 1752477192.247259 mapread 0x003e22 thru 0x00fa2b (0xbc0a bytes) 000092 1752477192.262650 write 0x0275fd thru 0x03427f (0xcc83 bytes) 000094 1752477192.288416 read 0x02f1a0 thru 0x03b79d (0xc5fe bytes) 000096 1752477192.289384 mapwrite 0x03d611 thru 0x03ffff (0x29ef bytes) 000097 1752477192.376641 MIRROR_EXTEND to 3 mirrors 000098 1752477193.189905 mapwrite 0x037c88 thru 0x03ffff (0x8378 bytes) 000099 1752477193.487742 mapwrite 0x006607 thru 0x0129aa (0xc3a4 bytes) 000103 1752477193.584282 MIRROR_RESYNC 3 mirrors 000104 1752477194.373254 trunc from 0x040000 to 0x007165 (0x38e9a bytes) 000105 1752477194.465916 write 0x023140 thru 0x02414d (0x100e bytes) 000106 1752477194.466836 trunc from 0x02414e to 0x01c2c7 (0x7e86 bytes) 000107 1752477194.490957 mapread 0x01a16f thru 0x01c2c7 (0x2159 bytes) 000110 1752477194.511274 mapwrite 0x03f071 thru 0x03ffff (0xf8f bytes) 000113 1752477194.570079 trunc from 0x040000 to 0x0206c8 (0x1f937 bytes) 000115 1752477194.592926 mapread 0x007596 thru 0x0131a9 (0xbc14 bytes) 000116 1752477194.611103 mapread 0x015dcb thru 0x0206c8 (0xa8fe bytes) 000118 1752477194.620111 write 0x023c79 thru 0x028d98 (0x5120 bytes) 000119 1752477194.623462 mapread 0x018b3b thru 0x01ff54 (0x741a bytes) 000120 1752477194.624395 mapwrite 0x038f13 thru 0x03a425 (0x1513 bytes) 000127 1752477194.701496 write 0x006fb7 thru 0x013640 (0xc68a bytes) 000128 1752477194.709462 MIRROR_RESYNC 3 mirrors 000129 1752477195.435125 mapwrite 0x031ebc thru 0x033161 (0x12a6 bytes) 000130 1752477195.621833 read 0x03539f thru 0x03a425 (0x5087 bytes) 000131 1752477195.622502 trunc from 0x03a426 to 0x014f11 (0x25514 bytes) 000132 1752477195.653632 read 0x005b87 thru 0x00ec61 (0x90db bytes) 000134 1752477195.667597 write 0x02a293 thru 0x03258c (0x82fa bytes) 000135 1752477195.668826 trunc from 0x03258d to 0x03a76c (0x81e0 bytes) 000138 1752477195.694679 mapread 0x014d6a thru 0x018d29 (0x3fc0 bytes) 000139 1752477195.717837 mapread 0x0378f2 thru 0x03a76c (0x2e7b bytes) 000140 1752477195.718615 mapwrite 0x0192e8 thru 0x026655 (0xd36e bytes) 000141 1752477195.812171 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000143 1752477196.119830 write 0x028df1 thru 0x02f00b (0x621b bytes) 000144 1752477196.139799 mapread 0x00fbc9 thru 0x011f4e (0x2386 bytes) 000145 1752477196.166702 read 0x034938 thru 0x03a76c (0x5e35 bytes) 000146 1752477196.183184 write 0x004ed0 thru 0x010b2e (0xbc5f bytes) 000147 1752477196.206238 mapread 0x012274 thru 0x020b80 (0xe90d bytes) 000149 1752477196.224458 read 0x0150fd thru 0x018cf2 (0x3bf6 bytes) 000150 1752477196.230064 mapwrite 0x02770b thru 0x036866 (0xf15c bytes) 000152 1752477196.334366 read 0x039683 thru 0x03a76c (0x10ea bytes) 000153 1752477196.340921 read 0x0390ca thru 0x03a76c (0x16a3 bytes) 000158 1752477196.349983 mapread 0x027ada thru 0x035c65 (0xe18c bytes) 000159 1752477196.350912 trunc from 0x03a76d to 0x0260d1 (0x1469b bytes) 000160 1752477196.391271 mapwrite 0x01f78f thru 0x0296e5 (0x9f57 bytes) 000161 1752477196.502886 mapwrite 0x03f4e9 thru 0x03ffff (0xb17 bytes) 000164 1752477196.690871 mapread 0x012688 thru 0x018d55 (0x66ce bytes) 000165 1752477196.704550 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000166 1752477196.891640 read 0x03e396 thru 0x03ffff (0x1c6a bytes) 000168 1752477196.911869 MIRROR_EXTEND to 2 mirrors 000169 1752477197.726217 mapread 0x014718 thru 0x01cd47 (0x8630 bytes) 000170 1752477197.749770 mapread 0x01854c thru 0x020602 (0x80b7 bytes) 000172 1752477197.764654 MIRROR_RESYNC 2 mirrors 000175 1752477197.975245 MIRROR_RESYNC 2 mirrors 000176 1752477198.192050 mapwrite 0x01ca26 thru 0x01d509 (0xae4 bytes) 000179 1752477198.392266 mapread 0x01eef8 thru 0x02656b (0x7674 bytes) 000181 1752477198.464243 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000184 1752477198.669888 mapwrite 0x018416 thru 0x0251f0 (0xcddb bytes) 000185 1752477198.882628 mapwrite 0x03cd17 thru 0x03ffff (0x32e9 bytes) 000186 1752477198.974709 mapread 0x03d2b5 thru 0x03ffff (0x2d4b bytes) 000187 1752477198.990517 MIRROR_EXTEND to 2 mirrors 000188 1752477199.764232 trunc from 0x040000 to 0x00636a (0x39c95 bytes) 000189 1752477199.869095 read 0x000bc3 thru 0x00636a (0x57a8 bytes) 000190 1752477199.884081 read 0x00096a thru 0x00636a (0x5a01 bytes) 000191 1752477199.884664 write 0x00519c thru 0x00c188 (0x6fed bytes) 000192 1752477199.885721 mapwrite 0x016eb9 thru 0x0259b3 (0xeafb bytes) 000193 1752477200.040794 mapread 0x022a17 thru 0x0259b3 (0x2f9d bytes) 000194 1752477200.046775 mapread 0x01f5f7 thru 0x0259b3 (0x63bd bytes) 000197 1752477200.055172 mapread 0x013019 thru 0x01c3d6 (0x93be bytes) 000198 1752477200.055882 write 0x023ce3 thru 0x030363 (0xc681 bytes) 000200 1752477200.057295 write 0x01a0b7 thru 0x027c8d (0xdbd7 bytes) 000201 1752477200.087721 MIRROR_EXTEND to 3 mirrors 000203 1752477200.907091 read 0x0243cc thru 0x026921 (0x2556 bytes) 000204 1752477200.919463 write 0x01b73c thru 0x028bf8 (0xd4bd bytes) 000207 1752477201.086997 mapread 0x026791 thru 0x02c217 (0x5a87 bytes) 000208 1752477201.105950 read 0x026222 thru 0x02f82a (0x9609 bytes) 000210 1752477201.114611 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000211 1752477201.407229 read 0x00f95c thru 0x01a952 (0xaff7 bytes) 000212 1752477201.423036 read 0x0040e9 thru 0x0054d8 (0x13f0 bytes) 000213 1752477201.434836 mapread 0x016715 thru 0x017ed6 (0x17c2 bytes) 000214 1752477201.453619 MIRROR_EXTEND to 3 mirrors 000215 1752477202.114795 mapwrite 0x01b076 thru 0x023cf2 (0x8c7d bytes) 000216 1752477202.301758 read 0x025166 thru 0x030363 (0xb1fe bytes) 000218 1752477202.312940 write 0x00d283 thru 0x0101ab (0x2f29 bytes) 000219 1752477202.321222 mapwrite 0x03c2e6 thru 0x03d6e4 (0x13ff bytes) 000220 1752477202.478610 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000222 1752477202.741139 MIRROR_EXTEND to 3 mirrors 000223 1752477203.385542 write 0x009601 thru 0x00aab9 (0x14b9 bytes) 000224 1752477203.498777 trunc from 0x03d6e5 to 0x0081f5 (0x354ef bytes) 000228 1752477203.530579 mapwrite 0x03f716 thru 0x03ffff (0x8ea bytes) 000229 1752477203.661914 write 0x00939d thru 0x0112f9 (0x7f5d bytes) 000231 1752477203.679952 trunc from 0x040000 to 0x009a7d (0x36582 bytes) 000232 1752477203.731215 trunc from 0x009a7e to 0x038ba8 (0x2f12b bytes) 000234 1752477203.763395 write 0x039bad thru 0x03ffff (0x6453 bytes) 000235 1752477203.764602 mapwrite 0x0341c1 thru 0x036c1a (0x2a5a bytes) 000236 1752477203.890971 trunc from 0x040000 to 0x008c2d (0x373d2 bytes) 000238 1752477203.925600 trunc from 0x008c2e to 0x009fe9 (0x13bc bytes) 000239 1752477203.958143 MIRROR_EXTEND to 4 mirrors 000240 1752477204.568492 mapwrite 0x015767 thru 0x01fabe (0xa358 bytes) 000241 1752477204.744261 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000242 1752477204.994968 write 0x000674 thru 0x0083f9 (0x7d86 bytes) 000243 1752477205.011913 trunc from 0x01fabf to 0x00444c (0x1b672 bytes) 000245 1752477205.033915 mapwrite 0x036ed5 thru 0x03ab96 (0x3cc2 bytes) 000246 1752477205.200230 mapwrite 0x0267ac thru 0x028317 (0x1b6c bytes) 000249 1752477205.248566 trunc from 0x03ab97 to 0x005d6c (0x34e2a bytes) 000250 1752477205.261939 mapwrite 0x02994b thru 0x036a92 (0xd148 bytes) 000254 1752477205.398225 MIRROR_RESYNC 3 mirrors 000256 1752477205.915959 read 0x01a247 thru 0x02822a (0xdfe4 bytes) 000257 1752477205.926459 MIRROR_EXTEND to 4 mirrors 000260 1752477206.575840 MIRROR_EXTEND to 5 mirrors 000261 1752477207.180072 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000262 1752477207.582922 mapread 0x00dbe0 thru 0x00efaa (0x13cb bytes) 000266 1752477207.670088 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000268 1752477208.119566 trunc from 0x036a93 to 0x006642 (0x30450 bytes) 000269 1752477208.211056 write 0x03c5f3 thru 0x03ffff (0x3a0d bytes) 000270 1752477208.212053 mapread 0x006cdd thru 0x00cb01 (0x5e25 bytes) 000271 1752477208.226706 write 0x0034ce thru 0x0124ee (0xf021 bytes) 000275 1752477208.232804 trunc from 0x040000 to 0x03c8b1 (0x374e bytes) 000277 1752477208.264128 trunc from 0x03c8b2 to 0x027c10 (0x14ca1 bytes) 000278 1752477208.279839 write 0x00a1d0 thru 0x00f404 (0x5235 bytes) 000279 1752477208.281414 mapwrite 0x03569e thru 0x03ffff (0xa962 bytes) 000280 1752477208.352627 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000281 1752477208.543724 read 0x031c27 thru 0x039d17 (0x80f1 bytes) 000284 1752477208.565022 trunc from 0x040000 to 0x0333e5 (0xcc1a bytes) 000286 1752477208.588585 mapwrite 0x024153 thru 0x02adf7 (0x6ca5 bytes) 000289 1752477208.649547 read 0x015802 thru 0x022a60 (0xd25f bytes) 000290 1752477208.675109 write 0x0252ec thru 0x0289f3 (0x3708 bytes) 000293 1752477208.676976 mapread 0x00a38e thru 0x019954 (0xf5c7 bytes) 000294 1752477208.680631 mapwrite 0x02811a thru 0x030124 (0x800b bytes) 000296 1752477208.724670 read 0x00700a thru 0x00a8b0 (0x38a7 bytes) 000297 1752477208.725506 trunc from 0x0333e6 to 0x00cd78 (0x2666d bytes) 000302 1752477208.742289 mapwrite 0x0371ac thru 0x03ffff (0x8e54 bytes) 000303 1752477208.847745 mapwrite 0x020f6f thru 0x02394a (0x29dc bytes) 000304 1752477208.883667 write 0x00599e thru 0x006ee9 (0x154c bytes) 000305 1752477208.884698 write 0x0291a1 thru 0x02a795 (0x15f5 bytes) 000309 1752477208.887672 mapread 0x021656 thru 0x02f6dc (0xe087 bytes) 000313 1752477208.891449 mapwrite 0x0044f3 thru 0x011442 (0xcf50 bytes) 000316 1752477208.946341 trunc from 0x040000 to 0x022e83 (0x1d17c bytes) 000317 1752477208.965713 mapwrite 0x00aef5 thru 0x010204 (0x5310 bytes) 000318 1752477209.011229 mapread 0x01fb91 thru 0x022e83 (0x32f3 bytes) 000319 1752477209.012055 mapwrite 0x000aa5 thru 0x001950 (0xeac bytes) 000320 1752477209.046875 trunc from 0x022e84 to 0x021561 (0x1922 bytes) 000321 1752477209.062251 mapread 0x000593 thru 0x0036c5 (0x3133 bytes) 000322 1752477209.063733 write 0x006bc3 thru 0x00d94e (0x6d8c bytes) 000323 1752477209.065192 read 0x0101ed thru 0x015634 (0x5448 bytes) 000325 1752477209.066977 mapwrite 0x03ff67 thru 0x03ffff (0x99 bytes) 000326 1752477209.131748 mapread 0x0088ed thru 0x017c92 (0xf3a6 bytes) 000327 1752477209.133272 read 0x008ec6 thru 0x0186b3 (0xf7ee bytes) 000328 1752477209.133778 write 0x020fb6 thru 0x02be9d (0xaee8 bytes) 000329 1752477209.134840 write 0x03bac0 thru 0x03ffff (0x4540 bytes) 000331 1752477209.136107 mapread 0x029f5a thru 0x02f50b (0x55b2 bytes) 000334 1752477209.137761 write 0x008eac thru 0x0116aa (0x87ff bytes) 000336 1752477209.139156 write 0x00db0f thru 0x01831a (0xa80c bytes) 000338 1752477209.140625 trunc from 0x040000 to 0x01bca2 (0x2435d bytes) 000339 1752477209.170080 MIRROR_RESYNC 2 mirrors 000340 1752477209.605892 mapwrite 0x03c020 thru 0x03ffff (0x3fe0 bytes) 000342 1752477209.740234 mapwrite 0x005e2e thru 0x00c78a (0x695d bytes) 000343 1752477209.785613 MIRROR_RESYNC 2 mirrors 000344 1752477210.384056 read 0x004f77 thru 0x00b461 (0x64eb bytes) 000346 1752477210.396980 read 0x007151 thru 0x00be8a (0x4d3a bytes) 000348 1752477210.399301 read 0x031d3f thru 0x03a732 (0x89f4 bytes) 000349 1752477210.415290 mapread 0x01c2a0 thru 0x01ddab (0x1b0c bytes) 000351 1752477210.420577 write 0x03ea3b thru 0x03ffff (0x15c5 bytes) 000352 1752477210.503209 write 0x013a5f thru 0x015992 (0x1f34 bytes) 000354 1752477210.511354 read 0x02f4d0 thru 0x0334eb (0x401c bytes) 000355 1752477210.533659 mapwrite 0x027717 thru 0x0292c3 (0x1bad bytes) 000357 1752477210.595874 read 0x027dda thru 0x029f1c (0x2143 bytes) 000359 1752477210.597100 trunc from 0x040000 to 0x026534 (0x19acb bytes) 000361 1752477210.629120 MIRROR_RESYNC 2 mirrors 000362 1752477210.943270 read 0x0049ec thru 0x0137c4 (0xedd9 bytes) 000363 1752477210.951250 write 0x003096 thru 0x006a87 (0x39f2 bytes) 000365 1752477211.018997 trunc from 0x026535 to 0x02ec3d (0x8709 bytes) 000366 1752477211.055661 mapwrite 0x0345f4 thru 0x03a5bf (0x5fcc bytes) 000367 1752477211.116775 mapread 0x0109aa thru 0x016963 (0x5fba bytes) 000368 1752477211.124226 mapwrite 0x03d900 thru 0x03ffff (0x2700 bytes) 000370 1752477211.182161 mapread 0x01898f thru 0x020fe0 (0x8652 bytes) 000372 1752477211.209729 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000373 1752477211.314429 mapread 0x03eaa8 thru 0x03ffff (0x1558 bytes) 000380 1752477211.326237 read 0x01e1b2 thru 0x02bc94 (0xdae3 bytes) 000382 1752477211.336856 read 0x027f5c thru 0x031125 (0x91ca bytes) 000384 1752477211.346005 write 0x00a63b thru 0x00cb88 (0x254e bytes) 000385 1752477211.346841 mapread 0x0215ff thru 0x02be53 (0xa855 bytes) 000391 1752477211.357330 mapread 0x02f830 thru 0x03a61c (0xaded bytes) 000395 1752477211.365750 MIRROR_EXTEND to 2 mirrors 000396 1752477211.848612 mapread 0x00d8b8 thru 0x00e7e1 (0xf2a bytes) 000398 1752477211.861692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000399 1752477212.008727 trunc from 0x040000 to 0x019a82 (0x2657d bytes) 000400 1752477212.068279 read 0x00fd75 thru 0x019a82 (0x9d0e bytes) 000401 1752477212.083238 mapwrite 0x026833 thru 0x02f062 (0x8830 bytes) 000402 1752477212.150457 MIRROR_EXTEND to 2 mirrors 000405 1752477212.540898 MIRROR_RESYNC 2 mirrors 000408 1752477212.651995 mapread 0x02e40b thru 0x02f062 (0xc58 bytes) 000409 1752477212.662712 mapread 0x00b538 thru 0x018f6a (0xda33 bytes) 000410 1752477212.683612 mapwrite 0x0139ad thru 0x01f377 (0xb9cb bytes) 000411 1752477212.881468 MIRROR_RESYNC 2 mirrors 000412 1752477213.355079 MIRROR_RESYNC 2 mirrors 000414 1752477213.481457 MIRROR_EXTEND to 3 mirrors 000415 1752477214.029251 mapwrite 0x01a993 thru 0x0293d8 (0xea46 bytes) 000416 1752477214.210921 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000418 1752477214.411694 write 0x003a59 thru 0x0048e5 (0xe8d bytes) 000424 1752477214.443639 mapwrite 0x03a645 thru 0x03c24d (0x1c09 bytes) 000430 1752477214.511510 read 0x034a45 thru 0x03c24d (0x7809 bytes) 000431 1752477214.514532 trunc from 0x03c24e to 0x01a3f2 (0x21e5b bytes) 000432 1752477214.528115 mapread 0x00ea74 thru 0x018378 (0x9905 bytes) 000433 1752477214.560135 MIRROR_RESYNC 2 mirrors 000436 1752477214.962484 read 0x002b65 thru 0x00caaa (0x9f46 bytes) 000437 1752477214.970628 trunc from 0x01a3f3 to 0x00038b (0x1a067 bytes) 000438 1752477215.061980 mapread 0x000093 thru 0x00038b (0x2f9 bytes) 000439 1752477215.068583 trunc from 0x00038c to 0x02f337 (0x2efac bytes) 000441 1752477215.101486 MIRROR_RESYNC 2 mirrors 000442 1752477215.510120 write 0x03b54e thru 0x03c73c (0x11ef bytes) 000443 1752477215.589541 trunc from 0x03c73d to 0x01841e (0x2431e bytes) 000444 1752477215.610181 write 0x00feb8 thru 0x011342 (0x148b bytes) 000445 1752477215.623665 trunc from 0x01841f to 0x00bf9b (0xc483 bytes) 000446 1752477215.650197 write 0x02b628 thru 0x02c431 (0xe0a bytes) 000447 1752477215.653739 write 0x03ecf8 thru 0x03ffff (0x1308 bytes) 000448 1752477215.662797 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000451 1752477215.799002 trunc from 0x040000 to 0x028226 (0x17dd9 bytes) 000452 1752477215.817926 trunc from 0x028227 to 0x00f755 (0x18ad1 bytes) 000454 1752477215.836551 MIRROR_EXTEND to 2 mirrors 000455 1752477216.362958 mapread 0x00c3e3 thru 0x00d135 (0xd53 bytes) 000456 1752477216.378933 read 0x00770d thru 0x00f755 (0x8049 bytes) 000457 1752477216.394084 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000458 1752477216.511675 trunc from 0x00f756 to 0x02cd4c (0x1d5f7 bytes) 000459 1752477216.542912 write 0x018269 thru 0x027c75 (0xfa0d bytes) 000461 1752477216.553165 read 0x00822a thru 0x00da8d (0x5864 bytes) 000462 1752477216.562236 write 0x03ad7d thru 0x03f499 (0x471d bytes) 000465 1752477216.564135 write 0x0364b0 thru 0x039d68 (0x38b9 bytes) 000467 1752477216.572799 mapread 0x038d89 thru 0x03f499 (0x6711 bytes) 000468 1752477216.580461 MIRROR_EXTEND to 2 mirrors 000471 1752477217.038407 write 0x03d7c2 thru 0x03ffff (0x283e bytes) 000473 1752477217.107945 mapwrite 0x01ad48 thru 0x01f57b (0x4834 bytes) 000474 1752477217.170356 read 0x02db1e thru 0x03abfc (0xd0df bytes) 000476 1752477217.177811 read 0x021ca5 thru 0x02975e (0x7aba bytes) 000477 1752477217.192047 trunc from 0x040000 to 0x01d0ec (0x22f13 bytes) 000478 1752477217.228057 write 0x03bcc0 thru 0x03de92 (0x21d3 bytes) 000481 1752477217.229935 mapread 0x014a4a thru 0x01ae3f (0x63f6 bytes) 000482 1752477217.230939 read 0x01f185 thru 0x02ac52 (0xbace bytes) 000485 1752477217.244473 MIRROR_RESYNC 2 mirrors 000486 1752477217.730866 MIRROR_EXTEND to 3 mirrors 000487 1752477218.227363 read 0x00e441 thru 0x014fb1 (0x6b71 bytes) 000488 1752477218.246641 MIRROR_EXTEND to 4 mirrors 000489 1752477218.805205 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000490 1752477219.059917 mapread 0x02fc99 thru 0x039713 (0x9a7b bytes) 000491 1752477219.082851 read 0x0246c3 thru 0x027820 (0x315e bytes) 000492 1752477219.091139 write 0x00dc0b thru 0x010418 (0x280e bytes) 000493 1752477219.144063 trunc from 0x03de93 to 0x01ef7b (0x1ef17 bytes) 000494 1752477219.166468 read 0x0070d3 thru 0x00cce8 (0x5c16 bytes) 000495 1752477219.167980 trunc from 0x01ef7c to 0x0372ee (0x18373 bytes) 000496 1752477219.185986 read 0x00aab8 thru 0x00f30a (0x4853 bytes) 000498 1752477219.190753 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000499 1752477219.434116 read 0x01828d thru 0x01e141 (0x5eb5 bytes) 000502 1752477219.442150 mapread 0x020431 thru 0x026465 (0x6035 bytes) 000503 1752477219.459733 mapwrite 0x015a65 thru 0x021692 (0xbc2e bytes) 000505 1752477219.525627 mapread 0x019ff9 thru 0x026e95 (0xce9d bytes) 000506 1752477219.534736 MIRROR_EXTEND to 3 mirrors 000507 1752477220.053750 MIRROR_RESYNC 3 mirrors 000508 1752477220.409092 mapread 0x0260d9 thru 0x0330a9 (0xcfd1 bytes) 000509 1752477220.443961 read 0x034db0 thru 0x036aa4 (0x1cf5 bytes) 000510 1752477220.444303 mapread 0x01ecdd thru 0x01f506 (0x82a bytes) 000511 1752477220.450834 mapwrite 0x00712c thru 0x011949 (0xa81e bytes) 000514 1752477220.608440 write 0x03291c thru 0x0369b2 (0x4097 bytes) 000517 1752477220.635269 read 0x008426 thru 0x017ec6 (0xfaa1 bytes) 000518 1752477220.644267 MIRROR_EXTEND to 4 mirrors 000519 1752477221.219991 write 0x03f7f0 thru 0x03ffff (0x810 bytes) 000522 1752477221.328239 MIRROR_EXTEND to 5 mirrors 000523 1752477221.955997 write 0x037706 thru 0x03ffff (0x88fa bytes) 000527 1752477222.019240 trunc from 0x040000 to 0x00f054 (0x30fab bytes) 000529 1752477222.055873 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000532 1752477222.243570 read 0x002fcb thru 0x00881e (0x5854 bytes) 000534 1752477222.259416 write 0x01d032 thru 0x023b50 (0x6b1f bytes) 000536 1752477222.265064 trunc from 0x023b51 to 0x0358b3 (0x11d63 bytes) 000537 1752477222.282633 mapread 0x01682f thru 0x019b91 (0x3363 bytes) 000539 1752477222.300924 mapwrite 0x036ce9 thru 0x03ffff (0x9317 bytes) 000541 1752477222.377788 write 0x031a8f thru 0x036701 (0x4c73 bytes) 000542 1752477222.378809 mapwrite 0x037b63 thru 0x03ffff (0x849d bytes) 000543 1752477222.443758 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000545 1752477222.641942 read 0x004d82 thru 0x00d4a4 (0x8723 bytes) 000546 1752477222.655698 mapwrite 0x00cc19 thru 0x014236 (0x761e bytes) 000547 1752477222.736625 MIRROR_RESYNC 3 mirrors 000548 1752477223.360286 read 0x02f36a thru 0x0317a6 (0x243d bytes) 000549 1752477223.369656 write 0x0072cb thru 0x009015 (0x1d4b bytes) 000550 1752477223.435194 trunc from 0x040000 to 0x00346e (0x3cb91 bytes) 000551 1752477223.463130 MIRROR_RESYNC 3 mirrors 000552 1752477223.941741 mapread 0x002ee1 thru 0x00346e (0x58e bytes) 000555 1752477223.952775 trunc from 0x00346f to 0x005548 (0x20da bytes) 000556 1752477224.045604 write 0x01ea3e thru 0x02cb29 (0xe0ec bytes) 000557 1752477224.051502 MIRROR_RESYNC 3 mirrors 000558 1752477224.559232 mapread 0x02423f thru 0x02a9ba (0x677c bytes) 000559 1752477224.574955 trunc from 0x02cb2a to 0x03df3e (0x11415 bytes) 000560 1752477224.636124 mapwrite 0x02e49e thru 0x02f9c0 (0x1523 bytes) 000561 1752477224.693463 mapread 0x026712 thru 0x03028d (0x9b7c bytes) 000562 1752477224.700553 trunc from 0x03df3f to 0x037268 (0x6cd6 bytes) 000563 1752477224.716751 mapread 0x01f991 thru 0x021d75 (0x23e5 bytes) 000564 1752477224.718772 mapwrite 0x01b136 thru 0x01cc0d (0x1ad8 bytes) 000566 1752477224.757014 write 0x031375 thru 0x03d55a (0xc1e6 bytes) 000567 1752477224.758977 trunc from 0x03d55b to 0x0015c1 (0x3bf99 bytes) 000568 1752477224.773444 read 0x0007aa thru 0x0015c1 (0xe18 bytes) 000571 1752477224.784738 MIRROR_RESYNC 3 mirrors 000574 1752477225.231760 mapwrite 0x03b937 thru 0x03ffff (0x46c9 bytes) 000576 1752477225.392840 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000577 1752477225.547159 mapwrite 0x006049 thru 0x0144dd (0xe495 bytes) 000578 1752477225.613898 mapread 0x00b757 thru 0x00b8f8 (0x1a2 bytes) 000580 1752477225.615579 mapread 0x011a23 thru 0x01578f (0x3d6d bytes) 000581 1752477225.621854 MIRROR_RESYNC 2 mirrors 000583 1752477226.016941 trunc from 0x040000 to 0x009911 (0x366ee bytes) 000584 1752477226.075769 mapread 0x004b09 thru 0x009911 (0x4e09 bytes) 000585 1752477226.088052 mapread 0x008999 thru 0x009215 (0x87d bytes) 000588 1752477226.090172 read 0x009497 thru 0x009911 (0x47b bytes) 000590 1752477226.091354 trunc from 0x009912 to 0x02c85d (0x22f4c bytes) 000595 1752477226.110829 mapread 0x026f5f thru 0x028a20 (0x1ac2 bytes) 000597 1752477226.131487 read 0x020d89 thru 0x02c167 (0xb3df bytes) 000599 1752477226.132600 mapread 0x002e63 thru 0x00a971 (0x7b0f bytes) 000601 1752477226.153411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000603 1752477226.300170 read 0x025ba3 thru 0x02c85d (0x6cbb bytes) 000604 1752477226.311868 mapwrite 0x013be8 thru 0x021752 (0xdb6b bytes) 000605 1752477226.384123 mapread 0x01dee7 thru 0x0291b9 (0xb2d3 bytes) 000606 1752477226.385113 mapwrite 0x01fe00 thru 0x029cb5 (0x9eb6 bytes) 000607 1752477226.444657 read 0x00718d thru 0x012fd9 (0xbe4d bytes) 000608 1752477226.453669 mapread 0x005fff thru 0x0093ac (0x33ae bytes) 000609 1752477226.454440 trunc from 0x02c85e to 0x036434 (0x9bd7 bytes) 000610 1752477226.465654 read 0x02002f thru 0x02bca6 (0xbc78 bytes) 000613 1752477226.467264 trunc from 0x036435 to 0x03620b (0x229 bytes) 000615 1752477226.480666 mapwrite 0x03c028 thru 0x03ffff (0x3fd8 bytes) 000622 1752477226.548119 read 0x03d174 thru 0x03ffff (0x2e8c bytes) 000624 1752477226.549048 mapread 0x00248e thru 0x00ec4c (0xc7bf bytes) 000625 1752477226.549928 read 0x010095 thru 0x0193cf (0x933b bytes) 000627 1752477226.552796 mapread 0x02cf27 thru 0x03a871 (0xd94b bytes) 000629 1752477226.556361 mapread 0x02627e thru 0x035d21 (0xfaa4 bytes) 000631 1752477226.558071 mapwrite 0x007f5c thru 0x008552 (0x5f7 bytes) 000632 1752477226.595849 read 0x02101c thru 0x025d2b (0x4d10 bytes) 000633 1752477226.600808 MIRROR_EXTEND to 2 mirrors 000634 1752477227.052909 read 0x01b981 thru 0x023a41 (0x80c1 bytes) 000636 1752477227.059975 write 0x0215ae thru 0x02b3d2 (0x9e25 bytes) 000639 1752477227.134996 MIRROR_RESYNC 2 mirrors 000640 1752477227.557918 mapwrite 0x01b44a thru 0x0214c9 (0x6080 bytes) 000641 1752477227.704723 MIRROR_EXTEND to 3 mirrors 000642 1752477228.228558 mapwrite 0x026674 thru 0x035c85 (0xf612 bytes) 000644 1752477228.442649 trunc from 0x040000 to 0x0088bc (0x37743 bytes) 000645 1752477228.477471 MIRROR_RESYNC 3 mirrors 000651 1752477228.886386 trunc from 0x0088bd to 0x017e02 (0xf546 bytes) 000652 1752477228.935264 read 0x00b7af thru 0x011653 (0x5ea5 bytes) 000653 1752477228.947854 mapwrite 0x001e0b thru 0x00d78d (0xb983 bytes) 000655 1752477229.016116 mapwrite 0x014170 thru 0x017fc6 (0x3e57 bytes) 000656 1752477229.091206 mapwrite 0x01bcbb thru 0x0248f8 (0x8c3e bytes) 000657 1752477229.174119 MIRROR_EXTEND to 4 mirrors 000658 1752477229.623471 write 0x007cc4 thru 0x0171b5 (0xf4f2 bytes) 000661 1752477229.694764 trunc from 0x0248f9 to 0x02f5b8 (0xacc0 bytes) 000662 1752477229.742344 mapwrite 0x02e62e thru 0x035a5d (0x7430 bytes) 000663 1752477229.835176 write 0x03453b thru 0x03c5b8 (0x807e bytes) 000667 1752477229.840178 mapread 0x0208b3 thru 0x023803 (0x2f51 bytes) 000668 1752477229.853476 write 0x03c356 thru 0x03e3f8 (0x20a3 bytes) 000670 1752477229.858077 read 0x01479c thru 0x0191a2 (0x4a07 bytes) 000671 1752477229.863906 mapwrite 0x00c59c thru 0x01924d (0xccb2 bytes) 000672 1752477229.912597 mapwrite 0x00a1b1 thru 0x019ce0 (0xfb30 bytes) 000673 1752477229.956386 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000675 1752477230.123495 write 0x00ff72 thru 0x01ede3 (0xee72 bytes) 000677 1752477230.149981 mapwrite 0x0379a8 thru 0x03ffff (0x8658 bytes) 000678 1752477230.227987 mapwrite 0x017f37 thru 0x02229d (0xa367 bytes) 000681 1752477230.300221 MIRROR_EXTEND to 4 mirrors 000682 1752477230.906683 write 0x02b0ad thru 0x02cf7f (0x1ed3 bytes) 000683 1752477230.992702 mapwrite 0x0330d0 thru 0x03ffff (0xcf30 bytes) 000684 1752477231.048092 mapwrite 0x023a46 thru 0x026d23 (0x32de bytes) 000686 1752477231.087235 trunc from 0x040000 to 0x006957 (0x396a8 bytes) 000689 1752477231.128376 mapread 0x0048af thru 0x006957 (0x20a9 bytes) 000692 1752477231.143930 mapwrite 0x028baa thru 0x03847a (0xf8d1 bytes) 000693 1752477231.237892 write 0x02e65d thru 0x03449e (0x5e42 bytes) 000696 1752477231.243189 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000697 1752477231.461689 mapwrite 0x03d583 thru 0x03ffff (0x2a7d bytes) 000698 1752477231.542512 trunc from 0x040000 to 0x033a5b (0xc5a4 bytes) 000699 1752477231.561971 MIRROR_EXTEND to 4 mirrors 000700 1752477232.155742 trunc from 0x033a5c to 0x03c365 (0x890a bytes) 000701 1752477232.238476 mapwrite 0x023e93 thru 0x027e3e (0x3fac bytes) 000702 1752477232.300073 trunc from 0x03c366 to 0x020d57 (0x1b60e bytes) 000703 1752477232.320756 trunc from 0x020d58 to 0x02819a (0x7443 bytes) 000704 1752477232.332955 mapwrite 0x00b2cb thru 0x00ca66 (0x179c bytes) 000705 1752477232.400269 mapread 0x008335 thru 0x011f84 (0x9c50 bytes) 000707 1752477232.408933 MIRROR_RESYNC 4 mirrors 000710 1752477232.989899 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000711 1752477233.187730 trunc from 0x02819b to 0x0110ff (0x1709b bytes) 000712 1752477233.290693 mapread 0x0016d3 thru 0x00c411 (0xad3f bytes) 000713 1752477233.305376 read 0x00662d thru 0x00ab6b (0x453f bytes) 000715 1752477233.306407 write 0x028140 thru 0x02b3b2 (0x3273 bytes) 000716 1752477233.307356 mapread 0x001fec thru 0x005a25 (0x3a3a bytes) 000717 1752477233.308080 mapwrite 0x01f2c8 thru 0x02cf57 (0xdc90 bytes) 000720 1752477233.426393 MIRROR_EXTEND to 4 mirrors 000721 1752477233.988055 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000722 1752477234.240969 trunc from 0x02cf58 to 0x02e7f2 (0x189b bytes) 000724 1752477234.335829 write 0x02ae1f thru 0x03485b (0x9a3d bytes) 000725 1752477234.352806 MIRROR_RESYNC 3 mirrors 000726 1752477234.840591 write 0x0348cf thru 0x03eb14 (0xa246 bytes) 000727 1752477234.891874 read 0x0149ec thru 0x0173d4 (0x29e9 bytes) 000728 1752477234.901769 MIRROR_EXTEND to 4 mirrors 000729 1752477235.435298 trunc from 0x03eb15 to 0x03e5f3 (0x521 bytes) 000730 1752477235.490724 read 0x03e2a4 thru 0x03e5f3 (0x350 bytes) 000732 1752477235.496612 trunc from 0x03e5f4 to 0x017096 (0x2755d bytes) 000733 1752477235.518265 mapwrite 0x02b214 thru 0x039b77 (0xe964 bytes) 000734 1752477235.638081 read 0x01ad4f thru 0x0279b1 (0xcc63 bytes) 000736 1752477235.654634 mapread 0x01eb1e thru 0x025e18 (0x72fb bytes) 000739 1752477235.656337 trunc from 0x039b78 to 0x012c21 (0x26f56 bytes) 000740 1752477235.675378 read 0x002dc8 thru 0x00ed39 (0xbf72 bytes) 000741 1752477235.676031 trunc from 0x012c22 to 0x01a8e7 (0x7cc6 bytes) 000742 1752477235.689230 write 0x03517f thru 0x03ffff (0xae81 bytes) 000743 1752477235.691688 mapread 0x011745 thru 0x0214a1 (0xfd5d bytes) 000744 1752477235.702180 mapwrite 0x027be5 thru 0x034646 (0xca62 bytes) 000746 1752477235.772015 read 0x033ddc thru 0x03db70 (0x9d95 bytes) 000747 1752477235.772707 mapread 0x0054dd thru 0x01482b (0xf34f bytes) 000750 1752477235.774393 mapread 0x032db0 thru 0x03c222 (0x9473 bytes) 000751 1752477235.782555 MIRROR_EXTEND to 5 mirrors 000753 1752477236.353969 MIRROR_EXTEND to 6 mirrors 000754 1752477236.854561 MIRROR_EXTEND to 7 mirrors 000755 1752477237.358892 trunc from 0x040000 to 0x018aac (0x27553 bytes) 000758 1752477237.484136 mapwrite 0x026a59 thru 0x02aa15 (0x3fbd bytes) 000760 1752477237.618705 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000762 1752477237.875824 trunc from 0x02aa16 to 0x030fc7 (0x65b2 bytes) 000764 1752477237.907634 MIRROR_RESYNC 6 mirrors 000767 1752477238.676519 read 0x0163a8 thru 0x020bfa (0xa853 bytes) 000768 1752477238.689013 mapwrite 0x019af2 thru 0x01fc2b (0x613a bytes) 000769 1752477238.833941 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000770 1752477239.046713 mapwrite 0x02278a thru 0x023277 (0xaee bytes) 000771 1752477239.105258 mapread 0x01b09e thru 0x029d32 (0xec95 bytes) 000772 1752477239.145573 MIRROR_RESYNC 5 mirrors 000773 1752477239.856767 mapwrite 0x021b21 thru 0x023f94 (0x2474 bytes) 000774 1752477239.997721 trunc from 0x030fc8 to 0x0171ab (0x19e1c bytes) 000775 1752477240.057171 mapread 0x00a99d thru 0x013564 (0x8bc8 bytes) 000777 1752477240.074763 write 0x03d6b6 thru 0x03ffff (0x294a bytes) 000778 1752477240.075746 read 0x02143a thru 0x023c5c (0x2823 bytes) 000780 1752477240.085027 MIRROR_EXTEND to 6 mirrors 000782 1752477240.669478 mapwrite 0x00d8c5 thru 0x00edda (0x1516 bytes) 000784 1752477240.854807 mapwrite 0x02224b thru 0x02bf8f (0x9d45 bytes) 000786 1752477240.915411 write 0x024a96 thru 0x02af80 (0x64eb bytes) 000788 1752477240.916824 mapwrite 0x0306e4 thru 0x03c260 (0xbb7d bytes) 000789 1752477240.979548 MIRROR_RESYNC 6 mirrors 000791 1752477241.703502 mapwrite 0x017162 thru 0x021e36 (0xacd5 bytes) 000793 1752477241.925213 mapwrite 0x035fbc thru 0x03ffff (0xa044 bytes) 000794 1752477241.972242 trunc from 0x040000 to 0x029e17 (0x161e8 bytes) 000796 1752477241.997874 mapread 0x0190f8 thru 0x01d91d (0x4826 bytes) 000797 1752477241.998854 mapwrite 0x00bfb5 thru 0x011f6d (0x5fb9 bytes) 000798 1752477242.046068 mapwrite 0x00220f thru 0x0107e1 (0xe5d3 bytes) 000799 1752477242.104226 mapwrite 0x01485a thru 0x0175e2 (0x2d89 bytes) 000803 1752477242.183507 MIRROR_RESYNC 6 mirrors 000806 1752477242.903236 MIRROR_RESYNC 6 mirrors 000807 1752477243.079039 trunc from 0x029e18 to 0x00a432 (0x1f9e5 bytes) 000808 1752477243.172232 mapread 0x007662 thru 0x00a432 (0x2dd1 bytes) 000809 1752477243.200875 read 0x0090fe thru 0x00a432 (0x1335 bytes) 000810 1752477243.201703 write 0x0246b4 thru 0x025faa (0x18f7 bytes) 000811 1752477243.203281 mapwrite 0x0228a5 thru 0x029f82 (0x76de bytes) 000812 1752477243.316106 trunc from 0x029f83 to 0x01e022 (0xbf60 bytes) 000813 1752477243.335444 write 0x02f917 thru 0x03abe5 (0xb2cf bytes) 000815 1752477243.338272 write 0x00d276 thru 0x015619 (0x83a4 bytes) 000816 1752477243.347997 mapwrite 0x01bbd9 thru 0x0257aa (0x9bd2 bytes) 000817 1752477243.406393 mapwrite 0x03f712 thru 0x03ffff (0x8ee bytes) 000818 1752477243.464894 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000819 1752477243.642814 mapwrite 0x019581 thru 0x0238db (0xa35b bytes) 000820 1752477243.705697 MIRROR_EXTEND to 6 mirrors 000824 1752477244.327211 MIRROR_EXTEND to 7 mirrors 000826 1752477244.872896 write 0x03392c thru 0x03ffff (0xc6d4 bytes) 000827 1752477244.939426 mapread 0x00db80 thru 0x01af60 (0xd3e1 bytes) 000828 1752477244.958639 write 0x03c3b9 thru 0x03ffff (0x3c47 bytes) 000829 1752477244.962454 MIRROR_EXTEND to 8 mirrors 000830 1752477245.557675 trunc from 0x040000 to 0x03d917 (0x26e8 bytes) 000832 1752477245.633383 mapread 0x01a6f1 thru 0x01aeb4 (0x7c4 bytes) 000833 1752477245.638131 mapwrite 0x00a94b thru 0x0105f6 (0x5cac bytes) 000834 1752477245.680542 trunc from 0x03d918 to 0x03949e (0x4479 bytes) 000835 1752477245.695727 mapread 0x007181 thru 0x00c5a2 (0x5422 bytes) 000836 1752477245.700232 trunc from 0x03949f to 0x02f045 (0xa459 bytes) 000838 1752477245.719603 trunc from 0x02f046 to 0x03d134 (0xe0ef bytes) 000842 1752477245.742131 mapwrite 0x0214b9 thru 0x029587 (0x80cf bytes) 000843 1752477245.795367 mapwrite 0x03c9d4 thru 0x03ffff (0x362c bytes) 000845 1752477245.870836 trunc from 0x040000 to 0x010f26 (0x2f0d9 bytes) 000846 1752477245.888714 mapwrite 0x036555 thru 0x03ffff (0x9aab bytes) 000847 1752477246.016430 mapread 0x025850 thru 0x033510 (0xdcc1 bytes) 000848 1752477246.044274 mapwrite 0x03e687 thru 0x03ffff (0x1979 bytes) 000850 1752477246.119825 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000855 1752477246.326286 write 0x008fc7 thru 0x011708 (0x8742 bytes) 000856 1752477246.341011 trunc from 0x040000 to 0x0020d4 (0x3df2b bytes) 000857 1752477246.356252 read 0x0003ac thru 0x0020d4 (0x1d29 bytes) 000858 1752477246.365127 trunc from 0x0020d5 to 0x01bfd5 (0x19f01 bytes) 000859 1752477246.385185 MIRROR_EXTEND to 8 mirrors 000860 1752477246.869933 read 0x003033 thru 0x00ea70 (0xba3e bytes) 000861 1752477246.898885 mapwrite 0x00ee23 thru 0x01521f (0x63fd bytes) 000862 1752477247.052410 mapread 0x01a808 thru 0x01bfd5 (0x17ce bytes) 000863 1752477247.067535 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000867 1752477247.299458 write 0x0004c5 thru 0x00ae43 (0xa97f bytes) 000868 1752477247.314063 read 0x00f879 thru 0x01437b (0x4b03 bytes) 000872 1752477247.332833 read 0x012e74 thru 0x019b0a (0x6c97 bytes) 000873 1752477247.341183 read 0x01b0ad thru 0x01bfd5 (0xf29 bytes) 000874 1752477247.341759 mapread 0x019bd3 thru 0x01bfd5 (0x2403 bytes) 000876 1752477247.343029 write 0x026c66 thru 0x0285ed (0x1988 bytes) 000877 1752477247.343925 read 0x007c6b thru 0x00b585 (0x391b bytes) 000879 1752477247.345847 mapwrite 0x03c7e5 thru 0x03ffff (0x381b bytes) 000880 1752477247.434142 trunc from 0x040000 to 0x0271b0 (0x18e4f bytes) 000881 1752477247.473591 MIRROR_RESYNC 7 mirrors 000882 1752477248.275906 write 0x03d507 thru 0x03ffff (0x2af9 bytes) 000883 1752477248.334053 MIRROR_EXTEND to 8 mirrors 000885 1752477248.902423 read 0x0021bd thru 0x00a380 (0x81c4 bytes) 000886 1752477248.938721 MIRROR_RESYNC 8 mirrors 000887 1752477249.751441 read 0x02eafa thru 0x02faed (0xff4 bytes) 000888 1752477249.758818 mapread 0x03cbe8 thru 0x03ffff (0x3418 bytes) 000889 1752477249.771442 trunc from 0x040000 to 0x020c35 (0x1f3ca bytes) 000890 1752477249.845829 mapread 0x01309f thru 0x01b352 (0x82b4 bytes) 000891 1752477249.853626 write 0x0046a1 thru 0x0049fc (0x35c bytes) 000892 1752477249.861350 mapwrite 0x03a283 thru 0x03ffff (0x5d7d bytes) 000893 1752477249.937867 mapwrite 0x02214c thru 0x022493 (0x348 bytes) 000894 1752477250.011001 trunc from 0x040000 to 0x03c107 (0x3ef8 bytes) 000896 1752477250.031586 mapwrite 0x014abe thru 0x01be8d (0x73d0 bytes) 000897 1752477250.082952 trunc from 0x03c108 to 0x01d6c1 (0x1ea46 bytes) 000898 1752477250.108302 write 0x02b520 thru 0x02e4b3 (0x2f94 bytes) 000901 1752477250.130602 MIRROR_RESYNC 8 mirrors 000903 1752477251.130914 write 0x0278ce thru 0x02d44b (0x5b7e bytes) 000906 1752477251.209057 mapread 0x00b26c thru 0x00bfa6 (0xd3b bytes) 000911 1752477251.222927 trunc from 0x02e4b4 to 0x02d20b (0x12a8 bytes) 000912 1752477251.258410 MIRROR_RESYNC 8 mirrors 000914 1752477252.215569 MIRROR_RESYNC 8 mirrors 000918 1752477252.409516 mapread 0x0074f7 thru 0x00f080 (0x7b8a bytes) 000919 1752477252.433755 trunc from 0x02d20c to 0x02512e (0x80dd bytes) 000920 1752477252.492023 read 0x002e00 thru 0x0111de (0xe3df bytes) 000922 1752477252.498551 write 0x023c9d thru 0x032170 (0xe4d4 bytes) 000923 1752477252.506860 mapread 0x0084fe thru 0x00c6b3 (0x41b6 bytes) 000924 1752477252.507633 mapwrite 0x011620 thru 0x0196e4 (0x80c5 bytes) 000926 1752477252.565791 mapwrite 0x036151 thru 0x03a2ed (0x419d bytes) 000927 1752477252.633251 mapwrite 0x00e9b4 thru 0x015843 (0x6e90 bytes) 000929 1752477252.678243 MIRROR_RESYNC 8 mirrors 000930 1752477253.445558 write 0x01e3f0 thru 0x02d4ac (0xf0bd bytes) 000931 1752477253.524718 read 0x0342b4 thru 0x039867 (0x55b4 bytes) 000932 1752477253.529581 write 0x02115e thru 0x0265ae (0x5451 bytes) 000934 1752477253.530944 mapwrite 0x035c87 thru 0x03bc08 (0x5f82 bytes) 000938 1752477253.620860 MIRROR_RESYNC 8 mirrors 000939 1752477254.385219 read 0x02e6b0 thru 0x0378eb (0x923c bytes) 000940 1752477254.395241 mapread 0x00af16 thru 0x00d60c (0x26f7 bytes) 000941 1752477254.420496 mapread 0x0130a0 thru 0x01ae23 (0x7d84 bytes) 000942 1752477254.434609 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) 000943 1752477254.680685 trunc from 0x03bc09 to 0x0189f9 (0x2320f bytes) 000946 1752477254.740227 read 0x00c573 thru 0x00ce9f (0x92d bytes) 000949 1752477254.744506 mapwrite 0x03679b thru 0x03748a (0xcf0 bytes) 000950 1752477254.834864 trunc from 0x03748b to 0x004fb0 (0x324da bytes) 000952 1752477254.859734 read 0x004bae thru 0x004fb0 (0x403 bytes) 000955 1752477254.870238 trunc from 0x004fb1 to 0x015f8e (0x10fde bytes) 000957 1752477254.890183 read 0x0148fa thru 0x015f8e (0x1695 bytes) 000958 1752477254.899778 trunc from 0x015f8f to 0x00b09b (0xaef3 bytes) 000961 1752477254.918773 trunc from 0x00b09c to 0x039b44 (0x2eaa9 bytes) 000962 1752477254.935114 mapread 0x02f554 thru 0x0378a5 (0x8352 bytes) 000964 1752477254.944492 read 0x029ce4 thru 0x03603b (0xc358 bytes) 000965 1752477254.945066 mapread 0x012bc2 thru 0x017baa (0x4fe9 bytes) 000966 1752477254.950659 mapwrite 0x00ea04 thru 0x012760 (0x3d5d bytes) 000968 1752477254.983497 mapread 0x0088a3 thru 0x0095f6 (0xd54 bytes) 000969 1752477254.984309 mapwrite 0x03a9f0 thru 0x03ffff (0x5610 bytes) 000971 1752477255.038954 mapread 0x03d41a thru 0x03ffff (0x2be6 bytes) 000974 1752477255.040629 read 0x01c041 thru 0x020f06 (0x4ec6 bytes) 000976 1752477255.042424 write 0x03e0a3 thru 0x03ffff (0x1f5d bytes) 000977 1752477255.044523 trunc from 0x040000 to 0x0264e9 (0x19b16 bytes) 000978 1752477255.059165 trunc from 0x0264ea to 0x0235d1 (0x2f18 bytes) 000979 1752477255.069792 write 0x00d929 thru 0x01cd1f (0xf3f7 bytes) 000982 1752477255.072005 mapwrite 0x025054 thru 0x02e2df (0x928c bytes) 000985 1752477255.143153 read 0x002bb0 thru 0x00b573 (0x89c4 bytes) 000986 1752477255.143826 write 0x026bed thru 0x0347ab (0xdbbf bytes) 000987 1752477255.145395 mapread 0x003193 thru 0x00c316 (0x9184 bytes) 000990 1752477255.147717 mapwrite 0x004291 thru 0x010424 (0xc194 bytes) 000991 1752477255.193782 trunc from 0x0347ac to 0x032d17 (0x1a94 bytes) 000994 1752477255.206543 mapwrite 0x02e0da thru 0x035252 (0x7179 bytes) 000996 1752477255.268013 read 0x031ff3 thru 0x035252 (0x3260 bytes) 000997 1752477255.268654 write 0x038f08 thru 0x03afbf (0x20b8 bytes) 000999 1752477255.270046 mapwrite 0x0006eb thru 0x00c6bf (0xbfd5 bytes) All operations completed A-OK! PASS 70a (72s) == sanity-flr test 71: check mirror extend parent fid ==== 03:14:18 (1752477258) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 03:14:20 (1752477260) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6909408 .. Extending file size to 8017536 .. Extending file size to 9111328 .. Extending file size to 9213888 .. Extending file size to 7527904 .. Extending file size to 7966176 .. Extending file size to 8233984 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8834720 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 55011 55012 55015 55017 55018 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:15:33 (1752477333) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3087424 .. Extending file size to 6063648 .. failed Extending file size to 4400960 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7361600 .. Extending file size to 6167584 .. Extending file size to 8147072 .. Extending file size to 7583296 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8643264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8995936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8803264 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9020288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9218432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8835392 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8905664 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9086208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57352 57353 57356 57357 57358 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:16:45 (1752477405) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0550719 s, 38.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.618014 s, 3.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:17:07 (1752477427) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13405 s, 15.6 MB/s PASS 202 (11s) == sanity-flr test 203: mirror file preserve mirror ID === 03:17:18 (1752477438) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:17:30 (1752477450) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265199 s, 7.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:17:39 (1752477459) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245022 s, 8.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:17:47 (1752477467) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151193 s, 13.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:18:18 (1752477498) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960668 s, 10.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:19:12 (1752477552) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132606 s, 15.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:20:07 (1752477607) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. Creating new pool test_204f oleg356-server: Pool lustre.test_204f created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184554 s, 11.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg356-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:21:21 (1752477681) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:21:27 (1752477687) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:21:35 (1752477695) oleg356-server: Pool lustre.test_206 created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg356-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:21:59 (1752477719) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101855 s, 10.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038483 s, 27.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf26c5544 Mirror 3: 0xf26c5544 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:22:07 (1752477727) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:22:11 (1752477731) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:22:14 (1752477734) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0643231 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2030000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2400000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc40000 Read this (4 bytes): BBBB 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:22:48 (1752477768) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0394763 s, 208 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:22:54 (1752477774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545126 s, 19.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27f lmm_fid: [0x200000404:0x27f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27f lmm_fid: [0x200000404:0x27f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834559 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-14 03:22:56.000000000 -0400 Modify: 2025-07-14 03:22:56.000000000 -0400 Change: 2025-07-14 03:22:57.000000000 -0400 Birth: 2025-07-14 03:22:55.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:23:02 (1752477782) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:23:05 (1752477785) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.49565 s, 82.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.084703 s, 48.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 14 03:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x281:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 14 03:23 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test complete, duration 1051 sec =========== 03:23:13 (1752477793) === sanity-flr: start cleanup 03:23:15 (1752477795) === === sanity-flr: finish cleanup 03:23:21 (1752477801) ===