-----============= acceptance-small: sanity-flr ============----- Fri Oct 24 01:21:34 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg429-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:21:41 (1761283301) === oleg429-client.virtnet: executing check_config_client /mnt/lustre oleg429-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg429-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d4b4552b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d4b4552b800.idle_timeout=debug disable quota as required oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:21:52 (1761283312) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7547] == sanity-flr test 0a: lfs mirror create with -N option == 01:21:53 (1761283313) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:21:58 (1761283318) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:22:00 (1761283320) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:22:01 (1761283321) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:22:09 (1761283329) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:22:11 (1761283331) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:22:12 (1761283332) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d4b4552b800.stats=clear osc.lustre-OST0001-osc-ffff9d4b4552b800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:22:15 (1761283335) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:22:19 (1761283339) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:22:22 (1761283342) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:22:24 (1761283344) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 01:22:27 (1761283347) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:22:30 (1761283350) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:22:31 (1761283351) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:22:34 (1761283354) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:22:38 (1761283358) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:22:40 (1761283360) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.851592 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564106 s, 18.6 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:22:45 (1761283365) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.11116 s, 104 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:22:48 (1761283368) Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:23:19 (1761283399) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.188737 s, 50.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:23:33 (1761283413) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:24:00 (1761283440) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32642 s, 39.5 MB/s -rw-r--r-- 1 root root 52428800 Oct 24 01:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.264s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.368s user 0m0.001s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.324s user 0m0.000s sys 0m0.176s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d4b4552b800.ost_server_uuid in FULL state after 0 sec PASS 33b (29s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:24:28 (1761283468) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:24:30 (1761283470) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:24:31 (1761283471) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:24:32 (1761283472) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 01:24:35 (1761283475) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.205875 s, 61.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:24:40 (1761283480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136472 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163373 s, 64.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:24:43 (1761283483) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:24:46 (1761283486) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000673301 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00077517 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000737322 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000692883 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 01:24:50 (1761283490) create test files with size 15746670 .. 63787748476a89e272a83d689bec22a1 - 024a72a8552b2bd0af24e9115878538c - bc2d56269bff79533f4f120c2e78d89a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 01:24:57 (1761283497) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:25:02 (1761283502) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:25:04 (1761283504) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376997 s, 27.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120367 s, 87.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 01:25:07 (1761283507) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423411 s, 49.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.126848 s, 33.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0592499 s, 35.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 01:25:11 (1761283511) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 01:25:13 (1761283513) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:25:14 (1761283514) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208466 s, 50.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 01:25:17 (1761283517) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:25:18 (1761283518) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:25:21 (1761283521) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110527 s, 94.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:25:42 (1761283542) llite.lustre-ffff9d4b697e0800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0916209 s, 114 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9d4b44df1000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:25:46 (1761283546) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (14s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:26:00 (1761283560) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:26:13 (1761283573) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:26:14 (1761283574) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.104326 s, 30.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0744697 s, 42.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b31bf0dcf231ab39d4ca7c613c4d55de - mirror 1 checksum: 4acdb4b4344ea859f2ef1a14575a1a9f - mirror 2 checksum: b31bf0dcf231ab39d4ca7c613c4d55de - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 24 01:26:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 24 01:26:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b31bf0dcf231ab39d4ca7c613c4d55de - mirror 1 checksum: 4acdb4b4344ea859f2ef1a14575a1a9f - mirror 2 checksum: 4acdb4b4344ea859f2ef1a14575a1a9f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 24 01:26:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 24 01:26:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:26:19 (1761283579) Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 01:26:22 (1761283582) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server Failover mds1 to oleg429-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 01:26:36 (1761283596) targets are mounted 01:26:36 (1761283596) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144589 s, 72.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:26:43 (1761283603) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:26:44 (1761283604) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:26:45 (1761283605) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00447342 s, 3.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:26:48 (1761283608) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:26:49 (1761283609) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:26:51 (1761283611) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:26:59 (1761283619) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:27:19 (1761283639) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:27:36 (1761283656) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:27:55 (1761283675) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 01:27:58 (1761283678) mirror create pid 52175 mirror split pid 52176 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52175 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52176 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 01:29:00 (1761283740) Seed set to 3741 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002 1761283741.501797 mapwrite 0x020afb thru 0x02a680 (0x9b86 bytes) 000003 1761283741.556479 mapwrite 0x013bc0 thru 0x01f630 (0xba71 bytes) 000005 1761283741.591552 mapwrite 0x03759c thru 0x03e249 (0x6cae bytes) 000006 1761283741.646790 read 0x01a794 thru 0x01ee6d (0x46da bytes) truncating to largest ever: 0x18c80 000010 1761283741.648369 trunc from 0x03e24a to 0x018c7f (0x255ca bytes) 000011 1761283741.655490 write 0x02c6cc thru 0x032502 (0x5e37 bytes) 000013 1761283741.659614 MIRROR_EXTEND to 2 mirrors 000014 1761283741.911373 read 0x02cafc thru 0x032502 (0x5a07 bytes) 000015 1761283741.915631 write 0x02c773 thru 0x02d120 (0x9ae bytes) 000017 1761283741.958079 MIRROR_RESYNC 2 mirrors 000018 1761283742.204070 read 0x01e191 thru 0x022eb1 (0x4d21 bytes) 000020 1761283742.208468 trunc from 0x032503 to 0x0135e2 (0x1ef20 bytes) 000022 1761283742.245794 mapread 0x002ae2 thru 0x01265c (0xfb7b bytes) 000023 1761283742.264645 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000024 1761283742.335109 write 0x03febb thru 0x03ffff (0x145 bytes) 000026 1761283742.336392 mapread 0x039786 thru 0x03f1ac (0x5a27 bytes) 000031 1761283742.350738 read 0x00fce0 thru 0x015a20 (0x5d41 bytes) 000032 1761283742.355369 write 0x015a4d thru 0x01ff16 (0xa4ca bytes) 000033 1761283742.359259 read 0x02a7cc thru 0x02f4f3 (0x4d28 bytes) truncating to largest ever: 0x2deea 000034 1761283742.364801 trunc from 0x040000 to 0x02dee9 (0x12116 bytes) 000035 1761283742.372352 mapwrite 0x0383d1 thru 0x03ffff (0x7c2f bytes) 000036 1761283742.427495 MIRROR_EXTEND to 2 mirrors 000037 1761283742.661284 write 0x01a249 thru 0x02156b (0x7323 bytes) 000038 1761283742.684617 trunc from 0x040000 to 0x026271 (0x19d8e bytes) 000040 1761283742.703548 mapwrite 0x002b7c thru 0x011105 (0xe58a bytes) 000041 1761283742.734902 read 0x01467a thru 0x017c59 (0x35e0 bytes) 000042 1761283742.735468 write 0x027bf5 thru 0x02a378 (0x2784 bytes) 000043 1761283742.736416 mapread 0x021664 thru 0x02a26c (0x8c09 bytes) 000045 1761283742.742510 mapread 0x01673d thru 0x0244d8 (0xdd9c bytes) 000046 1761283742.743429 trunc from 0x02a379 to 0x023975 (0x6a03 bytes) 000047 1761283742.754283 MIRROR_EXTEND to 3 mirrors 000048 1761283742.967800 mapwrite 0x038b31 thru 0x03ffff (0x74cf bytes) truncating to largest ever: 0x2fcce 000049 1761283743.028509 trunc from 0x040000 to 0x02fccd (0x10332 bytes) 000050 1761283743.035530 trunc from 0x02fcce to 0x00395c (0x2c371 bytes) 000051 1761283743.040978 read 0x003803 thru 0x00395c (0x15a bytes) 000052 1761283743.043137 mapread 0x00004c thru 0x00395c (0x3911 bytes) 000057 1761283743.047616 write 0x01720c thru 0x01d8e2 (0x66d7 bytes) 000058 1761283743.048765 mapread 0x00c9ad thru 0x01548a (0x8ade bytes) 000059 1761283743.055999 mapread 0x00470c thru 0x00e2d5 (0x9bca bytes) 000062 1761283743.060467 write 0x034eb8 thru 0x03b2ad (0x63f6 bytes) 000063 1761283743.061695 mapwrite 0x012778 thru 0x017d8e (0x5617 bytes) 000065 1761283743.086547 MIRROR_EXTEND to 4 mirrors 000067 1761283743.307825 mapread 0x019b5f thru 0x023736 (0x9bd8 bytes) 000068 1761283743.320832 trunc from 0x03b2ae to 0x01805d (0x23250 bytes) 000069 1761283743.352285 mapread 0x0142c3 thru 0x01805d (0x3d9b bytes) 000072 1761283743.358676 MIRROR_EXTEND to 5 mirrors 000073 1761283743.579293 mapwrite 0x0194b7 thru 0x01e66c (0x51b6 bytes) 000074 1761283743.654231 trunc from 0x01e66d to 0x00ecaf (0xf9bd bytes) 000077 1761283743.666663 write 0x0061e0 thru 0x00ba1f (0x5840 bytes) 000078 1761283743.671463 read 0x00c06a thru 0x00ecaf (0x2c46 bytes) 000079 1761283743.672031 read 0x0037a1 thru 0x00ecaf (0xb50f bytes) 000080 1761283743.672584 mapread 0x00a8c3 thru 0x00c7a3 (0x1ee1 bytes) 000082 1761283743.676498 MIRROR_RESYNC 5 mirrors 000083 1761283744.016942 MIRROR_EXTEND to 6 mirrors 000085 1761283744.266447 write 0x00a628 thru 0x00ef26 (0x48ff bytes) 000087 1761283744.303134 mapwrite 0x023ca7 thru 0x027231 (0x358b bytes) 000088 1761283744.345159 write 0x03ada9 thru 0x03ffff (0x5257 bytes) 000090 1761283744.346665 trunc from 0x040000 to 0x01fc75 (0x2038a bytes) 000093 1761283744.355069 read 0x015ced thru 0x018231 (0x2545 bytes) 000095 1761283744.359125 mapwrite 0x0082ba thru 0x00a160 (0x1ea7 bytes) 000096 1761283744.384107 mapread 0x00cda9 thru 0x010a60 (0x3cb8 bytes) 000098 1761283744.385392 read 0x00d9b8 thru 0x012b4e (0x5197 bytes) 000099 1761283744.390120 mapread 0x01600d thru 0x01fc75 (0x9c69 bytes) 000100 1761283744.391216 trunc from 0x01fc76 to 0x02b737 (0xbac2 bytes) 000102 1761283744.397683 mapwrite 0x024ad6 thru 0x027554 (0x2a7f bytes) 000104 1761283744.424832 write 0x025bf9 thru 0x026abd (0xec5 bytes) 000105 1761283744.425632 write 0x01ef3a thru 0x021681 (0x2748 bytes) 000107 1761283744.426690 write 0x0310ca thru 0x032ae1 (0x1a18 bytes) 000108 1761283744.427412 mapwrite 0x001dac thru 0x0078a2 (0x5af7 bytes) 000109 1761283744.449028 write 0x03ea4c thru 0x03ffff (0x15b4 bytes) 000110 1761283744.449973 mapread 0x0151f6 thru 0x01dd92 (0x8b9d bytes) 000111 1761283744.450764 read 0x03e4f4 thru 0x03ffff (0x1b0c bytes) 000112 1761283744.451290 trunc from 0x040000 to 0x02653f (0x19ac0 bytes) 000113 1761283744.460247 mapwrite 0x010b56 thru 0x012bf4 (0x209f bytes) 000114 1761283744.479157 write 0x004abe thru 0x012af3 (0xe036 bytes) 000115 1761283744.480354 read 0x003e34 thru 0x00bed9 (0x80a6 bytes) 000116 1761283744.480867 mapread 0x01764e thru 0x01aa65 (0x3418 bytes) truncating to largest ever: 0x3fc2b 000117 1761283744.481570 trunc from 0x026540 to 0x03fc2a (0x196eb bytes) 000119 1761283744.490314 write 0x027fb6 thru 0x0310cd (0x9118 bytes) 000121 1761283744.498654 MIRROR_EXTEND to 7 mirrors 000122 1761283744.727850 write 0x022ddc thru 0x02cc38 (0x9e5d bytes) 000123 1761283744.755363 read 0x00fc3e thru 0x01ab1b (0xaede bytes) 000124 1761283744.759158 trunc from 0x03fc2b to 0x02dc73 (0x11fb7 bytes) 000125 1761283744.772113 mapread 0x0002bc thru 0x007617 (0x735c bytes) 000128 1761283744.777237 mapread 0x0224f2 thru 0x02bba4 (0x96b3 bytes) 000129 1761283744.778068 mapwrite 0x016fb3 thru 0x01c853 (0x58a1 bytes) 000133 1761283744.800262 mapwrite 0x00c661 thru 0x01b431 (0xedd1 bytes) 000134 1761283744.821126 read 0x01ce5d thru 0x01f807 (0x29ab bytes) 000135 1761283744.821669 trunc from 0x02dc74 to 0x03a187 (0xc514 bytes) 000137 1761283744.827625 mapwrite 0x02c365 thru 0x034d4e (0x89ea bytes) 000138 1761283744.848188 MIRROR_RESYNC 7 mirrors 000139 1761283745.194879 mapread 0x01e96d thru 0x01fd73 (0x1407 bytes) 000142 1761283745.199334 trunc from 0x03a188 to 0x02a2be (0xfec9 bytes) 000143 1761283745.226215 read 0x012262 thru 0x016cf0 (0x4a8f bytes) 000145 1761283745.230632 read 0x029f93 thru 0x02a2be (0x32c bytes) 000147 1761283745.235422 write 0x0060f3 thru 0x0117a4 (0xb6b2 bytes) 000149 1761283745.236834 mapread 0x016cab thru 0x021626 (0xa97c bytes) 000151 1761283745.237868 mapread 0x01e0aa thru 0x01edd2 (0xd29 bytes) 000154 1761283745.239212 write 0x03d288 thru 0x03ffff (0x2d78 bytes) 000155 1761283745.239910 mapread 0x01a710 thru 0x01c84a (0x213b bytes) 000156 1761283745.240533 read 0x00b35f thru 0x0152e8 (0x9f8a bytes) 000157 1761283745.240967 mapread 0x03e7d3 thru 0x03ffff (0x182d bytes) 000158 1761283745.241652 read 0x0078db thru 0x00cc20 (0x5346 bytes) 000162 1761283745.243120 trunc from 0x040000 to 0x018f8f (0x27070 bytes) 000163 1761283745.251909 write 0x038bce thru 0x03d354 (0x4787 bytes) 000164 1761283745.252999 trunc from 0x03d355 to 0x007975 (0x359df bytes) 000166 1761283745.263341 MIRROR_RESYNC 7 mirrors 000167 1761283745.586940 mapread 0x001c91 thru 0x007975 (0x5ce5 bytes) 000170 1761283745.593545 mapwrite 0x023416 thru 0x02ecc6 (0xb8b1 bytes) 000171 1761283745.650682 mapwrite 0x02defd thru 0x0300df (0x21e3 bytes) 000173 1761283745.676488 mapread 0x00c4e0 thru 0x010640 (0x4161 bytes) 000175 1761283745.680885 write 0x014887 thru 0x01a587 (0x5d01 bytes) 000176 1761283745.686822 write 0x03e76f thru 0x03ffff (0x1891 bytes) 000177 1761283745.687530 mapwrite 0x01004d thru 0x0187ea (0x879e bytes) 000179 1761283745.707313 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000180 1761283745.781420 read 0x0243ae thru 0x0297cc (0x541f bytes) 000182 1761283745.785249 trunc from 0x040000 to 0x00f3f1 (0x30c0e bytes) 000183 1761283745.792885 MIRROR_RESYNC 6 mirrors 000185 1761283746.083408 trunc from 0x00f3f2 to 0x03aaf1 (0x2b700 bytes) 000186 1761283746.116698 MIRROR_EXTEND to 7 mirrors 000187 1761283746.334142 read 0x012863 thru 0x01f67e (0xce1c bytes) 000188 1761283746.340401 mapwrite 0x00e77b thru 0x01d04f (0xe8d5 bytes) 000190 1761283746.419788 read 0x0194fb thru 0x01f3d2 (0x5ed8 bytes) 000192 1761283746.420719 write 0x025201 thru 0x02bf98 (0x6d98 bytes) 000193 1761283746.426181 write 0x006848 thru 0x0163f0 (0xfba9 bytes) 000195 1761283746.427842 mapwrite 0x03ae34 thru 0x03ffff (0x51cc bytes) 000196 1761283746.465675 MIRROR_EXTEND to 8 mirrors 000197 1761283746.658998 MIRROR_RESYNC 8 mirrors 000198 1761283747.032293 write 0x0075d4 thru 0x013a4e (0xc47b bytes) 000199 1761283747.061315 read 0x02dbba thru 0x03125b (0x36a2 bytes) 000201 1761283747.065906 mapwrite 0x001b05 thru 0x00763c (0x5b38 bytes) 000203 1761283747.092387 read 0x0095a8 thru 0x00c561 (0x2fba bytes) 000209 1761283747.097496 MIRROR_RESYNC 8 mirrors 000212 1761283747.565870 mapread 0x017b9a thru 0x018699 (0xb00 bytes) 000214 1761283747.570244 mapread 0x009587 thru 0x009b42 (0x5bc bytes) 000218 1761283747.574289 mapwrite 0x039255 thru 0x03ffff (0x6dab bytes) 000219 1761283747.632078 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000220 1761283747.719122 trunc from 0x040000 to 0x000eac (0x3f153 bytes) 000221 1761283747.735700 trunc from 0x000ead to 0x023482 (0x225d6 bytes) 000222 1761283747.744694 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000224 1761283747.825170 write 0x015048 thru 0x018238 (0x31f1 bytes) 000226 1761283747.830387 trunc from 0x023483 to 0x030ea4 (0xda22 bytes) 000230 1761283747.840706 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000233 1761283747.928345 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000234 1761283748.005717 read 0x00d5bd thru 0x00d837 (0x27b bytes) 000235 1761283748.008383 mapread 0x02071e thru 0x0236e8 (0x2fcb bytes) 000238 1761283748.017554 mapread 0x00cc74 thru 0x01b547 (0xe8d4 bytes) 000239 1761283748.025996 read 0x00f444 thru 0x01ec14 (0xf7d1 bytes) 000240 1761283748.026621 read 0x0214bd thru 0x02e8dc (0xd420 bytes) 000241 1761283748.031257 read 0x012822 thru 0x01be49 (0x9628 bytes) 000244 1761283748.032814 mapread 0x00be81 thru 0x015584 (0x9704 bytes) 000245 1761283748.033459 trunc from 0x030ea5 to 0x039366 (0x84c2 bytes) 000246 1761283748.039987 write 0x02f5dc thru 0x038185 (0x8baa bytes) 000247 1761283748.043093 read 0x013f72 thru 0x01dfea (0xa079 bytes) 000248 1761283748.043625 write 0x021cf9 thru 0x02650c (0x4814 bytes) 000250 1761283748.044710 trunc from 0x039367 to 0x03ece7 (0x5981 bytes) 000251 1761283748.050879 read 0x02e864 thru 0x03d7ad (0xef4a bytes) 000253 1761283748.055154 mapwrite 0x02df17 thru 0x0384a7 (0xa591 bytes) 000254 1761283748.082684 MIRROR_RESYNC 4 mirrors 000255 1761283748.370651 mapwrite 0x03edb0 thru 0x03ffff (0x1250 bytes) 000256 1761283748.436460 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000257 1761283748.531229 trunc from 0x040000 to 0x001f7b (0x3e084 bytes) 000259 1761283748.537505 trunc from 0x001f7c to 0x03be76 (0x39efb bytes) 000261 1761283748.543647 mapread 0x03b7ef thru 0x03be76 (0x688 bytes) 000264 1761283748.548557 mapread 0x0281b3 thru 0x02d301 (0x514f bytes) 000266 1761283748.555526 mapwrite 0x03b495 thru 0x03ffff (0x4b6b bytes) 000267 1761283748.582118 mapwrite 0x02f656 thru 0x03e901 (0xf2ac bytes) 000268 1761283748.602019 mapwrite 0x00fd7d thru 0x018897 (0x8b1b bytes) 000269 1761283748.628289 read 0x03c36b thru 0x03ffff (0x3c95 bytes) 000270 1761283748.628922 mapread 0x03e2a2 thru 0x03ffff (0x1d5e bytes) 000271 1761283748.629700 mapread 0x030684 thru 0x0339bb (0x3338 bytes) 000272 1761283748.630503 write 0x03571b thru 0x03ffff (0xa8e5 bytes) 000273 1761283748.634123 MIRROR_RESYNC 3 mirrors 000274 1761283748.869404 trunc from 0x040000 to 0x01a250 (0x25daf bytes) 000276 1761283748.904589 read 0x01765b thru 0x01a250 (0x2bf6 bytes) 000278 1761283748.908388 read 0x013532 thru 0x01a250 (0x6d1f bytes) 000280 1761283748.913198 mapwrite 0x0287b3 thru 0x032c3b (0xa489 bytes) 000282 1761283748.943503 MIRROR_EXTEND to 4 mirrors 000283 1761283749.165293 mapwrite 0x00c3a3 thru 0x017008 (0xac66 bytes) 000284 1761283749.253937 read 0x01c521 thru 0x01fc16 (0x36f6 bytes) 000285 1761283749.257369 mapwrite 0x0249c3 thru 0x033564 (0xeba2 bytes) 000288 1761283749.288508 read 0x01816b thru 0x01b7fb (0x3691 bytes) 000289 1761283749.288957 read 0x02d617 thru 0x033564 (0x5f4e bytes) 000292 1761283749.290073 write 0x0142ca thru 0x018082 (0x3db9 bytes) 000293 1761283749.292508 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000294 1761283749.372228 read 0x01e094 thru 0x0298f6 (0xb863 bytes) 000296 1761283749.376695 mapwrite 0x02f5c6 thru 0x039d66 (0xa7a1 bytes) 000297 1761283749.411246 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000298 1761283749.513072 trunc from 0x039d67 to 0x01ea3b (0x1b32b bytes) 000299 1761283749.525102 MIRROR_RESYNC 2 mirrors 000301 1761283749.763271 read 0x018631 thru 0x01d49e (0x4e6e bytes) 000302 1761283749.769857 MIRROR_RESYNC 2 mirrors 000304 1761283749.853464 write 0x00bde6 thru 0x0111bb (0x53d6 bytes) 000305 1761283749.881488 write 0x03e036 thru 0x03ffff (0x1fca bytes) 000306 1761283749.882562 trunc from 0x040000 to 0x03667a (0x9985 bytes) 000307 1761283749.898289 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000310 1761283749.954430 read 0x01e00d thru 0x0266e6 (0x86da bytes) 000311 1761283749.958328 read 0x000d98 thru 0x0061bd (0x5426 bytes) 000314 1761283749.964388 mapwrite 0x01eefa thru 0x02b991 (0xca98 bytes) 000317 1761283749.993596 mapwrite 0x03eeca thru 0x03ffff (0x1136 bytes) 000318 1761283750.028153 mapread 0x016bf5 thru 0x01eeb9 (0x82c5 bytes) 000319 1761283750.029064 write 0x0244fc thru 0x024b4c (0x651 bytes) 000323 1761283750.030999 trunc from 0x040000 to 0x01a6af (0x25950 bytes) 000324 1761283750.040129 mapread 0x00d016 thru 0x00f4b2 (0x249d bytes) 000325 1761283750.041009 mapwrite 0x0148a5 thru 0x021604 (0xcd60 bytes) 000326 1761283750.085303 trunc from 0x021605 to 0x030b46 (0xf542 bytes) 000327 1761283750.091154 mapwrite 0x02935a thru 0x02e721 (0x53c8 bytes) 000328 1761283750.118201 trunc from 0x030b47 to 0x019a57 (0x170ef bytes) 000329 1761283750.124508 read 0x00342c thru 0x012673 (0xf248 bytes) 000330 1761283750.125150 write 0x03efaa thru 0x03ffff (0x1056 bytes) 000331 1761283750.125953 trunc from 0x040000 to 0x00049d (0x3fb62 bytes) 000336 1761283750.133702 write 0x038f4a thru 0x03fa99 (0x6b50 bytes) 000337 1761283750.136023 mapread 0x0229c3 thru 0x029658 (0x6c96 bytes) 000339 1761283750.141496 read 0x0292e1 thru 0x033c05 (0xa925 bytes) 000340 1761283750.142068 read 0x01e652 thru 0x0286cf (0xa07e bytes) 000342 1761283750.143216 write 0x02f889 thru 0x03dbfe (0xe376 bytes) 000345 1761283750.145063 write 0x01c852 thru 0x021189 (0x4938 bytes) 000355 1761283750.149028 mapwrite 0x0142a2 thru 0x01ca82 (0x87e1 bytes) 000357 1761283750.167331 read 0x01e7a0 thru 0x01fe1b (0x167c bytes) 000359 1761283750.168484 write 0x008b69 thru 0x016e66 (0xe2fe bytes) 000360 1761283750.169509 trunc from 0x03fa9a to 0x031b8d (0xdf0c bytes) 000361 1761283750.178571 mapwrite 0x028b6c thru 0x03582c (0xccc1 bytes) 000364 1761283750.210014 MIRROR_EXTEND to 2 mirrors 000366 1761283750.415998 mapread 0x035771 thru 0x03582c (0xbc bytes) 000367 1761283750.419344 mapwrite 0x00b322 thru 0x0121d8 (0x6eb7 bytes) 000368 1761283750.500918 mapwrite 0x000014 thru 0x0005f2 (0x5df bytes) 000369 1761283750.526105 trunc from 0x03582d to 0x00f458 (0x263d4 bytes) 000370 1761283750.541893 mapread 0x00e038 thru 0x00f458 (0x1421 bytes) 000371 1761283750.542996 read 0x001d37 thru 0x00da0b (0xbcd5 bytes) 000372 1761283750.543554 mapread 0x009226 thru 0x00f458 (0x6233 bytes) 000373 1761283750.544247 write 0x0235b8 thru 0x0289d6 (0x541f bytes) 000374 1761283750.545164 read 0x01e46f thru 0x01e516 (0xa8 bytes) 000376 1761283750.549569 mapread 0x01c9d6 thru 0x0289d6 (0xc001 bytes) 000377 1761283750.550295 mapwrite 0x03dfde thru 0x03ffff (0x2022 bytes) 000381 1761283750.584328 MIRROR_RESYNC 2 mirrors 000382 1761283750.786614 mapread 0x00067b thru 0x00d6e5 (0xd06b bytes) 000383 1761283750.797431 mapwrite 0x011ea6 thru 0x01576c (0x38c7 bytes) 000384 1761283750.870077 read 0x021264 thru 0x026e0a (0x5ba7 bytes) 000385 1761283750.876910 MIRROR_RESYNC 2 mirrors 000387 1761283751.067166 mapread 0x01d86c thru 0x0254e1 (0x7c76 bytes) 000389 1761283751.075532 mapread 0x03e03e thru 0x03ffff (0x1fc2 bytes) 000391 1761283751.080289 read 0x0311d6 thru 0x03b868 (0xa693 bytes) 000392 1761283751.086301 MIRROR_RESYNC 2 mirrors 000393 1761283751.171292 trunc from 0x040000 to 0x0054f9 (0x3ab06 bytes) 000394 1761283751.210482 mapwrite 0x01d4bb thru 0x01f138 (0x1c7e bytes) 000395 1761283751.253684 trunc from 0x01f139 to 0x033bf9 (0x14ac1 bytes) 000397 1761283751.264028 MIRROR_EXTEND to 3 mirrors 000398 1761283751.505397 mapread 0x01c5e4 thru 0x0251d1 (0x8bee bytes) 000399 1761283751.519918 write 0x0021e5 thru 0x00b0ac (0x8ec8 bytes) 000400 1761283751.554969 write 0x026ebf thru 0x029184 (0x22c6 bytes) 000402 1761283751.562836 mapwrite 0x03c49c thru 0x03ffff (0x3b64 bytes) 000403 1761283751.606991 trunc from 0x040000 to 0x0014f9 (0x3eb06 bytes) 000404 1761283751.613686 mapwrite 0x020de0 thru 0x02df6e (0xd18f bytes) 000405 1761283751.652050 mapwrite 0x01ce3e thru 0x02974d (0xc910 bytes) 000406 1761283751.672148 write 0x033671 thru 0x03ffff (0xc98f bytes) 000407 1761283751.673399 mapwrite 0x017d05 thru 0x01c52e (0x482a bytes) 000408 1761283751.694960 mapwrite 0x020bd5 thru 0x02d306 (0xc732 bytes) 000410 1761283751.714148 read 0x029f34 thru 0x0305cc (0x6699 bytes) 000413 1761283751.718351 read 0x02b5ca thru 0x03af03 (0xf93a bytes) 000415 1761283751.726885 MIRROR_RESYNC 3 mirrors 000416 1761283751.945714 trunc from 0x040000 to 0x00ec22 (0x313dd bytes) 000417 1761283751.982061 read 0x009a8e thru 0x00a91e (0xe91 bytes) 000418 1761283751.987965 MIRROR_EXTEND to 4 mirrors 000419 1761283752.199631 read 0x007d28 thru 0x008310 (0x5e9 bytes) 000420 1761283752.204721 mapread 0x00b5e2 thru 0x00ec22 (0x3641 bytes) 000422 1761283752.222060 MIRROR_RESYNC 4 mirrors 000425 1761283752.481196 mapread 0x0047b6 thru 0x00e862 (0xa0ad bytes) 000426 1761283752.489466 mapwrite 0x024155 thru 0x0325a6 (0xe452 bytes) 000427 1761283752.563641 read 0x000cb4 thru 0x00cf61 (0xc2ae bytes) 000428 1761283752.572318 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000430 1761283752.667225 trunc from 0x0325a7 to 0x0141cf (0x1e3d7 bytes) 000431 1761283752.680593 mapread 0x005d54 thru 0x012372 (0xc61f bytes) 000432 1761283752.694716 write 0x02edcd thru 0x036a9d (0x7cd1 bytes) 000434 1761283752.699092 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000436 1761283752.809567 trunc from 0x036a9e to 0x025058 (0x11a45 bytes) 000437 1761283752.817936 read 0x0051cc thru 0x008fd3 (0x3e08 bytes) 000438 1761283752.820658 mapwrite 0x031b27 thru 0x038f45 (0x741f bytes) 000440 1761283752.863607 mapwrite 0x0150d0 thru 0x01940d (0x433e bytes) 000443 1761283752.886276 trunc from 0x038f46 to 0x036cb5 (0x2290 bytes) 000444 1761283752.892521 write 0x01539b thru 0x018aea (0x3750 bytes) 000445 1761283752.893419 mapread 0x001298 thru 0x00c072 (0xaddb bytes) 000446 1761283752.898588 mapread 0x0352de thru 0x036cb5 (0x19d8 bytes) 000449 1761283752.900109 trunc from 0x036cb6 to 0x02fd75 (0x6f40 bytes) 000450 1761283752.905919 mapwrite 0x019aa9 thru 0x027259 (0xd7b1 bytes) 000454 1761283752.941959 mapread 0x019a56 thru 0x021b96 (0x8141 bytes) 000455 1761283752.942831 read 0x00ce1d thru 0x00e3c8 (0x15ac bytes) 000456 1761283752.943732 mapwrite 0x001780 thru 0x00c7b9 (0xb03a bytes) 000457 1761283752.973471 write 0x021148 thru 0x02633d (0x51f6 bytes) 000459 1761283752.974797 write 0x03f5ea thru 0x03ffff (0xa16 bytes) 000460 1761283752.975564 mapwrite 0x0100ec thru 0x01425d (0x4172 bytes) 000461 1761283753.003414 MIRROR_EXTEND to 3 mirrors 000467 1761283753.289611 mapwrite 0x0140af thru 0x01b1a7 (0x70f9 bytes) 000468 1761283753.364155 mapread 0x02f41c thru 0x03f122 (0xfd07 bytes) 000470 1761283753.374814 MIRROR_RESYNC 3 mirrors 000471 1761283753.638437 mapread 0x01fc8d thru 0x02db33 (0xdea7 bytes) 000472 1761283753.649393 trunc from 0x040000 to 0x03a1ad (0x5e52 bytes) 000474 1761283753.698461 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000478 1761283753.789120 mapread 0x01fa8b thru 0x026530 (0x6aa6 bytes) 000479 1761283753.799953 trunc from 0x03a1ae to 0x0230a8 (0x17105 bytes) 000480 1761283753.810443 mapread 0x01e015 thru 0x02179f (0x378b bytes) 000482 1761283753.811648 write 0x03c478 thru 0x03ffff (0x3b88 bytes) 000483 1761283753.812363 write 0x00c44c thru 0x016c9a (0xa84f bytes) 000484 1761283753.820681 mapread 0x0097c5 thru 0x009f61 (0x79d bytes) 000485 1761283753.821504 write 0x008ced thru 0x01297b (0x9c8f bytes) 000486 1761283753.822674 trunc from 0x040000 to 0x00cce4 (0x3331b bytes) 000488 1761283753.834445 MIRROR_RESYNC 2 mirrors 000489 1761283754.058800 mapwrite 0x035391 thru 0x039c5c (0x48cc bytes) 000491 1761283754.141168 mapwrite 0x005570 thru 0x00eae4 (0x9575 bytes) 000494 1761283754.166195 read 0x026028 thru 0x02989d (0x3876 bytes) 000495 1761283754.171300 MIRROR_EXTEND to 3 mirrors 000498 1761283754.382297 read 0x00fce3 thru 0x01cae0 (0xcdfe bytes) 000499 1761283754.392091 MIRROR_RESYNC 3 mirrors 000501 1761283754.623966 mapread 0x035e0c thru 0x039c5c (0x3e51 bytes) 000503 1761283754.633946 read 0x00c51f thru 0x00e955 (0x2437 bytes) 000504 1761283754.636346 mapwrite 0x034d33 thru 0x03ffff (0xb2cd bytes) 000505 1761283754.709539 trunc from 0x040000 to 0x021e01 (0x1e1fe bytes) 000506 1761283754.717155 write 0x023877 thru 0x02b8d2 (0x805c bytes) 000507 1761283754.720351 MIRROR_RESYNC 3 mirrors 000508 1761283754.992303 write 0x01fc04 thru 0x021b2e (0x1f2b bytes) 000509 1761283755.025615 MIRROR_EXTEND to 4 mirrors 000511 1761283755.292712 mapread 0x01eeff thru 0x02aec3 (0xbfc5 bytes) 000512 1761283755.307117 mapread 0x0073b9 thru 0x0158d8 (0xe520 bytes) 000516 1761283755.327450 read 0x0199bc thru 0x019b8d (0x1d2 bytes) 000518 1761283755.334248 MIRROR_RESYNC 4 mirrors 000519 1761283755.553991 write 0x0357b3 thru 0x03ffff (0xa84d bytes) 000521 1761283755.575139 mapwrite 0x03156d thru 0x037e03 (0x6897 bytes) 000523 1761283755.606189 read 0x019d04 thru 0x01d081 (0x337e bytes) 000524 1761283755.609974 trunc from 0x040000 to 0x039820 (0x67df bytes) 000525 1761283755.619800 write 0x01c746 thru 0x02967e (0xcf39 bytes) 000527 1761283755.623827 mapwrite 0x0281df thru 0x0284c2 (0x2e4 bytes) 000529 1761283755.647500 mapread 0x00dacc thru 0x01a66e (0xcba3 bytes) 000530 1761283755.657417 MIRROR_RESYNC 4 mirrors 000531 1761283755.921468 write 0x038bbc thru 0x03ffff (0x7444 bytes) 000535 1761283755.953170 write 0x006752 thru 0x006a90 (0x33f bytes) 000538 1761283755.956629 write 0x03b4cb thru 0x03ffff (0x4b35 bytes) 000540 1761283755.957972 mapwrite 0x009a3d thru 0x00d15f (0x3723 bytes) 000542 1761283755.985779 mapread 0x01d9ff thru 0x023c60 (0x6262 bytes) 000546 1761283755.992105 mapwrite 0x01a109 thru 0x0299dd (0xf8d5 bytes) 000547 1761283756.027470 MIRROR_RESYNC 4 mirrors 000548 1761283756.302062 MIRROR_RESYNC 4 mirrors 000550 1761283756.396502 mapread 0x0235d1 thru 0x032fe6 (0xfa16 bytes) 000551 1761283756.406619 read 0x0083ef thru 0x011623 (0x9235 bytes) 000552 1761283756.412119 MIRROR_EXTEND to 5 mirrors 000555 1761283756.640907 mapread 0x0383b2 thru 0x03fa9f (0x76ee bytes) 000556 1761283756.648299 read 0x03fef7 thru 0x03ffff (0x109 bytes) 000558 1761283756.648939 mapwrite 0x0186aa thru 0x01e200 (0x5b57 bytes) 000560 1761283756.723281 mapread 0x00cab6 thru 0x00e4e9 (0x1a34 bytes) 000561 1761283756.728578 write 0x02a0f5 thru 0x035370 (0xb27c bytes) 000563 1761283756.735820 trunc from 0x040000 to 0x01547d (0x2ab82 bytes) 000564 1761283756.748319 MIRROR_EXTEND to 6 mirrors 000565 1761283756.982210 mapwrite 0x01d737 thru 0x020c6f (0x3539 bytes) 000569 1761283757.052308 mapread 0x008bc5 thru 0x017daf (0xf1eb bytes) 000570 1761283757.061580 write 0x0360bf thru 0x0392fb (0x323d bytes) 000572 1761283757.062974 mapread 0x012f56 thru 0x01824d (0x52f8 bytes) 000573 1761283757.063611 write 0x009aeb thru 0x00e05d (0x4573 bytes) 000574 1761283757.064403 read 0x020dfc thru 0x03061d (0xf822 bytes) 000575 1761283757.070184 trunc from 0x0392fc to 0x02ca87 (0xc874 bytes) 000576 1761283757.079972 write 0x03ec0c thru 0x03ffff (0x13f4 bytes) 000577 1761283757.081161 read 0x03d29c thru 0x03ffff (0x2d64 bytes) 000581 1761283757.087328 read 0x00b766 thru 0x00d642 (0x1edd bytes) 000583 1761283757.088276 read 0x03cd4b thru 0x03ffff (0x32b5 bytes) 000585 1761283757.089256 mapread 0x017766 thru 0x023d61 (0xc5fc bytes) 000586 1761283757.093007 MIRROR_EXTEND to 7 mirrors 000589 1761283757.312797 trunc from 0x040000 to 0x0013a2 (0x3ec5d bytes) 000591 1761283757.343543 mapread 0x000824 thru 0x0013a2 (0xb7f bytes) 000593 1761283757.348385 read 0x000a32 thru 0x0013a2 (0x971 bytes) 000594 1761283757.348957 write 0x036143 thru 0x03eeb0 (0x8d6e bytes) 000595 1761283757.350125 write 0x0163dc thru 0x0167ca (0x3ef bytes) 000596 1761283757.352830 mapwrite 0x0101c6 thru 0x01aecf (0xad0a bytes) 000598 1761283757.381275 trunc from 0x03eeb1 to 0x02a905 (0x145ab bytes) 000599 1761283757.392388 MIRROR_RESYNC 7 mirrors 000600 1761283757.728319 mapwrite 0x00fdaf thru 0x01aee7 (0xb139 bytes) 000601 1761283757.797403 write 0x0267c0 thru 0x02d89c (0x70dd bytes) 000602 1761283757.803225 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000604 1761283757.876326 mapread 0x0087e5 thru 0x01212a (0x9946 bytes) 000606 1761283757.882776 mapread 0x011e51 thru 0x01442d (0x25dd bytes) 000611 1761283757.887344 mapread 0x001174 thru 0x00315f (0x1fec bytes) 000614 1761283757.891414 mapread 0x020462 thru 0x023c09 (0x37a8 bytes) 000615 1761283757.895303 write 0x031208 thru 0x03cc69 (0xba62 bytes) 000617 1761283757.896608 trunc from 0x03cc6a to 0x0129d3 (0x2a296 bytes) 000618 1761283757.907419 mapread 0x0037bd thru 0x0039c6 (0x20a bytes) 000621 1761283757.908986 mapread 0x00c70a thru 0x0129d3 (0x62ca bytes) 000623 1761283757.910125 mapread 0x008800 thru 0x0089ca (0x1cb bytes) 000624 1761283757.910772 write 0x001e6b thru 0x00838f (0x6525 bytes) 000625 1761283757.911519 mapwrite 0x026d2e thru 0x02ea87 (0x7d5a bytes) 000626 1761283757.935659 write 0x012f58 thru 0x019e9b (0x6f44 bytes) 000627 1761283757.938330 write 0x0312de thru 0x034248 (0x2f6b bytes) 000628 1761283757.941001 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000631 1761283758.005697 mapwrite 0x0386c3 thru 0x03ffff (0x793d bytes) 000632 1761283758.034203 write 0x009979 thru 0x01002f (0x66b7 bytes) 000633 1761283758.038934 write 0x00ba19 thru 0x016e9a (0xb482 bytes) 000635 1761283758.042596 mapread 0x0348f6 thru 0x038242 (0x394d bytes) 000637 1761283758.043905 mapwrite 0x004210 thru 0x009b87 (0x5978 bytes) 000638 1761283758.063198 read 0x0069ad thru 0x010377 (0x99cb bytes) 000640 1761283758.064129 mapwrite 0x034bb3 thru 0x036535 (0x1983 bytes) 000643 1761283758.084563 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000646 1761283758.155605 write 0x029492 thru 0x02dd3f (0x48ae bytes) 000648 1761283758.159733 mapwrite 0x03744a thru 0x03ffff (0x8bb6 bytes) 000649 1761283758.178602 trunc from 0x040000 to 0x0082c4 (0x37d3b bytes) 000650 1761283758.184012 read 0x00669f thru 0x0082c4 (0x1c26 bytes) 000652 1761283758.188277 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000653 1761283758.251135 trunc from 0x0082c5 to 0x0348a8 (0x2c5e4 bytes) 000654 1761283758.257567 mapread 0x00b619 thru 0x00d264 (0x1c4c bytes) 000655 1761283758.265145 read 0x01c2cc thru 0x01d7dd (0x1512 bytes) 000657 1761283758.271796 MIRROR_EXTEND to 4 mirrors 000658 1761283758.516785 write 0x025e74 thru 0x02c780 (0x690d bytes) 000659 1761283758.561457 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000660 1761283758.660630 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000663 1761283758.753701 write 0x0085d5 thru 0x011f8b (0x99b7 bytes) 000664 1761283758.764434 write 0x03655a thru 0x03ffff (0x9aa6 bytes) 000665 1761283758.768297 MIRROR_RESYNC 2 mirrors 000666 1761283758.995059 mapwrite 0x03cb90 thru 0x03ffff (0x3470 bytes) 000668 1761283759.063214 mapread 0x018e62 thru 0x01f76f (0x690e bytes) 000669 1761283759.072847 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000671 1761283759.123481 mapread 0x00a354 thru 0x00ef9d (0x4c4a bytes) 000672 1761283759.131435 mapwrite 0x03452f thru 0x03ffff (0xbad1 bytes) 000673 1761283759.156027 read 0x03166b thru 0x033fd6 (0x296c bytes) 000674 1761283759.156610 trunc from 0x040000 to 0x014844 (0x2b7bb bytes) 000675 1761283759.167166 mapread 0x011f14 thru 0x0125c4 (0x6b1 bytes) 000684 1761283759.171225 mapread 0x00a054 thru 0x014844 (0xa7f1 bytes) 000685 1761283759.172055 read 0x010a2a thru 0x012d1e (0x22f5 bytes) 000686 1761283759.172541 write 0x026942 thru 0x027edf (0x159e bytes) 000687 1761283759.173320 mapwrite 0x0343e2 thru 0x03ffff (0xbc1e bytes) 000688 1761283759.201634 mapwrite 0x01281d thru 0x01bdc6 (0x95aa bytes) 000689 1761283759.221145 read 0x00704a thru 0x00af9b (0x3f52 bytes) 000690 1761283759.221772 read 0x01c4da thru 0x01f487 (0x2fae bytes) 000692 1761283759.222717 mapwrite 0x00d267 thru 0x0179b2 (0xa74c bytes) 000695 1761283759.247046 write 0x00c495 thru 0x00c72e (0x29a bytes) 000696 1761283759.247845 mapwrite 0x00f47d thru 0x014c93 (0x5817 bytes) 000700 1761283759.267027 mapwrite 0x02cda2 thru 0x02e409 (0x1668 bytes) 000701 1761283759.286031 mapread 0x036b26 thru 0x03ffff (0x94da bytes) 000702 1761283759.286909 trunc from 0x040000 to 0x0045da (0x3ba25 bytes) 000704 1761283759.294426 read 0x002994 thru 0x0045da (0x1c47 bytes) 000705 1761283759.294978 read 0x002a5b thru 0x0045da (0x1b80 bytes) 000707 1761283759.295651 read 0x0021b4 thru 0x0045da (0x2427 bytes) 000710 1761283759.297022 trunc from 0x0045db to 0x00ff40 (0xb966 bytes) 000711 1761283759.303730 mapwrite 0x03f618 thru 0x03fac6 (0x4af bytes) 000713 1761283759.332268 read 0x005979 thru 0x00e8dc (0x8f64 bytes) 000714 1761283759.332712 trunc from 0x03fac7 to 0x001ef2 (0x3dbd4 bytes) 000715 1761283759.339967 mapread 0x000c24 thru 0x001ef2 (0x12cf bytes) 000716 1761283759.340782 trunc from 0x001ef3 to 0x004bee (0x2cfc bytes) 000717 1761283759.347315 trunc from 0x004bef to 0x023750 (0x1eb62 bytes) 000718 1761283759.353235 trunc from 0x023751 to 0x0267b5 (0x3065 bytes) 000719 1761283759.362826 write 0x02a5b8 thru 0x03a1f9 (0xfc42 bytes) 000722 1761283759.365068 trunc from 0x03a1fa to 0x0078f9 (0x32900 bytes) 000723 1761283759.372082 mapread 0x0018fc thru 0x006c05 (0x530a bytes) 000724 1761283759.375812 write 0x003821 thru 0x008fcd (0x57ad bytes) 000726 1761283759.376999 read 0x003cf4 thru 0x008c9a (0x4fa7 bytes) 000727 1761283759.377400 mapread 0x006354 thru 0x008fcd (0x2c7a bytes) 000728 1761283759.378056 mapread 0x006cc6 thru 0x008fcd (0x2308 bytes) 000730 1761283759.379183 write 0x02ebd9 thru 0x0365d7 (0x79ff bytes) 000733 1761283759.381184 mapread 0x016c4a thru 0x022afd (0xbeb4 bytes) 000734 1761283759.386643 read 0x0147f1 thru 0x0214ee (0xccfe bytes) 000735 1761283759.387290 trunc from 0x0365d8 to 0x01bdb4 (0x1a823 bytes) 000736 1761283759.394166 mapwrite 0x032366 thru 0x03ddfa (0xba95 bytes) 000737 1761283759.426486 read 0x016bb3 thru 0x01ac87 (0x40d5 bytes) 000741 1761283759.428208 mapread 0x031fa7 thru 0x038aad (0x6b07 bytes) 000742 1761283759.429052 mapwrite 0x0096ca thru 0x00d3ea (0x3d21 bytes) 000743 1761283759.448607 trunc from 0x03ddfb to 0x0208b2 (0x1d548 bytes) 000745 1761283759.455068 write 0x01eeda thru 0x020bdc (0x1d03 bytes) 000746 1761283759.455620 mapread 0x020824 thru 0x020bdc (0x3b9 bytes) 000748 1761283759.456530 mapwrite 0x0360d7 thru 0x0371d3 (0x10fd bytes) 000751 1761283759.482837 mapwrite 0x008c31 thru 0x00b2e3 (0x26b3 bytes) 000753 1761283759.501908 write 0x00257c thru 0x00d697 (0xb11c bytes) 000754 1761283759.502984 trunc from 0x0371d4 to 0x027f05 (0xf2ce bytes) 000755 1761283759.509274 mapread 0x006eac thru 0x00c16b (0x52c0 bytes) 000756 1761283759.512341 MIRROR_EXTEND to 2 mirrors 000757 1761283759.737719 write 0x02137f thru 0x0239ce (0x2650 bytes) 000759 1761283759.786138 mapread 0x017636 thru 0x01884a (0x1215 bytes) 000762 1761283759.794438 mapread 0x01ec78 thru 0x0218ab (0x2c34 bytes) 000763 1761283759.799600 read 0x01ef76 thru 0x01fd5c (0xde7 bytes) 000765 1761283759.800626 mapwrite 0x03486d thru 0x03ffff (0xb793 bytes) 000766 1761283759.860003 write 0x02c2c5 thru 0x034d27 (0x8a63 bytes) 000767 1761283759.864538 read 0x00f43d thru 0x0176cc (0x8290 bytes) 000768 1761283759.870209 read 0x0379b5 thru 0x03ffff (0x864b bytes) 000769 1761283759.870775 mapwrite 0x00854d thru 0x01596a (0xd41e bytes) 000771 1761283759.909060 read 0x015377 thru 0x01d786 (0x8410 bytes) 000772 1761283759.909651 mapread 0x01f837 thru 0x02e1c6 (0xe990 bytes) 000773 1761283759.910481 mapwrite 0x02585c thru 0x025cf1 (0x496 bytes) 000774 1761283759.936432 mapread 0x03bc2e thru 0x03c893 (0xc66 bytes) 000776 1761283759.937694 trunc from 0x040000 to 0x009923 (0x366dc bytes) 000780 1761283759.952543 mapwrite 0x01c45f thru 0x025ece (0x9a70 bytes) 000781 1761283759.999598 read 0x0065b9 thru 0x013bc8 (0xd610 bytes) 000782 1761283760.000309 read 0x00ba95 thru 0x01b18c (0xf6f8 bytes) 000785 1761283760.005376 MIRROR_EXTEND to 3 mirrors 000786 1761283760.269518 mapread 0x024fe7 thru 0x025ece (0xee8 bytes) 000787 1761283760.289415 mapwrite 0x0094a1 thru 0x00f81d (0x637d bytes) 000788 1761283760.396455 trunc from 0x025ecf to 0x03abda (0x14d0c bytes) 000792 1761283760.413929 MIRROR_RESYNC 3 mirrors 000793 1761283760.711819 MIRROR_EXTEND to 4 mirrors 000798 1761283760.960312 trunc from 0x03abdb to 0x0016b5 (0x39525 bytes) 000803 1761283761.001589 write 0x00ec62 thru 0x018233 (0x95d2 bytes) 000804 1761283761.003091 trunc from 0x018234 to 0x014feb (0x3248 bytes) 000806 1761283761.014309 mapread 0x00cebd thru 0x014feb (0x812f bytes) 000810 1761283761.020733 write 0x023559 thru 0x02a15f (0x6c07 bytes) 000812 1761283761.022290 trunc from 0x02a160 to 0x00d938 (0x1c827 bytes) 000813 1761283761.029795 mapwrite 0x02395d thru 0x028e8a (0x552e bytes) 000816 1761283761.058697 write 0x01d42e thru 0x02052c (0x30ff bytes) 000817 1761283761.059580 write 0x016c53 thru 0x02267b (0xba29 bytes) 000819 1761283761.065452 write 0x00636e thru 0x00c9ee (0x6681 bytes) 000820 1761283761.069651 trunc from 0x028e8b to 0x0301cb (0x7341 bytes) 000822 1761283761.077237 mapwrite 0x01d64d thru 0x023d67 (0x671b bytes) 000826 1761283761.100738 write 0x02161d thru 0x022830 (0x1214 bytes) 000828 1761283761.102149 mapread 0x006056 thru 0x006172 (0x11d bytes) 000829 1761283761.102918 trunc from 0x0301cc to 0x03edfc (0xec31 bytes) 000830 1761283761.113037 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000834 1761283761.211650 read 0x006e72 thru 0x00d5c8 (0x6757 bytes) 000838 1761283761.217808 mapwrite 0x02ff40 thru 0x0388c0 (0x8981 bytes) 000839 1761283761.248637 mapwrite 0x028c32 thru 0x0342bf (0xb68e bytes) 000840 1761283761.276481 mapwrite 0x01a8cd thru 0x01e055 (0x3789 bytes) 000842 1761283761.297550 write 0x02fe20 thru 0x033757 (0x3938 bytes) 000843 1761283761.298514 write 0x02b202 thru 0x033247 (0x8046 bytes) 000845 1761283761.302432 MIRROR_EXTEND to 4 mirrors 000846 1761283761.517945 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000847 1761283761.613223 read 0x02f948 thru 0x0383ae (0x8a67 bytes) 000848 1761283761.617407 mapread 0x00a014 thru 0x019766 (0xf753 bytes) 000850 1761283761.627988 write 0x01b13b thru 0x02717a (0xc040 bytes) 000851 1761283761.636874 mapread 0x039d81 thru 0x03edfc (0x507c bytes) 000852 1761283761.643064 MIRROR_RESYNC 3 mirrors 000853 1761283761.990962 write 0x0271fc thru 0x02a724 (0x3529 bytes) 000857 1761283762.022796 mapread 0x01ade0 thru 0x029f31 (0xf152 bytes) 000858 1761283762.031279 read 0x037320 thru 0x038b2a (0x180b bytes) 000859 1761283762.038341 MIRROR_RESYNC 3 mirrors 000860 1761283762.334175 mapwrite 0x0182ab thru 0x01bd5c (0x3ab2 bytes) 000864 1761283762.432272 MIRROR_EXTEND to 4 mirrors 000865 1761283762.673526 mapread 0x02edfd thru 0x0340d7 (0x52db bytes) 000866 1761283762.688438 mapwrite 0x027219 thru 0x02ffeb (0x8dd3 bytes) 000867 1761283762.789173 read 0x00738a thru 0x00e37b (0x6ff2 bytes) 000871 1761283762.801408 write 0x023c9b thru 0x031370 (0xd6d6 bytes) 000874 1761283762.808227 read 0x00c151 thru 0x01551c (0x93cc bytes) 000875 1761283762.814935 mapwrite 0x001f3e thru 0x00e6b8 (0xc77b bytes) 000877 1761283762.846386 write 0x00441d thru 0x00ecfb (0xa8df bytes) 000878 1761283762.847547 mapwrite 0x01ac0d thru 0x027024 (0xc418 bytes) 000880 1761283762.874303 write 0x01971b thru 0x01ee13 (0x56f9 bytes) 000881 1761283762.875247 mapwrite 0x0257fe thru 0x02cd96 (0x7599 bytes) 000882 1761283762.901146 mapwrite 0x02f74e thru 0x031cd9 (0x258c bytes) 000883 1761283762.921126 write 0x020613 thru 0x030024 (0xfa12 bytes) 000885 1761283762.922916 mapwrite 0x004a75 thru 0x00a17d (0x5709 bytes) 000886 1761283762.943810 write 0x016e9a thru 0x01c322 (0x5489 bytes) 000887 1761283762.946973 MIRROR_EXTEND to 5 mirrors 000891 1761283763.210026 mapread 0x01df8c thru 0x022fba (0x502f bytes) 000892 1761283763.228301 MIRROR_EXTEND to 6 mirrors 000893 1761283763.468492 trunc from 0x03edfd to 0x000ade (0x3e31e bytes) 000894 1761283763.505222 write 0x001653 thru 0x006101 (0x4aaf bytes) 000895 1761283763.508423 MIRROR_RESYNC 6 mirrors 000896 1761283763.932295 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000898 1761283764.050728 mapwrite 0x02eb8b thru 0x03b3b6 (0xc82c bytes) 000899 1761283764.110246 read 0x0261d3 thru 0x03455f (0xe38d bytes) 000900 1761283764.113077 read 0x01bd92 thru 0x01e84b (0x2aba bytes) 000902 1761283764.119020 write 0x023bd6 thru 0x02a16e (0x6599 bytes) 000903 1761283764.119629 trunc from 0x03b3b7 to 0x025502 (0x15eb4 bytes) 000905 1761283764.129159 mapwrite 0x0335f1 thru 0x03fec9 (0xc8d9 bytes) 000906 1761283764.175842 trunc from 0x03feca to 0x02f376 (0x10b53 bytes) 000910 1761283764.183056 read 0x0167c9 thru 0x01d656 (0x6e8e bytes) 000911 1761283764.183616 mapwrite 0x0379b3 thru 0x03ffff (0x864d bytes) 000912 1761283764.211838 mapread 0x021c6a thru 0x02e43b (0xc7d2 bytes) 000913 1761283764.214213 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000914 1761283764.284734 mapwrite 0x0051e9 thru 0x00d020 (0x7e38 bytes) 000918 1761283764.317694 MIRROR_EXTEND to 5 mirrors 000919 1761283764.510022 mapread 0x0282ad thru 0x031189 (0x8edd bytes) 000921 1761283764.520826 trunc from 0x040000 to 0x00a6cd (0x35932 bytes) 000925 1761283764.553345 read 0x000152 thru 0x002038 (0x1ee7 bytes) 000928 1761283764.556866 mapread 0x0038bd thru 0x007e11 (0x4555 bytes) 000929 1761283764.560501 mapwrite 0x0305ea thru 0x03c2f9 (0xbd10 bytes) 000930 1761283764.590532 mapread 0x030aa1 thru 0x03c2f9 (0xb859 bytes) 000931 1761283764.591434 trunc from 0x03c2fa to 0x03cf13 (0xc1a bytes) 000932 1761283764.596417 mapread 0x020987 thru 0x030142 (0xf7bc bytes) 000933 1761283764.600318 read 0x00c8af thru 0x0175d7 (0xad29 bytes) 000934 1761283764.604082 mapread 0x001c81 thru 0x003d1a (0x209a bytes) 000935 1761283764.604839 mapread 0x00c72a thru 0x00cdea (0x6c1 bytes) 000937 1761283764.605879 trunc from 0x03cf14 to 0x00169f (0x3b874 bytes) 000940 1761283764.613122 mapread 0x000094 thru 0x00169f (0x160c bytes) 000942 1761283764.614154 write 0x01aef0 thru 0x027b33 (0xcc44 bytes) 000943 1761283764.615245 read 0x014e8f thru 0x022cc2 (0xde34 bytes) 000944 1761283764.621654 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a8:0x0]: Invalid argument (22) 000945 1761283764.700080 trunc from 0x027b34 to 0x02d379 (0x5846 bytes) 000946 1761283764.706010 mapread 0x00fe2d thru 0x018b1b (0x8cef bytes) 000950 1761283764.716360 mapread 0x0032f8 thru 0x006c43 (0x394c bytes) 000952 1761283764.722845 MIRROR_EXTEND to 5 mirrors 000953 1761283764.938178 mapread 0x017fce thru 0x01d4af (0x54e2 bytes) 000954 1761283764.949048 write 0x02747d thru 0x02c42f (0x4fb3 bytes) 000955 1761283764.974373 write 0x0335d8 thru 0x03f4ad (0xbed6 bytes) 000956 1761283764.975615 mapwrite 0x03c383 thru 0x03d63e (0x12bc bytes) 000957 1761283764.992509 mapwrite 0x00ebc3 thru 0x0105d7 (0x1a15 bytes) 000962 1761283765.019242 MIRROR_EXTEND to 6 mirrors 000963 1761283765.254834 trunc from 0x03f4ae to 0x030479 (0xf034 bytes) 000965 1761283765.282875 read 0x0087c9 thru 0x00e4cc (0x5d04 bytes) 000967 1761283765.287236 read 0x01726d thru 0x0256b8 (0xe44c bytes) 000971 1761283765.293025 write 0x036b02 thru 0x03ffff (0x94fe bytes) 000972 1761283765.294203 trunc from 0x040000 to 0x00d82f (0x327d0 bytes) 000974 1761283765.304386 read 0x003e1b thru 0x00765d (0x3843 bytes) 000976 1761283765.305404 write 0x014d74 thru 0x0185f6 (0x3883 bytes) 000977 1761283765.306384 mapwrite 0x032496 thru 0x03ffff (0xdb6a bytes) 000978 1761283765.344330 write 0x024a67 thru 0x02d120 (0x86ba bytes) 000979 1761283765.345470 mapread 0x01a487 thru 0x0261a7 (0xbd21 bytes) 000980 1761283765.346287 mapwrite 0x0080cb thru 0x00917b (0x10b1 bytes) 000981 1761283765.365417 trunc from 0x040000 to 0x00349e (0x3cb61 bytes) 000982 1761283765.372941 read 0x00016f thru 0x00349e (0x3330 bytes) 000983 1761283765.373570 mapwrite 0x028806 thru 0x02ea9e (0x6299 bytes) 000984 1761283765.408386 read 0x0102c3 thru 0x01ca60 (0xc79e bytes) 000986 1761283765.409430 mapwrite 0x03c7b2 thru 0x03ffff (0x384e bytes) 000987 1761283765.441419 write 0x003b17 thru 0x00805e (0x4548 bytes) 000988 1761283765.442416 mapwrite 0x03c848 thru 0x03df9e (0x1757 bytes) 000989 1761283765.461331 read 0x02835e thru 0x032c8b (0xa92e bytes) 000990 1761283765.461967 trunc from 0x040000 to 0x026ed0 (0x1912f bytes) 000991 1761283765.469115 trunc from 0x026ed1 to 0x0361f9 (0xf329 bytes) 000994 1761283765.475682 read 0x026e37 thru 0x02fedb (0x90a5 bytes) 000995 1761283765.480206 mapread 0x006ec8 thru 0x00e2c5 (0x73fe bytes) 000997 1761283765.481559 read 0x00068c thru 0x0054aa (0x4e1f bytes) 000998 1761283765.482137 mapwrite 0x020029 thru 0x024d99 (0x4d71 bytes) All operations completed A-OK! PASS 70a (27s) == sanity-flr test 71: check mirror extend parent fid ==== 01:29:27 (1761283767) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 01:29:28 (1761283768) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8430720 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401600 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8813056 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9151008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8949696 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9211712 .. Extending file size to 9358400 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56772 56773 56775 56777 56779 done 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:30:35 (1761283835) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6170112 .. Extending file size to 6170112 .. done Extending file size to 6833184 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6833184 .. failed Extending file size to 7539840 .. Extending file size to 7539840 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8687200 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8687200 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9216896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9216896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60187 60188 60190 60192 60194 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:31:41 (1761283901) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0453336 s, 46.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116685 s, 18.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:31:48 (1761283908) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0416555 s, 50.3 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 01:31:51 (1761283911) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x6ab:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:31:54 (1761283914) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0578883 s, 36.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:31:57 (1761283917) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0719298 s, 29.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6af:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:32:00 (1761283920) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0754633 s, 27.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b1:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:32:25 (1761283945) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234454 s, 44.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6af:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:33:10 (1761283990) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0392406 s, 53.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:33:56 (1761284036) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. Creating new pool test_204f oleg429-server: Pool lustre.test_204f created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0375031 s, 55.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg429-server: Pool lustre.test_204f destroyed PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:34:54 (1761284094) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:34:57 (1761284097) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:34:59 (1761284099) oleg429-server: Pool lustre.test_206 created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg429-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:35:11 (1761284111) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267356 s, 39.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895903 s, 117 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x11bc384f Mirror 3: 0x11bc384f PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:35:14 (1761284114) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:35:15 (1761284115) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:35:16 (1761284116) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0213868 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xdf0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6f0000 Read this (4 bytes): BBBB 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:35:32 (1761284132) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00376552 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:35:34 (1761284134) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139571 s, 75.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x512 lmm_fid: [0x200000404:0x512:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x512 lmm_fid: [0x200000404:0x512:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835218 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:35:34.000000000 -0400 Modify: 2025-10-24 01:35:34.000000000 -0400 Change: 2025-10-24 01:35:35.000000000 -0400 Birth: 2025-10-24 01:35:35.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:35:36 (1761284136) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:35:37 (1761284137) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.106203 s, 386 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142641 s, 287 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 24 01:35 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x514:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 24 01:35 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 845 sec ============ 01:35:40 (1761284140) === sanity-flr: start cleanup 01:35:40 (1761284140) === === sanity-flr: finish cleanup 01:35:42 (1761284142) ===