-----============= acceptance-small: sanity-flr ============----- Thu Oct 23 12:43:13 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg647-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:43:30 (1761237810) === oleg647-client.virtnet: executing check_config_client /mnt/lustre oleg647-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg647-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94b708303000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94b708303000.idle_timeout=debug disable quota as required oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:43:58 (1761237838) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7597] == sanity-flr test 0a: lfs mirror create with -N option == 12:44:00 (1761237840) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:44:14 (1761237854) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:44:16 (1761237856) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:44:19 (1761237859) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:44:40 (1761237880) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:44:43 (1761237883) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:44:46 (1761237886) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94b708303000.stats=clear osc.lustre-OST0001-osc-ffff94b708303000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:44:53 (1761237893) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:45:02 (1761237902) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:45:10 (1761237910) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:45:13 (1761237913) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 12:45:21 (1761237921) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:45:28 (1761237928) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:45:30 (1761237930) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:45:37 (1761237937) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:45:46 (1761237946) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:45:52 (1761237952) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.56753 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139151 s, 7.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:46:02 (1761237962) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.136243 s, 69.3 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:46:09 (1761237969) Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (56s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:47:05 (1761238025) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11569 s, 18.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (29s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:47:34 (1761238054) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (57s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:48:31 (1761238111) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.85155 s, 13.6 MB/s -rw-r--r-- 1 root root 52428800 Oct 23 12:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.065s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.098s user 0m0.003s sys 0m0.184s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.960s user 0m0.001s sys 0m0.144s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b708303000.ost_server_uuid in FULL state after 0 sec PASS 33b (72s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:49:43 (1761238183) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:49:46 (1761238186) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:49:50 (1761238190) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 12:49:53 (1761238193) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 12:49:59 (1761238199) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.484876 s, 26.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:50:09 (1761238209) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178097 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336661 s, 31.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:50:15 (1761238215) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:50:23 (1761238223) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000767866 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000731974 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00133116 s, 769 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00228534 s, 448 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:50:32 (1761238232) create test files with size 15749693 .. 026e8919904543d237366c1c8a715405 - 109e2cc7dd08e5320888a8f6a9f373f7 - 0eca555cb98dae6e1c9f4e0090a305be - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 12:50:53 (1761238253) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:51:08 (1761238268) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:51:13 (1761238273) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145223 s, 7.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021887 s, 47.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 12:51:21 (1761238281) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11034 s, 19.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.256966 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241553 s, 8.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (10s) == sanity-flr test 42: lfs mirror verify ================= 12:51:32 (1761238292) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:51:35 (1761238295) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:51:38 (1761238298) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380094 s, 27.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 12:51:46 (1761238306) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:51:49 (1761238309) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:51:57 (1761238317) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.258041 s, 40.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:52:08 (1761238328) llite.lustre-ffff94b718787000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.348353 s, 30.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 20494 flags: 1 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 20494 flags: 1 llite.lustre-ffff94b7396ca000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:52:20 (1761238340) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:52:56 (1761238376) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:53:27 (1761238407) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 12:53:29 (1761238409) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138229 s, 22.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137628 s, 22.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 2e792f8bc2c7190e7adb14bfbfcb3c58 - mirror 1 checksum: 50bb0fa836e480758a8d69f762cdf04e - mirror 2 checksum: 2e792f8bc2c7190e7adb14bfbfcb3c58 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:53:30 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:53:30 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2e792f8bc2c7190e7adb14bfbfcb3c58 - mirror 1 checksum: 50bb0fa836e480758a8d69f762cdf04e - mirror 2 checksum: 50bb0fa836e480758a8d69f762cdf04e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:53:30 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:53:30 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (8s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:53:38 (1761238418) Failing mds1 on oleg647-server Stopping /mnt/lustre-mds1 (opts:) on oleg647-server 12:53:43 (1761238423) shut down facet: mds1 facet_host: oleg647-server facet_failover_host: oleg647-server Failover mds1 to oleg647-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 12:54:04 (1761238444) targets are mounted 12:54:04 (1761238444) facet_failover done oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198664 s, 52.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:54:18 (1761238458) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:54:21 (1761238461) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:54:24 (1761238464) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00715178 s, 2.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:54:30 (1761238470) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:54:33 (1761238473) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:54:36 (1761238476) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:54:58 (1761238498) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:55:26 (1761238526) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:55:48 (1761238548) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:56:11 (1761238571) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 12:56:17 (1761238577) mirror create pid 52306 mirror split pid 52308 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52306 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52308 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 12:57:23 (1761238643) Seed set to 8645 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004 1761238645.659684 mapwrite 0x01ff88 thru 0x021e2b (0x1ea4 bytes) 000005 1761238645.741730 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x5454 000009 1761238646.343192 trunc from 0x021e2c to 0x005453 (0x1c9d8 bytes) 000012 1761238646.451643 mapwrite 0x008b54 thru 0x00dc7a (0x5127 bytes) truncating to largest ever: 0x1dc7c 000013 1761238646.524881 trunc from 0x00dc7b to 0x01dc7b (0x10001 bytes) 000014 1761238646.549275 read 0x01a760 thru 0x01ad75 (0x616 bytes) 000015 1761238646.559580 mapread 0x00dc3a thru 0x012d7d (0x5144 bytes) 000016 1761238646.575447 trunc from 0x01dc7c to 0x00c417 (0x11864 bytes) 000020 1761238646.611946 read 0x002c8d thru 0x00a383 (0x76f7 bytes) 000021 1761238646.619937 write 0x00bd17 thru 0x00d9bb (0x1ca5 bytes) 000024 1761238646.625625 MIRROR_RESYNC 2 mirrors 000026 1761238647.124461 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000028 1761238647.275140 mapwrite 0x03e879 thru 0x03ffff (0x1787 bytes) 000031 1761238647.405395 write 0x03168a thru 0x033cfa (0x2671 bytes) 000032 1761238647.420665 read 0x036438 thru 0x03ffff (0x9bc8 bytes) truncating to largest ever: 0x26b41 000034 1761238647.438794 trunc from 0x040000 to 0x026b40 (0x194bf bytes) 000035 1761238647.460041 write 0x030ce4 thru 0x0372a5 (0x65c2 bytes) 000037 1761238647.461945 mapwrite 0x000fe5 thru 0x00cae3 (0xbaff bytes) 000038 1761238647.525267 write 0x030e11 thru 0x037388 (0x6578 bytes) 000039 1761238647.526860 write 0x027d58 thru 0x0291df (0x1488 bytes) 000040 1761238647.540384 mapread 0x0130ac thru 0x017fa7 (0x4efc bytes) 000041 1761238647.554720 mapread 0x003aab thru 0x00c914 (0x8e6a bytes) truncating to largest ever: 0x2fc0c 000042 1761238647.555706 trunc from 0x037389 to 0x02fc0b (0x777d bytes) 000043 1761238647.590397 trunc from 0x02fc0c to 0x014999 (0x1b272 bytes) 000044 1761238647.620125 read 0x00b07b thru 0x0146d9 (0x965f bytes) 000045 1761238647.620836 mapwrite 0x035e4a thru 0x03ffff (0xa1b6 bytes) 000050 1761238647.736237 read 0x00679d thru 0x00a886 (0x40ea bytes) 000053 1761238647.737768 mapread 0x01f689 thru 0x026fdd (0x7955 bytes) 000054 1761238647.738767 trunc from 0x040000 to 0x018aa5 (0x2755a bytes) 000060 1761238647.769267 read 0x0095dc thru 0x018aa5 (0xf4ca bytes) 000062 1761238647.771381 mapwrite 0x0217d3 thru 0x02e71d (0xcf4b bytes) 000064 1761238647.892348 read 0x01abbe thru 0x021db3 (0x71f6 bytes) 000065 1761238647.893104 mapwrite 0x03c250 thru 0x03ffff (0x3db0 bytes) 000067 1761238648.075491 mapwrite 0x022db1 thru 0x02462b (0x187b bytes) 000068 1761238648.135560 trunc from 0x040000 to 0x00c2d4 (0x33d2b bytes) 000070 1761238648.183974 mapwrite 0x016c5e thru 0x01b088 (0x442b bytes) 000076 1761238648.283541 mapwrite 0x01259f thru 0x01bf8d (0x99ef bytes) 000078 1761238648.386863 mapwrite 0x0321df thru 0x0379f7 (0x5819 bytes) 000079 1761238648.482755 mapwrite 0x000cc8 thru 0x00f9af (0xece8 bytes) 000081 1761238648.555689 trunc from 0x0379f8 to 0x00f212 (0x287e5 bytes) 000082 1761238648.578399 mapread 0x007750 thru 0x00f212 (0x7ac3 bytes) 000084 1761238648.580383 mapwrite 0x03123b thru 0x033f35 (0x2cfb bytes) 000086 1761238648.674858 write 0x006524 thru 0x00c424 (0x5f01 bytes) 000087 1761238648.676176 mapwrite 0x0225da thru 0x022950 (0x377 bytes) 000089 1761238648.721591 write 0x0354e6 thru 0x03ee7c (0x9997 bytes) 000091 1761238648.724275 read 0x032c6b thru 0x03cdfe (0xa194 bytes) 000093 1761238648.748456 mapwrite 0x020273 thru 0x02dcdd (0xda6b bytes) 000094 1761238648.862160 trunc from 0x03ee7d to 0x00d349 (0x31b33 bytes) 000095 1761238648.912947 mapwrite 0x00c009 thru 0x0156c2 (0x96ba bytes) 000097 1761238649.116676 MIRROR_EXTEND to 2 mirrors 000099 1761238649.866227 write 0x038e16 thru 0x03f8a3 (0x6a8e bytes) 000100 1761238649.963369 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000101 1761238650.125601 mapwrite 0x02d312 thru 0x035e7f (0x8b6e bytes) 000102 1761238650.209493 write 0x02102b thru 0x0269c8 (0x599e bytes) 000103 1761238650.221004 read 0x03a613 thru 0x03f8a3 (0x5291 bytes) truncating to largest ever: 0x30f24 000107 1761238650.231646 trunc from 0x03f8a4 to 0x030f23 (0xe980 bytes) 000110 1761238650.258555 read 0x02a85d thru 0x030f23 (0x66c7 bytes) 000113 1761238650.266123 MIRROR_EXTEND to 2 mirrors 000114 1761238650.778766 mapwrite 0x003a3e thru 0x0046b5 (0xc78 bytes) 000115 1761238650.933953 mapwrite 0x024d51 thru 0x0277f3 (0x2aa3 bytes) 000116 1761238651.002781 mapread 0x01daa2 thru 0x025661 (0x7bc0 bytes) 000120 1761238651.020307 MIRROR_EXTEND to 3 mirrors 000123 1761238651.771561 mapread 0x01135b thru 0x01b0a9 (0x9d4f bytes) 000124 1761238651.796112 trunc from 0x030f24 to 0x02dbc7 (0x335c bytes) 000125 1761238651.909122 mapwrite 0x025e33 thru 0x03590f (0xfadd bytes) 000126 1761238652.016434 mapwrite 0x01814c thru 0x025ad9 (0xd98e bytes) 000127 1761238652.103212 trunc from 0x035910 to 0x01dc2b (0x17ce4 bytes) 000128 1761238652.129506 trunc from 0x01dc2c to 0x02518b (0x7560 bytes) 000129 1761238652.143337 mapread 0x002088 thru 0x00d8d4 (0xb84d bytes) 000131 1761238652.162210 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000132 1761238652.410314 MIRROR_EXTEND to 3 mirrors 000133 1761238653.159877 mapwrite 0x0191b5 thru 0x023e56 (0xaca2 bytes) 000134 1761238653.338173 read 0x015b41 thru 0x0221b8 (0xc678 bytes) 000135 1761238653.338837 mapread 0x01f7bd thru 0x01fee9 (0x72d bytes) 000136 1761238653.339864 mapwrite 0x02d229 thru 0x036973 (0x974b bytes) 000137 1761238653.464584 mapread 0x01921a thru 0x023cec (0xaad3 bytes) 000138 1761238653.465498 trunc from 0x036974 to 0x01b404 (0x1b56f bytes) 000139 1761238653.494458 read 0x00256d thru 0x00a100 (0x7b94 bytes) truncating to largest ever: 0x352f4 000140 1761238653.504895 trunc from 0x01b405 to 0x0352f3 (0x19eef bytes) 000143 1761238653.535580 read 0x018952 thru 0x020ec0 (0x856f bytes) 000144 1761238653.553879 mapwrite 0x02558a thru 0x0281fc (0x2c73 bytes) 000151 1761238653.625619 write 0x03cafe thru 0x03ffff (0x3502 bytes) 000154 1761238653.631700 mapread 0x025c32 thru 0x02e439 (0x8808 bytes) 000155 1761238653.635256 read 0x009720 thru 0x015018 (0xb8f9 bytes) 000157 1761238653.650575 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000158 1761238653.882360 read 0x018f97 thru 0x026840 (0xd8aa bytes) 000159 1761238653.892472 mapread 0x00edd4 thru 0x01849a (0x96c7 bytes) 000160 1761238653.918074 mapread 0x03135f thru 0x03a43a (0x90dc bytes) 000161 1761238653.946888 trunc from 0x040000 to 0x01d8db (0x22724 bytes) 000166 1761238653.982560 MIRROR_RESYNC 2 mirrors 000167 1761238654.609761 read 0x00d738 thru 0x01d513 (0xfddc bytes) 000168 1761238654.622064 mapread 0x016dcc thru 0x01d8db (0x6b10 bytes) 000170 1761238654.623043 read 0x005384 thru 0x00e0df (0x8d5c bytes) 000171 1761238654.631015 mapread 0x0006ae thru 0x005b05 (0x5458 bytes) 000172 1761238654.631717 mapwrite 0x017eac thru 0x01f544 (0x7699 bytes) 000173 1761238654.817462 mapwrite 0x016a20 thru 0x01e429 (0x7a0a bytes) 000175 1761238654.882007 mapread 0x00e768 thru 0x0111bd (0x2a56 bytes) 000176 1761238654.906419 mapwrite 0x0220f2 thru 0x024da1 (0x2cb0 bytes) 000178 1761238655.001075 mapread 0x0030cf thru 0x003b04 (0xa36 bytes) 000181 1761238655.009261 mapwrite 0x03ba48 thru 0x03ffff (0x45b8 bytes) 000182 1761238655.094096 trunc from 0x040000 to 0x005ef3 (0x3a10c bytes) truncating to largest ever: 0x3ff7d 000183 1761238655.120771 trunc from 0x005ef4 to 0x03ff7c (0x3a089 bytes) 000184 1761238655.144085 mapread 0x00a98f thru 0x013355 (0x89c7 bytes) 000187 1761238655.210038 trunc from 0x03ff7d to 0x03efbe (0xfbe bytes) 000191 1761238655.234837 read 0x022a35 thru 0x02d72f (0xacfb bytes) 000192 1761238655.247254 read 0x0162ad thru 0x017c5a (0x19ae bytes) 000193 1761238655.247989 write 0x01353f thru 0x01e113 (0xabd5 bytes) 000194 1761238655.249257 read 0x022022 thru 0x026903 (0x48e2 bytes) 000195 1761238655.250853 mapread 0x039fa0 thru 0x03ae3d (0xe9e bytes) 000196 1761238655.252126 read 0x011e88 thru 0x0185d7 (0x6750 bytes) 000198 1761238655.263138 mapread 0x0014a3 thru 0x00b49d (0x9ffb bytes) 000199 1761238655.264117 write 0x03a2df thru 0x03c848 (0x256a bytes) 000200 1761238655.265382 mapwrite 0x003443 thru 0x00b786 (0x8344 bytes) 000202 1761238655.344267 mapwrite 0x010df2 thru 0x015bca (0x4dd9 bytes) 000204 1761238655.386804 mapread 0x019bff thru 0x01c7f9 (0x2bfb bytes) 000205 1761238655.387717 trunc from 0x03efbf to 0x030e62 (0xe15c bytes) 000207 1761238655.413894 mapwrite 0x02e88e thru 0x03260e (0x3d81 bytes) 000208 1761238655.492690 read 0x0198ba thru 0x025c6c (0xc3b3 bytes) 000209 1761238655.498295 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000210 1761238655.681186 trunc from 0x03260f to 0x019b5f (0x18aaf bytes) 000211 1761238655.711409 mapread 0x007bfe thru 0x011fb8 (0xa3bb bytes) 000215 1761238655.735917 mapread 0x0079d0 thru 0x0150b8 (0xd6e9 bytes) 000219 1761238655.745596 mapwrite 0x023d20 thru 0x0291b0 (0x5491 bytes) 000220 1761238655.837182 MIRROR_EXTEND to 2 mirrors 000222 1761238656.442709 read 0x00320b thru 0x00927f (0x6075 bytes) 000226 1761238656.458039 write 0x025845 thru 0x032983 (0xd13f bytes) 000227 1761238656.539211 trunc from 0x032984 to 0x03f920 (0xcf9d bytes) 000229 1761238656.571810 write 0x029b01 thru 0x03141f (0x791f bytes) 000231 1761238656.573863 write 0x008605 thru 0x00a14b (0x1b47 bytes) 000234 1761238656.587018 read 0x01f967 thru 0x01fc04 (0x29e bytes) 000235 1761238656.612830 MIRROR_RESYNC 2 mirrors 000237 1761238657.184310 trunc from 0x03f921 to 0x012f78 (0x2c9a8 bytes) 000240 1761238657.298150 mapwrite 0x02c003 thru 0x03afbb (0xefb9 bytes) 000244 1761238657.437425 write 0x038ecd thru 0x039ea2 (0xfd6 bytes) 000246 1761238657.444493 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000247 1761238657.613088 read 0x01b37d thru 0x026f31 (0xbbb5 bytes) 000249 1761238657.634241 mapread 0x00fe2c thru 0x0119ee (0x1bc3 bytes) 000251 1761238657.660963 read 0x036ebc thru 0x03afbb (0x4100 bytes) 000252 1761238657.673219 write 0x006482 thru 0x007c51 (0x17d0 bytes) 000253 1761238657.675341 read 0x0060bc thru 0x006c21 (0xb66 bytes) 000254 1761238657.675903 trunc from 0x03afbc to 0x027283 (0x13d38 bytes) 000255 1761238657.701072 mapwrite 0x03aa22 thru 0x03c38b (0x196a bytes) 000260 1761238657.774714 read 0x01212b thru 0x012690 (0x566 bytes) 000264 1761238657.778422 write 0x02c9e7 thru 0x03c926 (0xff40 bytes) 000266 1761238657.781600 write 0x01973e thru 0x0233fc (0x9cbf bytes) 000267 1761238657.783924 read 0x01bcdb thru 0x0270b7 (0xb3dd bytes) 000268 1761238657.785104 read 0x00db4a thru 0x012257 (0x470e bytes) 000269 1761238657.785683 write 0x007654 thru 0x015b3e (0xe4eb bytes) 000271 1761238657.788239 mapread 0x007cfa thru 0x0088af (0xbb6 bytes) 000276 1761238657.791418 read 0x02c0e6 thru 0x035437 (0x9352 bytes) 000277 1761238657.799470 MIRROR_EXTEND to 2 mirrors 000279 1761238658.419499 read 0x036130 thru 0x03c926 (0x67f7 bytes) 000280 1761238658.429383 mapwrite 0x00da00 thru 0x0127f4 (0x4df5 bytes) 000281 1761238658.563756 mapread 0x02d786 thru 0x03b324 (0xdb9f bytes) 000282 1761238658.598692 trunc from 0x03c927 to 0x020bc3 (0x1bd63 bytes) 000283 1761238658.631609 mapread 0x015085 thru 0x01a421 (0x539d bytes) 000285 1761238658.644778 mapread 0x006699 thru 0x011cc9 (0xb631 bytes) 000287 1761238658.656296 read 0x00e62b thru 0x012646 (0x401c bytes) 000288 1761238658.664346 MIRROR_EXTEND to 3 mirrors 000291 1761238659.151254 mapwrite 0x0375ad thru 0x03e6a1 (0x70f5 bytes) 000292 1761238659.364792 MIRROR_RESYNC 3 mirrors 000295 1761238659.999925 mapwrite 0x000e9f thru 0x009836 (0x8998 bytes) 000296 1761238660.160611 trunc from 0x03e6a2 to 0x002670 (0x3c031 bytes) 000302 1761238660.212374 read 0x001178 thru 0x002670 (0x14f9 bytes) 000303 1761238660.213651 trunc from 0x002671 to 0x03f021 (0x3c9b1 bytes) 000306 1761238660.232451 mapwrite 0x00b2ac thru 0x00ce6b (0x1bc0 bytes) 000307 1761238660.310113 mapwrite 0x00a8b1 thru 0x017876 (0xcfc6 bytes) 000308 1761238660.368205 MIRROR_EXTEND to 4 mirrors 000309 1761238660.921276 mapread 0x006ab0 thru 0x00eebf (0x8410 bytes) 000310 1761238660.938668 mapwrite 0x030d4a thru 0x034315 (0x35cc bytes) 000314 1761238661.131281 write 0x032283 thru 0x03de64 (0xbbe2 bytes) 000315 1761238661.146670 mapwrite 0x0230d4 thru 0x03039e (0xd2cb bytes) 000322 1761238661.243574 write 0x02c408 thru 0x034289 (0x7e82 bytes) 000326 1761238661.250707 MIRROR_RESYNC 4 mirrors 000328 1761238662.057023 read 0x025bb5 thru 0x02ffc9 (0xa415 bytes) 000333 1761238662.066991 mapread 0x01abbc thru 0x024aa7 (0x9eec bytes) 000336 1761238662.089670 mapwrite 0x0242eb thru 0x02e0e4 (0x9dfa bytes) 000338 1761238662.258727 mapwrite 0x000f18 thru 0x00b9d7 (0xaac0 bytes) 000343 1761238662.333756 trunc from 0x03f022 to 0x02dbc3 (0x1145e bytes) 000344 1761238662.398006 mapwrite 0x0124bc thru 0x020842 (0xe387 bytes) 000345 1761238662.511596 write 0x00a088 thru 0x014773 (0xa6ec bytes) 000346 1761238662.512938 mapread 0x004cab thru 0x01236e (0xd6c4 bytes) 000347 1761238662.515952 trunc from 0x02dbc4 to 0x015d53 (0x17e70 bytes) 000349 1761238662.564008 read 0x007891 thru 0x00e081 (0x67f1 bytes) 000350 1761238662.565003 mapwrite 0x02441d thru 0x025e9d (0x1a81 bytes) 000351 1761238662.640052 write 0x024754 thru 0x033b1a (0xf3c7 bytes) 000353 1761238662.642148 write 0x02053d thru 0x027d0d (0x77d1 bytes) 000354 1761238662.643245 write 0x003bcc thru 0x0108c0 (0xccf5 bytes) 000355 1761238662.649200 MIRROR_RESYNC 4 mirrors 000357 1761238663.415650 MIRROR_RESYNC 4 mirrors 000358 1761238663.680288 trunc from 0x033b1b to 0x00ce29 (0x26cf1 bytes) 000362 1761238663.752480 mapwrite 0x01698b thru 0x01ccfd (0x6373 bytes) 000364 1761238663.886668 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000366 1761238664.142804 read 0x009d88 thru 0x016f25 (0xd19e bytes) 000367 1761238664.163199 write 0x00cb35 thru 0x017639 (0xab05 bytes) 000368 1761238664.178731 trunc from 0x01ccfe to 0x029ba0 (0xcea3 bytes) 000370 1761238664.199118 mapread 0x0110a5 thru 0x0132e7 (0x2243 bytes) 000371 1761238664.200187 mapwrite 0x010234 thru 0x01881d (0x85ea bytes) 000372 1761238664.318834 write 0x0301be thru 0x037825 (0x7668 bytes) 000373 1761238664.350041 MIRROR_EXTEND to 4 mirrors 000374 1761238664.901464 read 0x02bdf0 thru 0x02e08a (0x229b bytes) 000375 1761238664.907387 write 0x039dbc thru 0x03a73a (0x97f bytes) 000379 1761238664.974946 write 0x024fd6 thru 0x0309da (0xba05 bytes) 000380 1761238664.988456 trunc from 0x03a73b to 0x001e2b (0x3890f bytes) 000381 1761238665.029807 mapwrite 0x03d4a2 thru 0x03ffff (0x2b5e bytes) 000386 1761238665.109805 mapread 0x014780 thru 0x0167f3 (0x2074 bytes) 000387 1761238665.124394 read 0x03eb03 thru 0x03ffff (0x14fd bytes) 000388 1761238665.125104 read 0x033c23 thru 0x03e5d9 (0xa9b7 bytes) 000390 1761238665.138062 trunc from 0x040000 to 0x03815a (0x7ea5 bytes) 000394 1761238665.165307 mapread 0x008d19 thru 0x016f1e (0xe206 bytes) 000395 1761238665.169011 trunc from 0x03815b to 0x027531 (0x10c29 bytes) 000396 1761238665.194579 read 0x00d80f thru 0x01d20e (0xfa00 bytes) 000399 1761238665.202483 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000400 1761238665.473893 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000401 1761238665.709933 write 0x02203c thru 0x02f047 (0xd00c bytes) 000403 1761238665.716872 write 0x00234a thru 0x002443 (0xfa bytes) 000404 1761238665.727685 mapwrite 0x034354 thru 0x03a0af (0x5d5c bytes) 000405 1761238665.804648 trunc from 0x03a0b0 to 0x03caaa (0x29fb bytes) 000406 1761238665.836705 trunc from 0x03caab to 0x02fd4b (0xcd5f bytes) 000410 1761238665.859706 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000412 1761238666.022585 trunc from 0x02fd4c to 0x035e88 (0x613d bytes) 000413 1761238666.054055 trunc from 0x035e89 to 0x01b80a (0x1a67e bytes) 000414 1761238666.089138 mapread 0x01b421 thru 0x01b80a (0x3ea bytes) 000415 1761238666.099425 read 0x019df9 thru 0x01b80a (0x1a12 bytes) 000416 1761238666.108838 mapwrite 0x0255f2 thru 0x02abf7 (0x5606 bytes) 000417 1761238666.186621 trunc from 0x02abf8 to 0x014ad6 (0x16121 bytes) 000420 1761238666.203435 read 0x007d95 thru 0x013a18 (0xbc84 bytes) 000421 1761238666.215214 trunc from 0x014ad7 to 0x020e15 (0xc33f bytes) 000427 1761238666.236239 trunc from 0x020e16 to 0x0291c5 (0x83b0 bytes) 000428 1761238666.259105 mapwrite 0x0069c8 thru 0x00c5eb (0x5c24 bytes) 000430 1761238666.308557 mapread 0x011571 thru 0x0150fc (0x3b8c bytes) 000431 1761238666.318329 trunc from 0x0291c6 to 0x0232b8 (0x5f0d bytes) 000432 1761238666.339704 write 0x03bdde thru 0x03f226 (0x3449 bytes) 000433 1761238666.341746 trunc from 0x03f227 to 0x022d9f (0x1c487 bytes) 000434 1761238666.357925 write 0x033de1 thru 0x037b3d (0x3d5d bytes) 000439 1761238666.360840 read 0x006b22 thru 0x00c52c (0x5a0b bytes) 000440 1761238666.369373 MIRROR_EXTEND to 2 mirrors 000442 1761238666.978454 mapread 0x02fbce thru 0x035dbe (0x61f1 bytes) 000443 1761238667.012923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000444 1761238667.183952 mapread 0x008a64 thru 0x00966b (0xc08 bytes) 000446 1761238667.196795 mapread 0x01920e thru 0x022abe (0x98b1 bytes) 000447 1761238667.214925 mapread 0x013063 thru 0x019707 (0x66a5 bytes) 000448 1761238667.217185 mapwrite 0x02dcc5 thru 0x03ac48 (0xcf84 bytes) 000449 1761238667.320456 mapread 0x031237 thru 0x0391c3 (0x7f8d bytes) 000450 1761238667.321436 read 0x013e61 thru 0x0203d8 (0xc578 bytes) 000453 1761238667.332498 mapwrite 0x01554b thru 0x017947 (0x23fd bytes) 000455 1761238667.371741 trunc from 0x03ac49 to 0x02602e (0x14c1a bytes) 000457 1761238667.396566 trunc from 0x02602f to 0x02d04a (0x701c bytes) 000458 1761238667.418768 mapwrite 0x00a875 thru 0x00e37c (0x3b08 bytes) 000459 1761238667.474862 mapread 0x027220 thru 0x02c942 (0x5723 bytes) 000460 1761238667.485717 trunc from 0x02d04b to 0x01fc23 (0xd427 bytes) 000461 1761238667.505800 write 0x001a6d thru 0x00d93c (0xbed0 bytes) 000463 1761238667.518394 mapwrite 0x030ad1 thru 0x0364e7 (0x5a17 bytes) 000465 1761238667.621779 mapread 0x01d57e thru 0x029ee0 (0xc963 bytes) 000466 1761238667.635688 mapread 0x02dc72 thru 0x02fa11 (0x1da0 bytes) 000468 1761238667.637500 trunc from 0x0364e8 to 0x00c2cb (0x2a21c bytes) 000469 1761238667.670730 trunc from 0x00c2cc to 0x036051 (0x29d86 bytes) 000470 1761238667.696405 write 0x0387ba thru 0x03ffff (0x7846 bytes) 000471 1761238667.698386 trunc from 0x040000 to 0x03ef7e (0x1081 bytes) 000472 1761238667.721439 read 0x01efb0 thru 0x02c13e (0xd18f bytes) 000474 1761238667.739258 mapread 0x005401 thru 0x00d920 (0x8520 bytes) 000475 1761238667.740244 mapread 0x033c60 thru 0x03aeda (0x727b bytes) 000476 1761238667.744149 mapwrite 0x018762 thru 0x023f0d (0xb7ac bytes) 000477 1761238667.805492 write 0x03bd70 thru 0x03ffff (0x4290 bytes) 000478 1761238667.806587 mapread 0x002cce thru 0x01223b (0xf56e bytes) 000481 1761238667.809730 write 0x00cd67 thru 0x00f1d9 (0x2473 bytes) 000486 1761238667.815047 trunc from 0x040000 to 0x03e144 (0x1ebb bytes) 000489 1761238667.829808 mapread 0x00afa4 thru 0x016c11 (0xbc6e bytes) 000491 1761238667.831182 trunc from 0x03e145 to 0x0153e7 (0x28d5d bytes) 000494 1761238667.848471 read 0x0015ee thru 0x0035c7 (0x1fda bytes) 000495 1761238667.852071 write 0x005ef1 thru 0x007d0b (0x1e1b bytes) 000497 1761238667.854680 mapwrite 0x01fbc5 thru 0x020f58 (0x1394 bytes) 000498 1761238667.961348 trunc from 0x020f59 to 0x00f59d (0x119bb bytes) 000501 1761238667.990160 trunc from 0x00f59e to 0x0011bd (0xe3e0 bytes) 000502 1761238668.022069 mapwrite 0x00dbe9 thru 0x01d016 (0xf42e bytes) 000504 1761238668.249206 MIRROR_EXTEND to 2 mirrors 000505 1761238668.848187 MIRROR_EXTEND to 3 mirrors 000506 1761238669.392281 read 0x0031a8 thru 0x009a8e (0x68e7 bytes) 000508 1761238669.403160 MIRROR_EXTEND to 4 mirrors 000509 1761238670.031507 read 0x00df40 thru 0x015441 (0x7502 bytes) 000510 1761238670.039434 read 0x0187cf thru 0x01abbd (0x23ef bytes) 000511 1761238670.047063 mapwrite 0x03788d thru 0x03ffff (0x8773 bytes) 000513 1761238670.221329 trunc from 0x040000 to 0x023447 (0x1cbb8 bytes) 000519 1761238670.255254 MIRROR_EXTEND to 5 mirrors 000522 1761238670.818793 read 0x018e16 thru 0x01c182 (0x336d bytes) 000523 1761238670.844454 write 0x01dcd6 thru 0x026c18 (0x8f43 bytes) 000526 1761238670.926151 trunc from 0x026c19 to 0x03c366 (0x1574e bytes) 000527 1761238670.957979 mapwrite 0x00e349 thru 0x01c517 (0xe1cf bytes) 000528 1761238671.068979 trunc from 0x03c367 to 0x03e534 (0x21ce bytes) 000529 1761238671.095807 mapwrite 0x013f40 thru 0x01d04f (0x9110 bytes) 000530 1761238671.164083 mapwrite 0x01b157 thru 0x0226b8 (0x7562 bytes) 000531 1761238671.211468 mapread 0x004972 thru 0x013905 (0xef94 bytes) 000533 1761238671.223786 mapwrite 0x00f386 thru 0x01b6fd (0xc378 bytes) 000534 1761238671.271257 mapread 0x001434 thru 0x00f473 (0xe040 bytes) 000535 1761238671.272238 read 0x00752f thru 0x01314a (0xbc1c bytes) 000536 1761238671.282276 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000537 1761238671.521281 write 0x013009 thru 0x0182d5 (0x52cd bytes) 000539 1761238671.532603 trunc from 0x03e535 to 0x039780 (0x4db4 bytes) 000540 1761238671.563702 MIRROR_EXTEND to 5 mirrors 000544 1761238672.307985 mapwrite 0x004ecc thru 0x010a23 (0xbb58 bytes) 000545 1761238672.498365 write 0x03c035 thru 0x03ffff (0x3fcb bytes) 000548 1761238672.500489 read 0x00d687 thru 0x00fe3e (0x27b8 bytes) 000549 1761238672.501766 write 0x01ac3c thru 0x023f55 (0x931a bytes) 000550 1761238672.516127 mapread 0x0158f5 thru 0x01c7a3 (0x6eaf bytes) 000551 1761238672.517888 write 0x030b6e thru 0x03ca5b (0xbeee bytes) 000553 1761238672.530118 trunc from 0x040000 to 0x0326ae (0xd951 bytes) 000558 1761238672.558787 mapwrite 0x00a27c thru 0x013689 (0x940e bytes) 000559 1761238672.616694 read 0x004f00 thru 0x007dde (0x2edf bytes) 000562 1761238672.618278 mapread 0x031611 thru 0x031e0e (0x7fe bytes) 000565 1761238672.625407 write 0x029fac thru 0x0314d5 (0x752a bytes) 000569 1761238672.640799 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000570 1761238672.881683 trunc from 0x0326af to 0x03119e (0x1510 bytes) 000573 1761238672.908218 mapread 0x0099f9 thru 0x019412 (0xfa1a bytes) 000574 1761238672.969263 MIRROR_RESYNC 4 mirrors 000576 1761238673.803809 mapwrite 0x01e54f thru 0x028ae4 (0xa596 bytes) 000583 1761238673.969774 trunc from 0x03119f to 0x039bae (0x8a10 bytes) 000584 1761238674.002870 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000585 1761238674.219992 mapwrite 0x036dd1 thru 0x03b21b (0x444b bytes) 000586 1761238674.308358 mapread 0x010950 thru 0x0183ec (0x7a9d bytes) 000588 1761238674.317504 mapread 0x01660f thru 0x024c90 (0xe682 bytes) 000590 1761238674.332365 trunc from 0x03b21c to 0x0084ea (0x32d31 bytes) 000591 1761238674.348580 mapread 0x00123c thru 0x0084ea (0x72af bytes) 000592 1761238674.366397 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000597 1761238674.561053 write 0x003e76 thru 0x0126b4 (0xe83f bytes) 000598 1761238674.573468 mapread 0x00b9c6 thru 0x0126b4 (0x6cef bytes) 000604 1761238674.578435 read 0x0115f5 thru 0x0126b4 (0x10c0 bytes) 000605 1761238674.580209 write 0x02a5e4 thru 0x036db0 (0xc7cd bytes) 000606 1761238674.581699 read 0x0018ab thru 0x008f80 (0x76d6 bytes) 000609 1761238674.595028 mapwrite 0x00a076 thru 0x0135a6 (0x9531 bytes) 000611 1761238674.636822 mapwrite 0x000e4d thru 0x01016a (0xf31e bytes) 000614 1761238674.692282 trunc from 0x036db1 to 0x002648 (0x34768 bytes) 000615 1761238674.717114 mapread 0x000683 thru 0x002648 (0x1fc6 bytes) 000618 1761238674.719101 mapread 0x000d6d thru 0x00124c (0x4e0 bytes) 000619 1761238674.727117 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000620 1761238674.867945 write 0x01b182 thru 0x0271b4 (0xc033 bytes) 000621 1761238674.869423 mapread 0x025070 thru 0x0271b4 (0x2145 bytes) 000622 1761238674.870226 mapread 0x01df6b thru 0x026344 (0x83da bytes) 000623 1761238674.871821 trunc from 0x0271b5 to 0x014ef1 (0x122c3 bytes) 000624 1761238674.891332 trunc from 0x014ef2 to 0x019589 (0x4698 bytes) 000625 1761238674.908278 trunc from 0x01958a to 0x0178ae (0x1cdb bytes) 000628 1761238674.924900 mapwrite 0x02f0e0 thru 0x031670 (0x2591 bytes) 000630 1761238675.008390 mapread 0x00eb74 thru 0x019fca (0xb457 bytes) 000631 1761238675.039843 mapread 0x007394 thru 0x008acf (0x173c bytes) 000632 1761238675.041276 mapread 0x0114b7 thru 0x014f35 (0x3a7f bytes) 000633 1761238675.043422 write 0x030171 thru 0x03e670 (0xe500 bytes) 000634 1761238675.045192 mapwrite 0x011224 thru 0x01d497 (0xc274 bytes) 000637 1761238675.094456 trunc from 0x03e671 to 0x0356f6 (0x8f7a bytes) 000643 1761238675.110928 read 0x0237fd thru 0x02d2ec (0x9af0 bytes) 000646 1761238675.124303 write 0x02dc14 thru 0x03b0eb (0xd4d8 bytes) 000648 1761238675.127456 write 0x017996 thru 0x01c8a6 (0x4f11 bytes) 000649 1761238675.129164 mapread 0x032437 thru 0x0345eb (0x21b5 bytes) 000650 1761238675.130194 trunc from 0x03b0ec to 0x0100ec (0x2afff bytes) 000651 1761238675.145627 trunc from 0x0100ed to 0x029bdf (0x19af3 bytes) 000654 1761238675.165245 read 0x0229d1 thru 0x0270d7 (0x4707 bytes) 000655 1761238675.178864 trunc from 0x029be0 to 0x033319 (0x973a bytes) 000656 1761238675.193718 trunc from 0x03331a to 0x010ad6 (0x22843 bytes) 000658 1761238675.210466 mapwrite 0x022672 thru 0x0269bb (0x434a bytes) 000660 1761238675.323095 trunc from 0x0269bc to 0x01a6f3 (0xc2c8 bytes) 000662 1761238675.339639 read 0x00c629 thru 0x015583 (0x8f5b bytes) 000664 1761238675.347141 read 0x005d08 thru 0x00bed5 (0x61ce bytes) 000666 1761238675.348227 mapread 0x00e9d8 thru 0x01642d (0x7a56 bytes) 000669 1761238675.350229 write 0x003a62 thru 0x012a15 (0xefb4 bytes) 000670 1761238675.351727 mapwrite 0x01a2e1 thru 0x0282f7 (0xe017 bytes) 000672 1761238675.449566 mapread 0x0029fc thru 0x005bf1 (0x31f6 bytes) 000675 1761238675.452571 mapwrite 0x017906 thru 0x021435 (0x9b30 bytes) 000676 1761238675.512447 trunc from 0x0282f8 to 0x028bf2 (0x8fb bytes) 000679 1761238675.534445 mapwrite 0x03cd70 thru 0x03ffff (0x3290 bytes) 000680 1761238675.601764 read 0x03405f thru 0x0399ba (0x595c bytes) 000682 1761238675.603924 trunc from 0x040000 to 0x01b111 (0x24eee bytes) 000683 1761238675.633084 read 0x00b871 thru 0x0108d8 (0x5068 bytes) 000684 1761238675.633807 read 0x010156 thru 0x01b111 (0xafbc bytes) 000688 1761238675.637253 trunc from 0x01b112 to 0x038607 (0x1d4f6 bytes) 000689 1761238675.654740 write 0x02e7fd thru 0x03d4ba (0xecbe bytes) 000691 1761238675.664133 mapread 0x01c128 thru 0x02bf3e (0xfe17 bytes) 000692 1761238675.665106 mapread 0x02f0ec thru 0x03c168 (0xd07d bytes) 000693 1761238675.666119 mapwrite 0x019420 thru 0x024eab (0xba8c bytes) 000695 1761238675.724429 mapread 0x01607e thru 0x0187a2 (0x2725 bytes) 000696 1761238675.725237 mapwrite 0x03972c thru 0x03ffff (0x68d4 bytes) 000698 1761238675.823838 mapread 0x007140 thru 0x00f4fc (0x83bd bytes) 000699 1761238675.824673 trunc from 0x040000 to 0x00cd06 (0x332f9 bytes) 000703 1761238675.848219 mapread 0x006dbf thru 0x00cd06 (0x5f48 bytes) 000707 1761238675.851125 mapwrite 0x027d8e thru 0x02c955 (0x4bc8 bytes) 000708 1761238675.948916 MIRROR_EXTEND to 2 mirrors 000709 1761238676.529268 read 0x025435 thru 0x02c955 (0x7521 bytes) 000710 1761238676.538149 mapwrite 0x009f9b thru 0x019b1d (0xfb83 bytes) 000711 1761238676.766675 MIRROR_RESYNC 2 mirrors 000712 1761238677.381372 mapread 0x013ef9 thru 0x01d213 (0x931b bytes) 000713 1761238677.405332 trunc from 0x02c956 to 0x021bc2 (0xad93 bytes) 000714 1761238677.490282 mapread 0x00ec9c thru 0x00ed68 (0xcd bytes) 000715 1761238677.500879 trunc from 0x021bc3 to 0x005398 (0x1c82a bytes) 000717 1761238677.528314 mapwrite 0x02795f thru 0x035a0b (0xe0ad bytes) 000718 1761238677.634699 read 0x02e460 thru 0x033956 (0x54f7 bytes) 000720 1761238677.635766 trunc from 0x035a0c to 0x018c5b (0x1cdb0 bytes) 000721 1761238677.650400 mapwrite 0x03ef79 thru 0x03ffff (0x1087 bytes) 000722 1761238677.724561 trunc from 0x040000 to 0x00ddad (0x32252 bytes) 000723 1761238677.749585 mapwrite 0x00e9c1 thru 0x010fca (0x260a bytes) 000724 1761238677.809295 mapread 0x00e050 thru 0x010fca (0x2f7b bytes) 000726 1761238677.810926 mapread 0x00b4b0 thru 0x00be16 (0x967 bytes) 000727 1761238677.811715 mapwrite 0x0002e2 thru 0x009ede (0x9bfd bytes) 000729 1761238677.867422 mapread 0x0000f4 thru 0x002d90 (0x2c9d bytes) 000730 1761238677.868208 mapwrite 0x000e90 thru 0x002073 (0x11e4 bytes) 000731 1761238677.913077 write 0x023bf8 thru 0x029a04 (0x5e0d bytes) 000732 1761238677.914210 write 0x02ca78 thru 0x033912 (0x6e9b bytes) 000733 1761238677.915236 trunc from 0x033913 to 0x01f80b (0x14107 bytes) 000734 1761238677.941424 MIRROR_RESYNC 2 mirrors 000735 1761238678.486617 read 0x0177cf thru 0x01f80b (0x803d bytes) 000737 1761238678.504168 MIRROR_EXTEND to 3 mirrors 000738 1761238679.140499 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000740 1761238679.535579 mapread 0x0044a5 thru 0x011bae (0xd70a bytes) 000741 1761238679.570151 MIRROR_RESYNC 2 mirrors 000742 1761238679.770501 mapwrite 0x02962e thru 0x0344b8 (0xae8b bytes) 000744 1761238680.003306 trunc from 0x0344b9 to 0x00b7e9 (0x28ccf bytes) 000745 1761238680.030487 write 0x00bfed thru 0x011ff2 (0x6006 bytes) 000747 1761238680.048010 mapread 0x00b93f thru 0x011ff2 (0x66b4 bytes) 000748 1761238680.052989 MIRROR_RESYNC 2 mirrors 000750 1761238680.691161 write 0x00fc04 thru 0x0170fd (0x74fa bytes) 000751 1761238680.764140 mapwrite 0x03fbee thru 0x03ffff (0x412 bytes) 000752 1761238680.875775 read 0x03dc6b thru 0x03ffff (0x2395 bytes) 000753 1761238680.891301 mapread 0x032812 thru 0x0355ec (0x2ddb bytes) 000754 1761238680.906609 mapread 0x013d5d thru 0x021288 (0xd52c bytes) 000755 1761238680.927815 MIRROR_RESYNC 2 mirrors 000756 1761238681.587211 MIRROR_RESYNC 2 mirrors 000757 1761238681.794042 mapwrite 0x00aa99 thru 0x01430f (0x9877 bytes) 000759 1761238681.980985 mapwrite 0x014b83 thru 0x01a83b (0x5cb9 bytes) 000762 1761238682.067571 mapread 0x03cb14 thru 0x03e74f (0x1c3c bytes) 000764 1761238682.081548 mapwrite 0x01db63 thru 0x025905 (0x7da3 bytes) 000768 1761238682.155215 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000772 1761238682.363352 mapread 0x00bf20 thru 0x018729 (0xc80a bytes) 000775 1761238682.390201 mapread 0x03384a thru 0x03e113 (0xa8ca bytes) 000776 1761238682.414376 mapread 0x034080 thru 0x03ede8 (0xad69 bytes) 000778 1761238682.419946 MIRROR_EXTEND to 2 mirrors 000779 1761238683.039301 trunc from 0x040000 to 0x013aea (0x2c515 bytes) 000780 1761238683.153898 trunc from 0x013aeb to 0x011def (0x1cfb bytes) 000781 1761238683.187855 read 0x00ce9e thru 0x011def (0x4f52 bytes) 000783 1761238683.195605 write 0x0366b4 thru 0x03849d (0x1dea bytes) 000784 1761238683.196604 write 0x03cb69 thru 0x03ffff (0x3497 bytes) 000785 1761238683.198755 mapread 0x018419 thru 0x02029e (0x7e86 bytes) 000786 1761238683.243430 read 0x016fc9 thru 0x01c669 (0x56a1 bytes) 000788 1761238683.245160 mapread 0x0166a1 thru 0x018fe3 (0x2943 bytes) 000789 1761238683.246427 mapread 0x000ae9 thru 0x008a9e (0x7fb6 bytes) 000793 1761238683.256666 write 0x033e17 thru 0x03d069 (0x9253 bytes) 000794 1761238683.257995 trunc from 0x040000 to 0x03a7ee (0x5811 bytes) 000798 1761238683.278148 trunc from 0x03a7ef to 0x019d16 (0x20ad8 bytes) 000799 1761238683.314042 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000800 1761238683.460215 read 0x00d639 thru 0x016137 (0x8aff bytes) 000803 1761238683.470961 trunc from 0x019d17 to 0x037bd9 (0x1dec3 bytes) 000804 1761238683.488335 trunc from 0x037bda to 0x017451 (0x20788 bytes) 000806 1761238683.504820 mapwrite 0x03784e thru 0x03a566 (0x2d19 bytes) 000808 1761238683.588249 mapwrite 0x00d120 thru 0x014854 (0x7735 bytes) 000809 1761238683.628456 write 0x03cb12 thru 0x03ffff (0x34ee bytes) 000811 1761238683.630132 mapwrite 0x021c51 thru 0x0308c4 (0xec74 bytes) 000812 1761238683.696005 mapread 0x01d329 thru 0x024e8a (0x7b62 bytes) 000817 1761238683.700203 mapwrite 0x0370ba thru 0x03cda0 (0x5ce7 bytes) 000821 1761238683.762393 mapwrite 0x026eca thru 0x02d2e8 (0x641f bytes) 000822 1761238683.813601 mapwrite 0x01ff0f thru 0x02db61 (0xdc53 bytes) 000823 1761238683.873479 mapread 0x03a550 thru 0x03c051 (0x1b02 bytes) 000825 1761238683.881007 MIRROR_EXTEND to 2 mirrors 000826 1761238684.657956 mapwrite 0x0080fe thru 0x00d310 (0x5213 bytes) 000827 1761238684.821647 write 0x00f8f1 thru 0x017a37 (0x8147 bytes) 000828 1761238684.829885 write 0x0134e5 thru 0x018ffd (0x5b19 bytes) 000831 1761238684.846333 MIRROR_EXTEND to 3 mirrors 000834 1761238685.757778 mapread 0x0349ca thru 0x036237 (0x186e bytes) 000835 1761238685.780841 mapread 0x00fa8f thru 0x01068f (0xc01 bytes) 000836 1761238685.797235 mapwrite 0x03da5e thru 0x03e9ed (0xf90 bytes) 000837 1761238685.948760 mapread 0x01c180 thru 0x01f529 (0x33aa bytes) 000838 1761238685.961688 write 0x020c7f thru 0x0277f9 (0x6b7b bytes) 000840 1761238685.967824 mapwrite 0x00d6f9 thru 0x00e01f (0x927 bytes) 000842 1761238686.036103 trunc from 0x040000 to 0x017ece (0x28131 bytes) 000844 1761238686.065170 write 0x01f99f thru 0x02061c (0xc7e bytes) 000846 1761238686.066777 read 0x01daff thru 0x01de3b (0x33d bytes) 000849 1761238686.073292 mapread 0x01b16d thru 0x02061c (0x54b0 bytes) 000851 1761238686.092370 MIRROR_RESYNC 3 mirrors 000858 1761238686.700587 mapwrite 0x038de6 thru 0x03d601 (0x481c bytes) 000859 1761238686.848910 trunc from 0x03d602 to 0x03e89a (0x1299 bytes) 000860 1761238686.875798 trunc from 0x03e89b to 0x0207ff (0x1e09b bytes) 000861 1761238686.893713 write 0x02e520 thru 0x03489a (0x637b bytes) 000862 1761238686.895259 trunc from 0x03489b to 0x024400 (0x1049a bytes) 000863 1761238686.909594 mapwrite 0x02095c thru 0x02292c (0x1fd1 bytes) 000866 1761238686.957688 trunc from 0x024401 to 0x02aa8a (0x668a bytes) 000868 1761238686.978132 trunc from 0x02aa8b to 0x001368 (0x29722 bytes) 000871 1761238687.003593 mapwrite 0x016a65 thru 0x01d377 (0x6913 bytes) 000872 1761238687.102007 mapread 0x00a73a thru 0x012199 (0x7a60 bytes) 000874 1761238687.114841 write 0x016170 thru 0x01ded8 (0x7d69 bytes) 000876 1761238687.116362 mapread 0x0069db thru 0x015a75 (0xf09b bytes) 000879 1761238687.118690 trunc from 0x01ded9 to 0x031784 (0x138ac bytes) 000880 1761238687.139666 mapread 0x024e51 thru 0x02718c (0x233c bytes) 000883 1761238687.150093 write 0x01a578 thru 0x02050b (0x5f94 bytes) 000884 1761238687.151391 trunc from 0x031785 to 0x02c434 (0x5350 bytes) 000885 1761238687.166100 mapwrite 0x0190ea thru 0x027000 (0xdf17 bytes) 000887 1761238687.214578 mapread 0x0087e2 thru 0x0110d4 (0x88f3 bytes) 000889 1761238687.215696 write 0x008826 thru 0x00af08 (0x26e3 bytes) 000892 1761238687.217388 trunc from 0x02c435 to 0x02a905 (0x1b2f bytes) 000894 1761238687.232346 trunc from 0x02a906 to 0x02494d (0x5fb8 bytes) 000895 1761238687.245434 read 0x01d165 thru 0x01d64f (0x4eb bytes) 000896 1761238687.246088 read 0x001e7c thru 0x004bf8 (0x2d7d bytes) 000897 1761238687.256480 read 0x00e275 thru 0x016bcc (0x8958 bytes) 000899 1761238687.257575 trunc from 0x02494e to 0x038f28 (0x145db bytes) 000900 1761238687.280752 mapwrite 0x038806 thru 0x03ffff (0x77fa bytes) 000902 1761238687.349011 trunc from 0x040000 to 0x0205d5 (0x1fa2a bytes) 000904 1761238687.375957 MIRROR_EXTEND to 4 mirrors 000906 1761238687.862075 mapread 0x018f0e thru 0x0205d5 (0x76c8 bytes) 000907 1761238687.884965 MIRROR_RESYNC 4 mirrors 000908 1761238688.484084 MIRROR_RESYNC 4 mirrors 000909 1761238688.697890 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000911 1761238688.999008 mapread 0x019bde thru 0x0205d5 (0x69f8 bytes) 000914 1761238689.040017 mapread 0x000016 thru 0x00dc34 (0xdc1f bytes) 000915 1761238689.056585 trunc from 0x0205d6 to 0x01c4bf (0x4116 bytes) 000916 1761238689.125298 write 0x00e7c8 thru 0x00faae (0x12e7 bytes) 000917 1761238689.133175 write 0x0280d2 thru 0x0312c9 (0x91f8 bytes) 000919 1761238689.138230 MIRROR_RESYNC 3 mirrors 000920 1761238689.745243 mapwrite 0x01ea62 thru 0x024344 (0x58e3 bytes) 000921 1761238689.909020 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000922 1761238690.095834 read 0x01620c thru 0x0189e1 (0x27d6 bytes) 000925 1761238690.103795 trunc from 0x0312ca to 0x01e215 (0x130b4 bytes) 000927 1761238690.120199 write 0x00cf59 thru 0x0114ec (0x4594 bytes) 000928 1761238690.133050 read 0x008516 thru 0x00c307 (0x3df2 bytes) 000931 1761238690.142977 mapwrite 0x000f90 thru 0x002683 (0x16f4 bytes) 000932 1761238690.209679 mapread 0x01027b thru 0x0186f9 (0x847f bytes) 000933 1761238690.210662 mapread 0x00afeb thru 0x00b447 (0x45d bytes) 000934 1761238690.211426 mapwrite 0x018b20 thru 0x019926 (0xe07 bytes) 000935 1761238690.253327 read 0x00afba thru 0x011fb7 (0x6ffe bytes) 000936 1761238690.254034 read 0x00d446 thru 0x016e74 (0x9a2f bytes) 000937 1761238690.254666 mapwrite 0x03f40d thru 0x03ffff (0xbf3 bytes) 000940 1761238690.321158 trunc from 0x040000 to 0x03c1e9 (0x3e16 bytes) 000942 1761238690.337074 mapwrite 0x034be7 thru 0x03f424 (0xa83e bytes) 000944 1761238690.400255 trunc from 0x03f425 to 0x03601f (0x9405 bytes) 000946 1761238690.422115 mapwrite 0x03cd12 thru 0x03ffff (0x32ee bytes) 000947 1761238690.513659 MIRROR_EXTEND to 3 mirrors 000948 1761238691.003009 trunc from 0x040000 to 0x004e58 (0x3b1a7 bytes) 000949 1761238691.132386 write 0x0160e9 thru 0x0197f3 (0x370b bytes) 000950 1761238691.133502 write 0x011a67 thru 0x018996 (0x6f30 bytes) 000951 1761238691.141306 read 0x014d17 thru 0x0197f3 (0x4add bytes) 000952 1761238691.146374 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000953 1761238691.346036 read 0x006aca thru 0x008300 (0x1837 bytes) 000956 1761238691.353728 read 0x00e2da thru 0x015899 (0x75c0 bytes) 000957 1761238691.367914 MIRROR_RESYNC 2 mirrors 000962 1761238691.884415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) 000963 1761238692.005648 mapread 0x00631d thru 0x009940 (0x3624 bytes) 000964 1761238692.025188 mapread 0x011aa5 thru 0x0197f3 (0x7d4f bytes) 000965 1761238692.032447 mapwrite 0x000dd6 thru 0x00400f (0x323a bytes) 000966 1761238692.087397 write 0x02431a thru 0x027b55 (0x383c bytes) 000970 1761238692.090261 mapread 0x022534 thru 0x027b55 (0x5622 bytes) 000972 1761238692.099498 mapwrite 0x028143 thru 0x036512 (0xe3d0 bytes) 000978 1761238692.192958 mapread 0x010b24 thru 0x0152ea (0x47c7 bytes) 000981 1761238692.201735 write 0x01efb9 thru 0x02cbab (0xdbf3 bytes) 000982 1761238692.204081 read 0x02c049 thru 0x02d078 (0x1030 bytes) 000985 1761238692.205672 mapread 0x004088 thru 0x00e597 (0xa510 bytes) 000987 1761238692.207552 read 0x00faf5 thru 0x017111 (0x761d bytes) 000989 1761238692.208647 mapread 0x02978e thru 0x036512 (0xcd85 bytes) 000991 1761238692.210493 read 0x00538f thru 0x007210 (0x1e82 bytes) 000992 1761238692.211652 mapread 0x01e434 thru 0x025f96 (0x7b63 bytes) 000993 1761238692.212682 trunc from 0x036513 to 0x0246d3 (0x11e3f bytes) 000996 1761238692.232101 trunc from 0x0246d4 to 0x019617 (0xb0bc bytes) 000997 1761238692.247656 read 0x00a6f8 thru 0x00b304 (0xc0d bytes) 000999 1761238692.248855 read 0x0032b1 thru 0x007161 (0x3eb1 bytes) 001000 1761238692.249418 read 0x008175 thru 0x013d11 (0xbb9d bytes) All operations completed A-OK! PASS 70a (52s) == sanity-flr test 71: check mirror extend parent fid ==== 12:58:15 (1761238695) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 12:58:18 (1761238698) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2148800 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3564064 .. failed Extending file size to 8344800 .. Extending file size to 5829248 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8534400 .. Extending file size to 8278560 .. Extending file size to 8996864 .. Extending file size to 8596832 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8620128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9028224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9189824 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 54927 54929 54930 54932 54933 done 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) 192.168.206.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:59:30 (1761238770) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1634688 .. Extending file size to 8621088 .. done Extending file size to 8542976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8823488 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8942912 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9126912 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57142 57143 57144 57147 57148 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) 192.168.206.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:00:42 (1761238842) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106077 s, 19.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320912 s, 6.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:00:56 (1761238856) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0695652 s, 30.1 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 13:01:01 (1761238861) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:01:07 (1761238867) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143929 s, 14.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:01:13 (1761238873) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21864 s, 9.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:01:18 (1761238878) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194811 s, 10.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:01:49 (1761238909) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884532 s, 11.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:02:46 (1761238966) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223134 s, 9.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:03:40 (1761239020) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. Creating new pool test_204f oleg647-server: Pool lustre.test_204f created Adding targets to pool oleg647-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg647-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298766 s, 7.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg647-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg647-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg647-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:04:52 (1761239092) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:04:58 (1761239098) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:05:03 (1761239103) oleg647-server: Pool lustre.test_206 created Adding targets to pool oleg647-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg647-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg647-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg647-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg647-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:05:24 (1761239124) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639123 s, 16.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468424 s, 22.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x36002f69 Mirror 3: 0x36002f69 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:05:31 (1761239131) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:05:34 (1761239134) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:05:36 (1761239136) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0991737 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x780000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1640000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1640000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1640000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2140000 Read this (4 bytes): BBBB 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:06:07 (1761239167) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0350544 s, 234 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:06:13 (1761239173) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551773 s, 19.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x250 lmm_fid: [0x200000404:0x250:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x250 lmm_fid: [0x200000404:0x250:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834512 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-23 13:06:15.000000000 -0400 Modify: 2025-10-23 13:06:15.000000000 -0400 Change: 2025-10-23 13:06:16.000000000 -0400 Birth: 2025-10-23 13:06:13.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:06:21 (1761239181) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:06:25 (1761239185) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.414676 s, 98.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0478784 s, 85.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 23 13:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x252:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 23 13:06 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test complete, duration 1399 sec =========== 13:06:33 (1761239193) === sanity-flr: start cleanup 13:06:35 (1761239195) === === sanity-flr: finish cleanup 13:06:41 (1761239201) ===