-----============= acceptance-small: sanity-flr ============----- Fri Oct 31 22:07:09 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg337-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:07:15 (1761962835) === oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff932487daf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff932487daf800.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:07:25 (1761962845) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7418] == sanity-flr test 0a: lfs mirror create with -N option == 22:07:25 (1761962845) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:07:30 (1761962850) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:07:31 (1761962851) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:07:32 (1761962852) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:07:38 (1761962858) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:07:39 (1761962859) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 22:07:40 (1761962860) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff932487daf800.stats=clear osc.lustre-OST0001-osc-ffff932487daf800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:07:43 (1761962863) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:07:46 (1761962866) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:07:48 (1761962868) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:07:49 (1761962869) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 22:07:52 (1761962872) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:07:54 (1761962874) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:07:55 (1761962875) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:07:58 (1761962878) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:08:01 (1761962881) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:08:03 (1761962883) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.245788 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312121 s, 33.6 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:08:06 (1761962886) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131235 s, 79.9 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:08:09 (1761962889) Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:08:36 (1761962916) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0599057 s, 52.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:08:48 (1761962928) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:09:13 (1761962953) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08223 s, 48.4 MB/s -rw-r--r-- 1 root root 52428800 Oct 31 22:09 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.212s user 0m0.001s sys 0m0.129s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.252s user 0m0.001s sys 0m0.144s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.266s user 0m0.001s sys 0m0.149s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932487daf800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:09:41 (1761962981) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:09:42 (1761962982) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:09:43 (1761962983) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 22:09:44 (1761962984) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 22:09:46 (1761962986) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.1566 s, 80.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:09:50 (1761962990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133814 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126442 s, 82.9 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:09:52 (1761962992) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:09:56 (1761962996) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00062387 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000594751 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000496458 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000440157 s, 2.3 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 22:09:59 (1761962999) create test files with size 15749642 .. 18270abc207b30d5f4f389b3a24c9324 - bcf0af8a5cf7b5310f4e15b7353aa354 - d65c2ebf5d1da087d4c9a0871d01ed6c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 22:10:05 (1761963005) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:10:08 (1761963008) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:10:11 (1761963011) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036685 s, 28.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134885 s, 77.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 22:10:13 (1761963013) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0528267 s, 39.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.141584 s, 29.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0712779 s, 29.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 22:10:17 (1761963017) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 22:10:18 (1761963018) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:10:19 (1761963019) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146946 s, 71.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 22:10:22 (1761963022) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:10:23 (1761963023) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:10:26 (1761963026) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0984822 s, 106 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:10:47 (1761963047) llite.lustre-ffff93248e24c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0783645 s, 134 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9324a0272000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:10:51 (1761963051) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:11:04 (1761963064) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:11:15 (1761963075) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 22:11:16 (1761963076) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0975542 s, 32.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0593859 s, 53.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 5b277a27b4fdb2e7fc118fa5241d26f6 - mirror 1 checksum: bbad21b34c8b718ee531f669ea44f33b - mirror 2 checksum: 5b277a27b4fdb2e7fc118fa5241d26f6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 31 22:11:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 31 22:11:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5b277a27b4fdb2e7fc118fa5241d26f6 - mirror 1 checksum: bbad21b34c8b718ee531f669ea44f33b - mirror 2 checksum: bbad21b34c8b718ee531f669ea44f33b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 31 22:11:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 31 22:11:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:11:20 (1761963080) Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 22:11:24 (1761963084) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 22:11:38 (1761963098) targets are mounted 22:11:38 (1761963098) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104846 s, 100 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:11:44 (1761963104) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:11:45 (1761963105) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:11:46 (1761963106) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00229565 s, 7.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:11:48 (1761963108) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:11:49 (1761963109) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:11:50 (1761963110) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:11:56 (1761963116) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:12:15 (1761963135) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:12:33 (1761963153) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:12:51 (1761963171) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 22:12:53 (1761963173) mirror create pid 51099 mirror split pid 51100 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 51099 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 51100 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 22:13:56 (1761963236) Seed set to 3236 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1761963236.803066 mapwrite 0x036c97 thru 0x03ffff (0x9369 bytes) truncating to largest ever: 0x11fca 000002 1761963236.827672 trunc from 0x040000 to 0x011fc9 (0x2e036 bytes) 000003 1761963236.832092 write 0x036cb9 thru 0x038017 (0x135f bytes) 000004 1761963236.834027 MIRROR_EXTEND to 2 mirrors 000006 1761963236.981772 mapwrite 0x01a536 thru 0x01b72c (0x11f7 bytes) 000007 1761963237.029162 read 0x0010a8 thru 0x00fda6 (0xecff bytes) 000008 1761963237.032826 read 0x01dea1 thru 0x02129c (0x33fc bytes) 000010 1761963237.035582 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) truncating to largest ever: 0x1fd78 000011 1761963237.073133 trunc from 0x038018 to 0x01fd77 (0x182a0 bytes) 000012 1761963237.081183 mapwrite 0x01f609 thru 0x025377 (0x5d6f bytes) 000014 1761963237.104370 MIRROR_EXTEND to 2 mirrors 000015 1761963237.253491 write 0x017ec8 thru 0x021424 (0x955d bytes) 000017 1761963237.273340 trunc from 0x025378 to 0x012d58 (0x1261f bytes) 000023 1761963237.284574 mapwrite 0x01ea6c thru 0x025ec8 (0x745d bytes) 000026 1761963237.305931 MIRROR_EXTEND to 3 mirrors 000027 1761963237.446577 mapread 0x01d13c thru 0x025ec8 (0x8d8d bytes) 000028 1761963237.452322 write 0x02c0e5 thru 0x03340f (0x732b bytes) 000029 1761963237.468655 mapwrite 0x024740 thru 0x0342bf (0xfb80 bytes) 000030 1761963237.493508 mapread 0x02e583 thru 0x02ff1b (0x1999 bytes) 000033 1761963237.494617 write 0x028c98 thru 0x029cec (0x1055 bytes) 000035 1761963237.495503 mapread 0x021363 thru 0x030b6e (0xf80c bytes) 000037 1761963237.496367 read 0x006a53 thru 0x015b01 (0xf0af bytes) 000038 1761963237.499505 mapwrite 0x01d0ae thru 0x01e16d (0x10c0 bytes) 000044 1761963237.512963 read 0x0263c0 thru 0x02846f (0x20b0 bytes) 000045 1761963237.513361 mapread 0x00c8cc thru 0x016449 (0x9b7e bytes) 000049 1761963237.518997 MIRROR_EXTEND to 4 mirrors 000050 1761963237.658180 mapread 0x01f6be thru 0x02e086 (0xe9c9 bytes) 000051 1761963237.664400 read 0x02c801 thru 0x030461 (0x3c61 bytes) 000056 1761963237.665637 mapwrite 0x0265bb thru 0x030ad3 (0xa519 bytes) 000057 1761963237.713525 write 0x03bafe thru 0x03ffff (0x4502 bytes) 000059 1761963237.714874 write 0x029094 thru 0x032a94 (0x9a01 bytes) 000060 1761963237.715793 mapread 0x0273bc thru 0x0337ba (0xc3ff bytes) 000062 1761963237.718573 MIRROR_EXTEND to 5 mirrors 000063 1761963237.867452 mapwrite 0x03dd90 thru 0x03f67b (0x18ec bytes) 000064 1761963237.924407 read 0x017de8 thru 0x021523 (0x973c bytes) truncating to largest ever: 0x2f5e2 000067 1761963237.928404 trunc from 0x040000 to 0x02f5e1 (0x10a1e bytes) 000068 1761963237.939726 read 0x009f93 thru 0x011d29 (0x7d97 bytes) 000069 1761963237.943546 read 0x00430a thru 0x00a27e (0x5f75 bytes) 000070 1761963237.944180 read 0x008524 thru 0x012b12 (0xa5ef bytes) 000075 1761963237.945864 read 0x023299 thru 0x02be66 (0x8bce bytes) truncating to largest ever: 0x36537 000076 1761963237.946260 trunc from 0x02f5e2 to 0x036536 (0x6f55 bytes) 000077 1761963237.951391 write 0x03d6b1 thru 0x03ffff (0x294f bytes) 000080 1761963237.952590 trunc from 0x040000 to 0x0116ee (0x2e911 bytes) 000083 1761963237.959278 trunc from 0x0116ef to 0x03085b (0x1f16d bytes) 000084 1761963237.964563 trunc from 0x03085c to 0x033523 (0x2cc8 bytes) 000085 1761963237.969219 mapwrite 0x01e0e1 thru 0x0231a7 (0x50c7 bytes) 000086 1761963237.988773 read 0x0331ed thru 0x033523 (0x337 bytes) 000088 1761963237.989800 write 0x004ce8 thru 0x012b86 (0xde9f bytes) 000089 1761963237.991041 trunc from 0x033524 to 0x02efda (0x4549 bytes) 000090 1761963237.997245 read 0x008c8f thru 0x009642 (0x9b4 bytes) 000092 1761963237.998240 trunc from 0x02efdb to 0x004af3 (0x2a4e7 bytes) 000097 1761963238.005055 read 0x003e9f thru 0x004af3 (0xc55 bytes) 000099 1761963238.005713 mapread 0x002fe2 thru 0x004af3 (0x1b12 bytes) 000101 1761963238.006570 mapread 0x0004da thru 0x004af3 (0x461a bytes) 000104 1761963238.007605 write 0x01bdfc thru 0x01eecc (0x30d1 bytes) 000105 1761963238.010252 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000110 1761963238.078986 trunc from 0x01eecd to 0x01bc07 (0x32c5 bytes) 000111 1761963238.083598 read 0x00214d thru 0x00931d (0x71d1 bytes) 000112 1761963238.086565 mapread 0x00becc thru 0x01a399 (0xe4ce bytes) 000120 1761963238.096681 mapwrite 0x029b36 thru 0x02dcdf (0x41aa bytes) 000122 1761963238.119909 write 0x031d28 thru 0x03a0c4 (0x839d bytes) 000125 1761963238.121211 mapread 0x02af59 thru 0x02d234 (0x22dc bytes) 000126 1761963238.121779 trunc from 0x03a0c5 to 0x02d6bc (0xca08 bytes) 000127 1761963238.126313 mapwrite 0x037a35 thru 0x03b6a9 (0x3c75 bytes) 000128 1761963238.150789 MIRROR_RESYNC 4 mirrors 000129 1761963238.376305 read 0x028d92 thru 0x02a667 (0x18d6 bytes) 000130 1761963238.378272 mapwrite 0x0057c3 thru 0x00641d (0xc5b bytes) 000132 1761963238.430265 MIRROR_EXTEND to 5 mirrors 000137 1761963238.601462 mapread 0x01960f thru 0x020d01 (0x76f3 bytes) 000139 1761963238.612397 write 0x028d97 thru 0x037c99 (0xef03 bytes) 000140 1761963238.642478 mapwrite 0x00b47d thru 0x0109ea (0x556e bytes) 000141 1761963238.663301 write 0x01b859 thru 0x027464 (0xbc0c bytes) 000143 1761963238.666946 mapread 0x035c7b thru 0x03b6a9 (0x5a2f bytes) 000147 1761963238.669933 trunc from 0x03b6aa to 0x021a15 (0x19c94 bytes) 000150 1761963238.683705 mapwrite 0x00c062 thru 0x017353 (0xb2f2 bytes) 000151 1761963238.703502 MIRROR_RESYNC 5 mirrors 000154 1761963238.986945 mapwrite 0x03c73d thru 0x03ffff (0x38c3 bytes) 000155 1761963239.034182 mapread 0x00d067 thru 0x0189e7 (0xb981 bytes) 000159 1761963239.039564 write 0x03c974 thru 0x03ffff (0x368c bytes) 000161 1761963239.041707 MIRROR_EXTEND to 6 mirrors 000163 1761963239.213344 mapwrite 0x014cbd thru 0x014e71 (0x1b5 bytes) 000165 1761963239.267245 mapwrite 0x004aff thru 0x0112da (0xc7dc bytes) 000166 1761963239.288913 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000167 1761963239.359781 MIRROR_RESYNC 5 mirrors 000168 1761963239.636430 mapread 0x0279e1 thru 0x02d611 (0x5c31 bytes) 000171 1761963239.642758 read 0x035f33 thru 0x03a93e (0x4a0c bytes) 000173 1761963239.645474 mapread 0x00ee8a thru 0x014f91 (0x6108 bytes) truncating to largest ever: 0x375dc 000174 1761963239.647823 trunc from 0x040000 to 0x0375db (0x8a24 bytes) 000175 1761963239.673584 mapread 0x00a0ee thru 0x00acf1 (0xc04 bytes) 000176 1761963239.677638 read 0x033d24 thru 0x0375db (0x38b8 bytes) 000179 1761963239.682500 read 0x012336 thru 0x01258b (0x256 bytes) 000181 1761963239.683438 mapread 0x02709f thru 0x027839 (0x79b bytes) 000183 1761963239.684419 mapread 0x00495f thru 0x008293 (0x3935 bytes) 000188 1761963239.686398 mapread 0x0128c1 thru 0x021cb0 (0xf3f0 bytes) 000192 1761963239.688440 mapwrite 0x03525a thru 0x036860 (0x1607 bytes) 000196 1761963239.704497 mapread 0x019084 thru 0x01ca3b (0x39b8 bytes) 000199 1761963239.708034 MIRROR_EXTEND to 6 mirrors 000200 1761963239.897050 read 0x01c5f5 thru 0x02bdb4 (0xf7c0 bytes) 000201 1761963239.901679 write 0x000a32 thru 0x0033bd (0x298c bytes) 000204 1761963239.925718 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000205 1761963239.983774 mapread 0x00f6a7 thru 0x01bf51 (0xc8ab bytes) 000206 1761963239.992749 trunc from 0x0375dc to 0x01cd40 (0x1a89b bytes) truncating to largest ever: 0x3826d 000208 1761963240.003549 trunc from 0x01cd41 to 0x03826c (0x1b52c bytes) 000210 1761963240.009037 mapwrite 0x02a518 thru 0x0368a0 (0xc389 bytes) 000211 1761963240.031469 trunc from 0x03826d to 0x003e6d (0x343ff bytes) 000212 1761963240.036268 mapread 0x001af8 thru 0x002d2a (0x1233 bytes) 000214 1761963240.038779 read 0x0032db thru 0x003e6d (0xb93 bytes) 000215 1761963240.039180 mapread 0x003b16 thru 0x003e6d (0x358 bytes) 000217 1761963240.039930 mapwrite 0x0061dc thru 0x00cb9c (0x69c1 bytes) 000221 1761963240.061963 mapwrite 0x015163 thru 0x024acf (0xf96d bytes) 000222 1761963240.089597 read 0x0247d6 thru 0x024acf (0x2fa bytes) 000223 1761963240.090123 read 0x00f02b thru 0x0144b9 (0x548f bytes) 000224 1761963240.093947 mapwrite 0x00ea77 thru 0x00f78e (0xd18 bytes) 000226 1761963240.106163 mapwrite 0x01e802 thru 0x025f3b (0x773a bytes) 000228 1761963240.130833 mapread 0x015912 thru 0x021c74 (0xc363 bytes) 000229 1761963240.133625 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000232 1761963240.191773 mapread 0x010fde thru 0x01a2a0 (0x92c3 bytes) 000235 1761963240.200940 mapread 0x01eb5a thru 0x01ffb3 (0x145a bytes) 000236 1761963240.204738 mapwrite 0x03e57b thru 0x03ffff (0x1a85 bytes) 000237 1761963240.228614 read 0x031e71 thru 0x034446 (0x25d6 bytes) 000238 1761963240.231564 read 0x026157 thru 0x032313 (0xc1bd bytes) 000239 1761963240.236504 read 0x002219 thru 0x00b522 (0x930a bytes) 000240 1761963240.239172 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000241 1761963240.302748 write 0x030787 thru 0x031afe (0x1378 bytes) 000242 1761963240.306114 trunc from 0x040000 to 0x0201ee (0x1fe11 bytes) 000245 1761963240.311765 trunc from 0x0201ef to 0x02b7e1 (0xb5f3 bytes) 000248 1761963240.317538 write 0x008a4a thru 0x0124e6 (0x9a9d bytes) 000251 1761963240.322841 write 0x02c2b8 thru 0x0306ba (0x4403 bytes) 000252 1761963240.323917 trunc from 0x0306bb to 0x007f28 (0x28792 bytes) 000257 1761963240.330970 mapwrite 0x036c67 thru 0x03e12c (0x74c6 bytes) truncating to largest ever: 0x3abc0 000259 1761963240.355027 trunc from 0x03e12d to 0x03abbf (0x356d bytes) 000261 1761963240.360772 mapwrite 0x03d9ff thru 0x03ffff (0x2601 bytes) 000262 1761963240.382060 mapread 0x037799 thru 0x038ccf (0x1537 bytes) 000264 1761963240.383120 read 0x0122e6 thru 0x02205a (0xfd75 bytes) 000265 1761963240.386583 mapwrite 0x001fa5 thru 0x00b53d (0x9599 bytes) 000266 1761963240.406597 trunc from 0x040000 to 0x0063fc (0x39c03 bytes) 000269 1761963240.413381 write 0x035971 thru 0x03f9da (0xa06a bytes) truncating to largest ever: 0x3e914 000270 1761963240.414761 trunc from 0x03f9db to 0x03e913 (0x10c7 bytes) 000275 1761963240.423217 mapwrite 0x02e595 thru 0x03b169 (0xcbd5 bytes) 000279 1761963240.445875 write 0x01341d thru 0x016bb3 (0x3797 bytes) 000280 1761963240.449999 read 0x0001b7 thru 0x00fc9e (0xfae8 bytes) 000284 1761963240.456269 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000285 1761963240.533604 mapwrite 0x018f3b thru 0x026305 (0xd3cb bytes) 000287 1761963240.567565 MIRROR_RESYNC 2 mirrors 000288 1761963240.746765 trunc from 0x03e914 to 0x000453 (0x3e4c0 bytes) 000289 1761963240.773581 read 0x0001a9 thru 0x000453 (0x2ab bytes) 000291 1761963240.776110 mapwrite 0x03e71f thru 0x03ffff (0x18e1 bytes) 000295 1761963240.806310 write 0x013bc1 thru 0x019d6f (0x61af bytes) 000297 1761963240.810753 trunc from 0x040000 to 0x026970 (0x1968f bytes) 000298 1761963240.817352 mapread 0x005e34 thru 0x009f51 (0x411e bytes) 000300 1761963240.822222 trunc from 0x026971 to 0x021707 (0x5269 bytes) 000302 1761963240.829772 mapread 0x020281 thru 0x021707 (0x1487 bytes) 000303 1761963240.834215 trunc from 0x021708 to 0x0018d3 (0x1fe34 bytes) 000304 1761963240.840425 trunc from 0x0018d4 to 0x010b2e (0xf25b bytes) 000307 1761963240.848333 read 0x00c3a1 thru 0x010b2e (0x478e bytes) 000309 1761963240.852492 read 0x009f50 thru 0x010b2e (0x6bdf bytes) 000310 1761963240.853139 read 0x007d4c thru 0x010b2e (0x8de3 bytes) 000313 1761963240.854469 read 0x00de1e thru 0x010b2e (0x2d11 bytes) 000315 1761963240.855159 mapwrite 0x038cbb thru 0x03dc7f (0x4fc5 bytes) 000316 1761963240.878549 mapread 0x03017e thru 0x03ac8c (0xab0f bytes) 000317 1761963240.879447 trunc from 0x03dc80 to 0x002388 (0x3b8f7 bytes) 000318 1761963240.886380 MIRROR_RESYNC 2 mirrors 000320 1761963241.044598 write 0x026c00 thru 0x0360e7 (0xf4e8 bytes) 000321 1761963241.063212 mapread 0x01f9e2 thru 0x0280a5 (0x86c4 bytes) 000323 1761963241.068404 trunc from 0x0360e8 to 0x02d894 (0x8853 bytes) 000324 1761963241.079595 mapwrite 0x030bdc thru 0x03f43b (0xe860 bytes) 000325 1761963241.108008 trunc from 0x03f43c to 0x02c7f3 (0x12c48 bytes) 000326 1761963241.116948 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000327 1761963241.163748 read 0x02c1f1 thru 0x02c7f3 (0x603 bytes) 000329 1761963241.166473 write 0x010e65 thru 0x012d29 (0x1ec5 bytes) 000331 1761963241.170827 trunc from 0x02c7f4 to 0x002884 (0x29f6f bytes) 000332 1761963241.175841 read 0x000c47 thru 0x002884 (0x1c3e bytes) 000336 1761963241.180091 write 0x003d13 thru 0x00d821 (0x9b0f bytes) 000338 1761963241.181588 trunc from 0x00d822 to 0x009d72 (0x3aaf bytes) 000340 1761963241.186764 trunc from 0x009d73 to 0x004382 (0x59f0 bytes) 000344 1761963241.193528 write 0x020ad4 thru 0x0255bf (0x4aec bytes) 000345 1761963241.194672 write 0x008235 thru 0x013686 (0xb452 bytes) 000346 1761963241.198806 mapwrite 0x012e76 thru 0x0224a8 (0xf633 bytes) 000348 1761963241.217297 MIRROR_EXTEND to 2 mirrors 000349 1761963241.375039 trunc from 0x0255c0 to 0x02a4b9 (0x4efa bytes) 000350 1761963241.406214 MIRROR_RESYNC 2 mirrors 000351 1761963241.575536 trunc from 0x02a4ba to 0x03621c (0xbd63 bytes) 000356 1761963241.602061 read 0x00977d thru 0x011a91 (0x8315 bytes) 000357 1761963241.605486 read 0x0030c1 thru 0x00562f (0x256f bytes) 000359 1761963241.609917 mapwrite 0x03d559 thru 0x03ffff (0x2aa7 bytes) 000364 1761963241.635431 write 0x024ae4 thru 0x032c95 (0xe1b2 bytes) 000365 1761963241.636600 mapread 0x0290bd thru 0x02a16a (0x10ae bytes) 000366 1761963241.637344 mapwrite 0x02a0d4 thru 0x02d870 (0x379d bytes) 000367 1761963241.652653 trunc from 0x040000 to 0x0301e2 (0xfe1d bytes) 000369 1761963241.658773 mapread 0x00effc thru 0x01e901 (0xf906 bytes) 000370 1761963241.659717 trunc from 0x0301e3 to 0x030611 (0x42f bytes) 000371 1761963241.666381 read 0x007a46 thru 0x0083ab (0x966 bytes) 000373 1761963241.667418 mapwrite 0x03ede7 thru 0x03ffff (0x1219 bytes) 000376 1761963241.689886 mapwrite 0x016973 thru 0x024022 (0xd6b0 bytes) 000378 1761963241.706201 read 0x01c3c1 thru 0x02839a (0xbfda bytes) 000379 1761963241.706806 read 0x00b6a0 thru 0x00f584 (0x3ee5 bytes) 000384 1761963241.708879 write 0x02a75e thru 0x0376d4 (0xcf77 bytes) 000385 1761963241.709736 read 0x019c0f thru 0x02936b (0xf75d bytes) 000388 1761963241.710953 read 0x02fcbe thru 0x031676 (0x19b9 bytes) 000389 1761963241.711369 trunc from 0x040000 to 0x00d033 (0x32fcc bytes) 000390 1761963241.719032 MIRROR_EXTEND to 3 mirrors 000392 1761963241.911361 mapread 0x0063e4 thru 0x006fd5 (0xbf2 bytes) 000393 1761963241.915350 mapread 0x00a8a0 thru 0x00d033 (0x2794 bytes) 000394 1761963241.922029 trunc from 0x00d034 to 0x01902a (0xbff7 bytes) 000395 1761963241.952489 read 0x00afd6 thru 0x0174b6 (0xc4e1 bytes) 000396 1761963241.957378 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000399 1761963242.018568 read 0x0178e9 thru 0x017baf (0x2c7 bytes) 000400 1761963242.020758 mapread 0x0131aa thru 0x015c3d (0x2a94 bytes) 000401 1761963242.026304 write 0x02ed6a thru 0x035897 (0x6b2e bytes) 000404 1761963242.027770 read 0x01c617 thru 0x01fb21 (0x350b bytes) 000408 1761963242.032192 read 0x00a04e thru 0x01768b (0xd63e bytes) 000409 1761963242.034365 MIRROR_RESYNC 2 mirrors 000412 1761963242.216886 mapread 0x01b09f thru 0x023116 (0x8078 bytes) 000415 1761963242.225432 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000417 1761963242.267680 write 0x01091f thru 0x017cb0 (0x7392 bytes) 000418 1761963242.275305 trunc from 0x035898 to 0x011c7e (0x23c19 bytes) 000421 1761963242.289023 MIRROR_EXTEND to 2 mirrors 000422 1761963242.461633 MIRROR_RESYNC 2 mirrors 000423 1761963242.534040 mapwrite 0x03d251 thru 0x03ffff (0x2daf bytes) 000424 1761963242.573958 mapread 0x024999 thru 0x0290a3 (0x470b bytes) 000425 1761963242.577284 mapread 0x0187a7 thru 0x02245c (0x9cb6 bytes) 000427 1761963242.580863 write 0x008032 thru 0x00fd2d (0x7cfc bytes) 000429 1761963242.585254 write 0x01b820 thru 0x0231b5 (0x7996 bytes) 000432 1761963242.586830 write 0x026b5c thru 0x02e9a0 (0x7e45 bytes) 000433 1761963242.589792 write 0x0134b9 thru 0x01493c (0x1484 bytes) 000434 1761963242.590450 read 0x01ccd5 thru 0x02bd91 (0xf0bd bytes) 000436 1761963242.591651 mapread 0x036cb7 thru 0x03ffff (0x9349 bytes) 000437 1761963242.596554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) truncating to largest ever: 0x3f901 000439 1761963242.636792 trunc from 0x040000 to 0x03f900 (0x6ff bytes) 000442 1761963242.641034 read 0x009a28 thru 0x00c31a (0x28f3 bytes) 000443 1761963242.643126 read 0x00ce62 thru 0x0192c2 (0xc461 bytes) 000447 1761963242.647469 mapread 0x01acfd thru 0x029ab3 (0xedb7 bytes) 000448 1761963242.648081 read 0x009638 thru 0x014539 (0xaf02 bytes) 000449 1761963242.648392 mapread 0x03891a thru 0x03f900 (0x6fe7 bytes) 000451 1761963242.649281 trunc from 0x03f901 to 0x003275 (0x3c68b bytes) 000452 1761963242.653719 read 0x00061c thru 0x003275 (0x2c5a bytes) 000455 1761963242.654559 read 0x000b3b thru 0x003275 (0x273b bytes) 000457 1761963242.655161 trunc from 0x003276 to 0x03b756 (0x384e1 bytes) 000459 1761963242.658806 mapwrite 0x024a51 thru 0x03324b (0xe7fb bytes) 000461 1761963242.679711 trunc from 0x03b757 to 0x00f2db (0x2c47b bytes) 000466 1761963242.684904 read 0x00e01b thru 0x00f2db (0x12c1 bytes) 000469 1761963242.685833 mapread 0x006d51 thru 0x00b5a1 (0x4851 bytes) 000470 1761963242.686343 write 0x03e506 thru 0x03ffff (0x1afa bytes) 000472 1761963242.687410 write 0x01dfdf thru 0x022110 (0x4132 bytes) 000474 1761963242.691518 mapread 0x029a42 thru 0x02a85a (0xe19 bytes) 000475 1761963242.692319 mapwrite 0x017988 thru 0x01b469 (0x3ae2 bytes) 000476 1761963242.706725 write 0x037452 thru 0x03c8b6 (0x5465 bytes) 000477 1761963242.707720 trunc from 0x040000 to 0x003de4 (0x3c21b bytes) 000478 1761963242.713489 trunc from 0x003de5 to 0x00d2d5 (0x94f1 bytes) 000479 1761963242.717236 mapwrite 0x0164f4 thru 0x01cf58 (0x6a65 bytes) 000481 1761963242.749223 mapwrite 0x02d253 thru 0x034be6 (0x7994 bytes) 000483 1761963242.788058 mapread 0x017124 thru 0x025d82 (0xec5f bytes) 000486 1761963242.801779 mapread 0x018424 thru 0x019d32 (0x190f bytes) 000487 1761963242.802489 mapread 0x00139c thru 0x00c025 (0xac8a bytes) 000490 1761963242.815077 mapread 0x0081bc thru 0x016b76 (0xe9bb bytes) 000493 1761963242.827329 read 0x02c27e thru 0x034be6 (0x8969 bytes) 000496 1761963242.830423 write 0x024cd0 thru 0x026159 (0x148a bytes) 000501 1761963242.832128 trunc from 0x034be7 to 0x033a5d (0x1189 bytes) 000504 1761963242.837457 mapread 0x02d074 thru 0x033a5d (0x69ea bytes) 000507 1761963242.838742 read 0x018b48 thru 0x01a124 (0x15dd bytes) 000508 1761963242.839126 trunc from 0x033a5e to 0x019d6b (0x19cf2 bytes) 000509 1761963242.844511 mapwrite 0x00c19c thru 0x01554c (0x93b1 bytes) 000512 1761963242.858921 mapwrite 0x01b263 thru 0x0260e8 (0xae86 bytes) 000513 1761963242.878948 mapwrite 0x01da97 thru 0x020865 (0x2dcf bytes) 000516 1761963242.889804 read 0x0040c2 thru 0x0090fd (0x503c bytes) 000517 1761963242.890305 read 0x00dad1 thru 0x0171f1 (0x9721 bytes) 000518 1761963242.890672 read 0x00dba2 thru 0x0163c3 (0x8822 bytes) 000520 1761963242.891614 write 0x0323f1 thru 0x0393e4 (0x6ff4 bytes) 000523 1761963242.892890 read 0x0356c8 thru 0x038257 (0x2b90 bytes) 000524 1761963242.893220 read 0x03416c thru 0x0393e4 (0x5279 bytes) 000528 1761963242.894576 mapread 0x015c4b thru 0x01aa02 (0x4db8 bytes) 000531 1761963242.895539 trunc from 0x0393e5 to 0x012a34 (0x269b0 bytes) 000533 1761963242.900793 mapwrite 0x03d52d thru 0x03ffff (0x2ad3 bytes) 000535 1761963242.921651 MIRROR_EXTEND to 2 mirrors 000536 1761963243.094704 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000538 1761963243.128403 mapread 0x02b01f thru 0x03507d (0xa05f bytes) 000541 1761963243.134501 mapread 0x0212ad thru 0x02189b (0x5ef bytes) 000543 1761963243.137730 write 0x01356c thru 0x01eecc (0xb961 bytes) 000544 1761963243.145420 write 0x00f7ea thru 0x01004f (0x866 bytes) 000545 1761963243.146298 mapwrite 0x00d148 thru 0x016667 (0x9520 bytes) 000546 1761963243.162061 mapread 0x010767 thru 0x019a57 (0x92f1 bytes) 000548 1761963243.163146 mapread 0x0111b0 thru 0x012cf8 (0x1b49 bytes) 000549 1761963243.163842 trunc from 0x040000 to 0x00169b (0x3e964 bytes) 000550 1761963243.169781 read 0x00118c thru 0x00169b (0x510 bytes) 000551 1761963243.171744 mapread 0x000b2b thru 0x00169b (0xb71 bytes) 000553 1761963243.174396 mapwrite 0x03910b thru 0x03ffff (0x6ef5 bytes) 000556 1761963243.196248 read 0x027f30 thru 0x032661 (0xa732 bytes) 000557 1761963243.202378 MIRROR_EXTEND to 2 mirrors 000560 1761963243.359049 mapread 0x00276d thru 0x0048e7 (0x217b bytes) 000562 1761963243.365492 mapread 0x02c6b0 thru 0x03a577 (0xdec8 bytes) 000563 1761963243.371098 mapwrite 0x012a5b thru 0x0137db (0xd81 bytes) 000564 1761963243.423550 mapwrite 0x00b447 thru 0x00ed32 (0x38ec bytes) 000565 1761963243.437119 mapwrite 0x027710 thru 0x029779 (0x206a bytes) 000568 1761963243.455803 write 0x01b8a9 thru 0x02463f (0x8d97 bytes) 000569 1761963243.460273 MIRROR_RESYNC 2 mirrors 000571 1761963243.642305 write 0x016805 thru 0x02538e (0xeb8a bytes) 000572 1761963243.666396 mapwrite 0x0389e3 thru 0x03e62b (0x5c49 bytes) 000574 1761963243.685330 trunc from 0x040000 to 0x01ff0d (0x200f2 bytes) 000575 1761963243.697466 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000578 1761963243.738523 read 0x018fc2 thru 0x01ff0d (0x6f4c bytes) 000579 1761963243.742046 mapwrite 0x0012af thru 0x002b32 (0x1884 bytes) 000581 1761963243.758899 mapread 0x0161d9 thru 0x01ff0d (0x9d35 bytes) 000583 1761963243.764059 trunc from 0x01ff0e to 0x03118b (0x1127e bytes) 000585 1761963243.770282 trunc from 0x03118c to 0x02c999 (0x47f2 bytes) 000586 1761963243.776508 read 0x01d05c thru 0x027083 (0xa028 bytes) 000588 1761963243.781740 write 0x00cd3d thru 0x010caa (0x3f6e bytes) 000590 1761963243.783053 read 0x00932a thru 0x00edc2 (0x5a99 bytes) 000592 1761963243.784022 write 0x02f1f3 thru 0x02fc84 (0xa92 bytes) 000596 1761963243.786114 mapwrite 0x00345c thru 0x00d41a (0x9fbf bytes) 000598 1761963243.805998 mapread 0x010e54 thru 0x018571 (0x771e bytes) 000603 1761963243.808725 trunc from 0x02fc85 to 0x023633 (0xc651 bytes) 000604 1761963243.814836 read 0x019551 thru 0x023633 (0xa0e3 bytes) 000607 1761963243.816254 write 0x0112de thru 0x01164e (0x371 bytes) 000608 1761963243.816953 write 0x03ae68 thru 0x03ffff (0x5198 bytes) 000609 1761963243.817879 write 0x026620 thru 0x02a6fa (0x40db bytes) 000612 1761963243.822397 mapwrite 0x011eb4 thru 0x016454 (0x45a1 bytes) 000613 1761963243.836904 mapwrite 0x0271d2 thru 0x02b414 (0x4243 bytes) 000615 1761963243.850548 read 0x023de2 thru 0x024b27 (0xd46 bytes) 000618 1761963243.852126 write 0x0102cf thru 0x0149d6 (0x4708 bytes) 000619 1761963243.852753 write 0x0324f5 thru 0x03e592 (0xc09e bytes) 000620 1761963243.853527 trunc from 0x040000 to 0x0328cb (0xd734 bytes) 000622 1761963243.859060 read 0x01bf8d thru 0x01d4a8 (0x151c bytes) 000623 1761963243.859569 trunc from 0x0328cc to 0x025ea0 (0xca2b bytes) 000626 1761963243.864676 read 0x004a89 thru 0x0088e2 (0x3e5a bytes) 000627 1761963243.865201 trunc from 0x025ea1 to 0x016701 (0xf79f bytes) 000628 1761963243.870116 read 0x007b59 thru 0x01288c (0xad34 bytes) 000629 1761963243.870667 mapwrite 0x009191 thru 0x00c48e (0x32fe bytes) 000630 1761963243.884228 read 0x00c2ef thru 0x016701 (0xa413 bytes) 000631 1761963243.884603 mapwrite 0x01d393 thru 0x02b2eb (0xdf59 bytes) 000632 1761963243.905713 trunc from 0x02b2ec to 0x038df6 (0xdb0b bytes) 000634 1761963243.910918 write 0x002c63 thru 0x008925 (0x5cc3 bytes) 000637 1761963243.912734 write 0x03906c thru 0x03aa9d (0x1a32 bytes) 000639 1761963243.913916 mapread 0x02c791 thru 0x035470 (0x8ce0 bytes) 000640 1761963243.917527 trunc from 0x03aa9e to 0x02c8da (0xe1c3 bytes) 000641 1761963243.922665 write 0x03b549 thru 0x03ffff (0x4ab7 bytes) 000642 1761963243.923408 trunc from 0x040000 to 0x022c6f (0x1d390 bytes) 000643 1761963243.929023 mapwrite 0x00b37e thru 0x017914 (0xc597 bytes) 000644 1761963243.954803 trunc from 0x022c70 to 0x0006c5 (0x225aa bytes) 000645 1761963243.960593 write 0x01fd87 thru 0x028a51 (0x8ccb bytes) 000651 1761963243.963816 read 0x00d48f thru 0x010d8f (0x3901 bytes) 000654 1761963243.968172 mapread 0x000fe5 thru 0x00c555 (0xb571 bytes) 000655 1761963243.968770 mapwrite 0x00f94b thru 0x019fd9 (0xa68f bytes) 000656 1761963243.985157 write 0x029250 thru 0x031286 (0x8037 bytes) 000659 1761963243.987007 mapread 0x025c1d thru 0x031286 (0xb66a bytes) 000662 1761963243.988837 read 0x00aaf3 thru 0x011e62 (0x7370 bytes) 000664 1761963243.989735 mapread 0x028c8d thru 0x02e3c6 (0x573a bytes) 000665 1761963243.990433 mapread 0x00c663 thru 0x019562 (0xcf00 bytes) 000667 1761963243.991558 mapread 0x017dc3 thru 0x027d92 (0xffd0 bytes) 000671 1761963243.993795 write 0x02669b thru 0x029213 (0x2b79 bytes) 000674 1761963243.995424 mapwrite 0x00bd5d thru 0x010e92 (0x5136 bytes) 000677 1761963244.012121 write 0x0002d7 thru 0x006c52 (0x697c bytes) 000679 1761963244.013531 write 0x00934a thru 0x00f5d3 (0x628a bytes) 000681 1761963244.014628 trunc from 0x031287 to 0x0009c4 (0x308c2 bytes) 000682 1761963244.019908 mapwrite 0x009711 thru 0x016df6 (0xd6e6 bytes) 000683 1761963244.042911 write 0x01b6e3 thru 0x020ba5 (0x54c3 bytes) 000686 1761963244.044168 write 0x027dd5 thru 0x036c7a (0xeea6 bytes) 000687 1761963244.045197 trunc from 0x036c7b to 0x0380a3 (0x1429 bytes) 000689 1761963244.050329 mapread 0x00a5e9 thru 0x00c7e0 (0x21f8 bytes) 000690 1761963244.050888 read 0x01ca50 thru 0x02b524 (0xead5 bytes) 000691 1761963244.053715 trunc from 0x0380a4 to 0x000cb3 (0x373f0 bytes) 000692 1761963244.058577 mapwrite 0x01be5f thru 0x01e486 (0x2628 bytes) 000693 1761963244.082203 mapwrite 0x0106b4 thru 0x01a91b (0xa268 bytes) 000696 1761963244.115588 write 0x00f997 thru 0x0155e1 (0x5c4b bytes) 000698 1761963244.118629 write 0x03d989 thru 0x03ffff (0x2677 bytes) 000701 1761963244.120340 mapwrite 0x031016 thru 0x03c763 (0xb74e bytes) 000703 1761963244.154953 MIRROR_EXTEND to 2 mirrors 000704 1761963244.307738 trunc from 0x040000 to 0x01d206 (0x22df9 bytes) 000707 1761963244.333827 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000708 1761963244.369065 write 0x01e921 thru 0x02154d (0x2c2d bytes) 000709 1761963244.370245 trunc from 0x02154e to 0x00b750 (0x15dfd bytes) 000710 1761963244.374804 write 0x034c6d thru 0x03a19f (0x5533 bytes) 000711 1761963244.375666 mapwrite 0x0370aa thru 0x03ffff (0x8f56 bytes) 000712 1761963244.397523 trunc from 0x040000 to 0x03627f (0x9d80 bytes) 000713 1761963244.401829 mapread 0x000951 thru 0x004e0c (0x44bc bytes) 000715 1761963244.406768 MIRROR_EXTEND to 2 mirrors 000716 1761963244.562386 MIRROR_EXTEND to 3 mirrors 000717 1761963244.722200 write 0x030d23 thru 0x033531 (0x280f bytes) 000718 1761963244.743037 mapread 0x021027 thru 0x02fe4c (0xee26 bytes) 000723 1761963244.750654 trunc from 0x036280 to 0x004a80 (0x317ff bytes) 000725 1761963244.763947 write 0x01d141 thru 0x02b678 (0xe538 bytes) 000726 1761963244.765125 mapwrite 0x037b18 thru 0x03a294 (0x277d bytes) 000727 1761963244.789149 read 0x010eab thru 0x01924d (0x83a3 bytes) 000728 1761963244.792294 read 0x007c24 thru 0x011d87 (0xa164 bytes) 000730 1761963244.796166 mapwrite 0x02ea6a thru 0x0321ff (0x3796 bytes) 000731 1761963244.810447 mapwrite 0x01ed07 thru 0x020244 (0x153e bytes) 000732 1761963244.821595 read 0x01e554 thru 0x02bd16 (0xd7c3 bytes) 000733 1761963244.821998 trunc from 0x03a295 to 0x033f5b (0x6339 bytes) 000735 1761963244.826945 write 0x037455 thru 0x03ffff (0x8bab bytes) 000737 1761963244.828441 mapread 0x02c15c thru 0x0325c9 (0x646e bytes) 000738 1761963244.829247 mapwrite 0x0014ef thru 0x00c8c1 (0xb3d3 bytes) 000740 1761963244.844685 read 0x003565 thru 0x00eb00 (0xb59c bytes) 000741 1761963244.845065 write 0x002878 thru 0x00a988 (0x8111 bytes) 000742 1761963244.845690 trunc from 0x040000 to 0x033c2c (0xc3d3 bytes) 000743 1761963244.852360 MIRROR_EXTEND to 4 mirrors 000746 1761963244.999419 mapread 0x016ec5 thru 0x01d761 (0x689d bytes) 000751 1761963245.005551 mapread 0x025367 thru 0x025723 (0x3bd bytes) 000753 1761963245.009185 mapread 0x0010c1 thru 0x00c19f (0xb0df bytes) 000754 1761963245.012199 trunc from 0x033c2d to 0x02705e (0xcbce bytes) 000755 1761963245.038840 mapread 0x0163a2 thru 0x025b0e (0xf76d bytes) 000757 1761963245.046698 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000758 1761963245.105241 MIRROR_RESYNC 3 mirrors 000763 1761963245.311564 MIRROR_EXTEND to 4 mirrors 000764 1761963245.497634 trunc from 0x02705f to 0x0061fb (0x20e63 bytes) 000766 1761963245.523588 mapread 0x0042fe thru 0x0061fb (0x1efe bytes) 000769 1761963245.529827 write 0x014592 thru 0x017098 (0x2b07 bytes) 000772 1761963245.531141 mapread 0x014ea1 thru 0x017098 (0x21f8 bytes) 000773 1761963245.531783 mapread 0x006577 thru 0x00bb2f (0x55b9 bytes) 000774 1761963245.535264 mapread 0x011218 thru 0x017098 (0x5e81 bytes) 000777 1761963245.538012 write 0x0373dd thru 0x03ffff (0x8c23 bytes) 000782 1761963245.540085 read 0x035de6 thru 0x03de66 (0x8081 bytes) 000788 1761963245.544616 write 0x01a05c thru 0x028f1c (0xeec1 bytes) 000791 1761963245.551395 trunc from 0x040000 to 0x01b8b3 (0x2474c bytes) 000792 1761963245.560061 MIRROR_EXTEND to 5 mirrors 000794 1761963245.748468 trunc from 0x01b8b4 to 0x0108d2 (0xafe1 bytes) 000795 1761963245.776520 trunc from 0x0108d3 to 0x038639 (0x27d67 bytes) 000797 1761963245.784972 trunc from 0x03863a to 0x02c365 (0xc2d4 bytes) 000798 1761963245.792871 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000800 1761963245.854464 trunc from 0x02c366 to 0x03ae9c (0xeb37 bytes) 000801 1761963245.859026 write 0x02b988 thru 0x038901 (0xcf7a bytes) 000802 1761963245.864049 MIRROR_EXTEND to 5 mirrors 000803 1761963246.039787 write 0x00e764 thru 0x0114c3 (0x2d60 bytes) 000804 1761963246.063093 trunc from 0x03ae9d to 0x03708a (0x3e12 bytes) 000805 1761963246.074302 write 0x03e8ea thru 0x03ffff (0x1716 bytes) 000806 1761963246.076367 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000807 1761963246.138456 mapwrite 0x03d77b thru 0x03e727 (0xfad bytes) 000808 1761963246.161806 MIRROR_RESYNC 4 mirrors 000809 1761963246.385267 trunc from 0x040000 to 0x033624 (0xc9db bytes) 000810 1761963246.410156 trunc from 0x033625 to 0x02dbbc (0x5a68 bytes) 000811 1761963246.416133 write 0x01c881 thru 0x021b9a (0x531a bytes) 000813 1761963246.420021 trunc from 0x02dbbd to 0x03c755 (0xeb99 bytes) 000814 1761963246.424528 mapwrite 0x01bfa5 thru 0x01d1b0 (0x120c bytes) 000816 1761963246.439575 mapwrite 0x03f0c4 thru 0x03ffff (0xf3c bytes) 000817 1761963246.464239 MIRROR_RESYNC 4 mirrors 000820 1761963246.664577 MIRROR_RESYNC 4 mirrors 000821 1761963246.748597 write 0x0246e9 thru 0x02c737 (0x804f bytes) 000822 1761963246.773431 read 0x0096cd thru 0x00dfa4 (0x48d8 bytes) 000823 1761963246.776179 mapwrite 0x01efe0 thru 0x02c4e8 (0xd509 bytes) 000824 1761963246.797285 mapwrite 0x018be0 thru 0x019e85 (0x12a6 bytes) 000828 1761963246.819380 MIRROR_EXTEND to 5 mirrors 000830 1761963246.992119 mapread 0x021501 thru 0x02654f (0x504f bytes) 000832 1761963247.000724 write 0x03b887 thru 0x03ffff (0x4779 bytes) 000834 1761963247.021826 write 0x030ae7 thru 0x03e533 (0xda4d bytes) 000835 1761963247.025242 mapread 0x00470b thru 0x0074ea (0x2de0 bytes) 000836 1761963247.027716 mapread 0x01c69a thru 0x028d55 (0xc6bc bytes) 000837 1761963247.031253 read 0x038cf9 thru 0x03ffff (0x7307 bytes) 000838 1761963247.031587 write 0x002ea5 thru 0x00455f (0x16bb bytes) 000842 1761963247.032956 read 0x0176c0 thru 0x025c69 (0xe5aa bytes) 000843 1761963247.033303 trunc from 0x040000 to 0x0315a2 (0xea5d bytes) 000845 1761963247.043460 MIRROR_EXTEND to 6 mirrors 000846 1761963247.246837 trunc from 0x0315a3 to 0x0224e9 (0xf0b9 bytes) 000849 1761963247.271872 mapread 0x00d013 thru 0x00d7ef (0x7dd bytes) 000850 1761963247.274182 mapwrite 0x037fad thru 0x03ffff (0x8053 bytes) 000852 1761963247.313358 mapread 0x029533 thru 0x02ed1a (0x57e8 bytes) 000854 1761963247.317720 read 0x00b835 thru 0x014ada (0x92a6 bytes) 000855 1761963247.320962 mapread 0x033c6f thru 0x03ea52 (0xade4 bytes) 000857 1761963247.322279 trunc from 0x040000 to 0x02bad7 (0x14528 bytes) 000859 1761963247.328553 mapread 0x00de85 thru 0x01d3ba (0xf536 bytes) 000860 1761963247.332207 mapread 0x0289aa thru 0x02bad7 (0x312e bytes) 000861 1761963247.334975 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000862 1761963247.392932 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000863 1761963247.460926 trunc from 0x02bad8 to 0x01f84d (0xc28a bytes) 000865 1761963247.470781 MIRROR_EXTEND to 5 mirrors 000866 1761963247.638510 mapwrite 0x01fe58 thru 0x023a39 (0x3be2 bytes) 000868 1761963247.688514 trunc from 0x023a3a to 0x0315d5 (0xdb9c bytes) 000869 1761963247.693979 mapwrite 0x01f741 thru 0x026ef4 (0x77b4 bytes) 000870 1761963247.709461 trunc from 0x0315d6 to 0x037b36 (0x6561 bytes) 000873 1761963247.714838 write 0x017d36 thru 0x0267ae (0xea79 bytes) 000874 1761963247.718361 mapread 0x01f1e2 thru 0x02cb73 (0xd992 bytes) 000877 1761963247.720163 mapread 0x01fb31 thru 0x027564 (0x7a34 bytes) 000878 1761963247.720940 mapwrite 0x0377eb thru 0x03ffff (0x8815 bytes) 000880 1761963247.742036 trunc from 0x040000 to 0x0259b9 (0x1a646 bytes) 000882 1761963247.746695 write 0x008f6a thru 0x015cd3 (0xcd6a bytes) 000883 1761963247.749309 mapwrite 0x033379 thru 0x0348e6 (0x156e bytes) 000884 1761963247.772902 mapread 0x019bd9 thru 0x0204b2 (0x68da bytes) 000885 1761963247.773698 trunc from 0x0348e7 to 0x004bea (0x2fcfc bytes) 000887 1761963247.779004 write 0x02483a thru 0x025037 (0x7fe bytes) 000888 1761963247.779592 trunc from 0x025038 to 0x010d29 (0x1430e bytes) 000890 1761963247.784688 mapread 0x00a0ec thru 0x00eb9e (0x4ab3 bytes) 000893 1761963247.788988 trunc from 0x010d2a to 0x0236e7 (0x129be bytes) 000896 1761963247.794778 trunc from 0x0236e8 to 0x030073 (0xc98c bytes) 000897 1761963247.799215 mapread 0x01c4d5 thru 0x024599 (0x80c5 bytes) 000898 1761963247.804114 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000902 1761963247.865257 read 0x013e70 thru 0x017e20 (0x3fb1 bytes) 000904 1761963247.868408 trunc from 0x030074 to 0x00782d (0x28846 bytes) 000905 1761963247.873014 mapwrite 0x03756d thru 0x037b75 (0x609 bytes) 000906 1761963247.894159 trunc from 0x037b76 to 0x03f3a6 (0x7831 bytes) 000908 1761963247.898535 mapwrite 0x001a41 thru 0x00a303 (0x88c3 bytes) 000909 1761963247.915011 trunc from 0x03f3a7 to 0x02d729 (0x11c7d bytes) 000910 1761963247.919208 trunc from 0x02d72a to 0x036aa6 (0x937d bytes) 000911 1761963247.922984 trunc from 0x036aa7 to 0x01d53b (0x1956b bytes) 000912 1761963247.927252 mapread 0x00c056 thru 0x015eff (0x9eaa bytes) 000914 1761963247.930572 read 0x017688 thru 0x01a8a4 (0x321d bytes) 000915 1761963247.933438 trunc from 0x01d53c to 0x02dfd6 (0x10a9b bytes) 000917 1761963247.939191 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000918 1761963247.997615 write 0x0152f4 thru 0x01ce99 (0x7ba6 bytes) 000923 1761963248.002337 mapwrite 0x007a59 thru 0x0092bb (0x1863 bytes) 000925 1761963248.017239 mapread 0x00cb68 thru 0x01552d (0x89c6 bytes) 000926 1761963248.019485 mapread 0x026b0c thru 0x02dfd6 (0x74cb bytes) 000927 1761963248.022156 mapread 0x00fda1 thru 0x018807 (0x8a67 bytes) 000929 1761963248.022905 read 0x01d33b thru 0x025eff (0x8bc5 bytes) 000932 1761963248.025157 mapread 0x006b04 thru 0x00bbb2 (0x50af bytes) 000935 1761963248.026805 mapwrite 0x007811 thru 0x00f235 (0x7a25 bytes) 000937 1761963248.039279 trunc from 0x02dfd7 to 0x03500e (0x7038 bytes) 000938 1761963248.044061 trunc from 0x03500f to 0x02f50d (0x5b01 bytes) 000940 1761963248.048286 mapwrite 0x00a9a8 thru 0x010aa3 (0x60fc bytes) 000943 1761963248.062284 write 0x004da6 thru 0x00ba84 (0x6cdf bytes) 000944 1761963248.062933 trunc from 0x02f50e to 0x038306 (0x8df9 bytes) 000945 1761963248.067816 trunc from 0x038307 to 0x029047 (0xf2bf bytes) 000946 1761963248.072328 trunc from 0x029048 to 0x03f339 (0x162f2 bytes) 000948 1761963248.077833 read 0x015c10 thru 0x019167 (0x3558 bytes) 000949 1761963248.078389 mapwrite 0x031145 thru 0x03bcc8 (0xab84 bytes) 000953 1761963248.097899 write 0x00d86a thru 0x0169c2 (0x9159 bytes) 000954 1761963248.100285 MIRROR_RESYNC 3 mirrors 000955 1761963248.301282 mapread 0x015ce1 thru 0x0222ea (0xc60a bytes) 000956 1761963248.308370 mapread 0x0006bc thru 0x00a067 (0x99ac bytes) 000959 1761963248.312751 MIRROR_RESYNC 3 mirrors 000960 1761963248.388346 MIRROR_RESYNC 3 mirrors 000961 1761963248.461817 MIRROR_RESYNC 3 mirrors 000962 1761963248.535051 read 0x03b6a6 thru 0x03f339 (0x3c94 bytes) 000964 1761963248.538973 MIRROR_RESYNC 3 mirrors 000967 1761963248.607552 mapread 0x0281fa thru 0x02ae31 (0x2c38 bytes) 000968 1761963248.611709 trunc from 0x03f33a to 0x034bef (0xa74a bytes) 000969 1761963248.637472 trunc from 0x034bf0 to 0x02792a (0xd2c5 bytes) 000972 1761963248.644426 read 0x02336d thru 0x02792a (0x45be bytes) 000973 1761963248.647193 trunc from 0x02792b to 0x008df0 (0x1eb3a bytes) 000974 1761963248.652220 mapread 0x007cc6 thru 0x008df0 (0x112b bytes) 000975 1761963248.655960 write 0x01f559 thru 0x02d300 (0xdda8 bytes) 000978 1761963248.658357 mapwrite 0x019ea1 thru 0x0282c7 (0xe427 bytes) 000979 1761963248.681203 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x589:0x0]: Invalid argument (22) 000980 1761963248.743925 read 0x029dbc thru 0x02b19e (0x13e3 bytes) 000981 1761963248.746270 write 0x02996d thru 0x032c1a (0x92ae bytes) 000984 1761963248.748398 mapread 0x005f83 thru 0x00a6b0 (0x472e bytes) 000985 1761963248.754265 mapread 0x02e965 thru 0x032c1a (0x42b6 bytes) 000986 1761963248.754991 mapwrite 0x02875b thru 0x02a255 (0x1afb bytes) 000987 1761963248.770014 trunc from 0x032c1b to 0x01778f (0x1b48b bytes) 000989 1761963248.775166 trunc from 0x017790 to 0x03a3cb (0x22c3c bytes) 000990 1761963248.780520 trunc from 0x03a3cc to 0x02de22 (0xc5a9 bytes) 000992 1761963248.786874 trunc from 0x02de23 to 0x0264ce (0x7954 bytes) 000993 1761963248.792433 read 0x00d8d9 thru 0x01c208 (0xe930 bytes) 000994 1761963248.796119 read 0x00f5fd thru 0x017116 (0x7b1a bytes) 000996 1761963248.796751 mapwrite 0x00011e thru 0x006392 (0x6275 bytes) 000997 1761963248.814792 MIRROR_EXTEND to 3 mirrors 000999 1761963248.980597 read 0x01c110 thru 0x0264ce (0xa3bf bytes) All operations completed A-OK! PASS 70a (14s) == sanity-flr test 71: check mirror extend parent fid ==== 22:14:10 (1761963250) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 22:14:11 (1761963251) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7307456 .. Extending file size to 6014304 .. Extending file size to 6178368 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7727072 .. Extending file size to 8215392 .. Extending file size to 7782784 .. done Extending file size to 8585984 .. Extending file size to 8430944 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8641120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8712896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9071008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8439968 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8619872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8817568 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8943360 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 56140 56141 56143 56145 56147 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:15:16 (1761963316) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3205312 .. Extending file size to 1676320 .. done Extending file size to 8736832 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6896000 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8590944 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9244128 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9010848 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9357984 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 59755 59756 59758 59760 59762 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:16:21 (1761963381) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4864 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4864 2203392 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0396149 s, 52.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.094606 s, 22.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:16:27 (1761963387) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0248976 s, 84.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 22:16:29 (1761963389) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x7e9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:16:31 (1761963391) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0523805 s, 40.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:16:34 (1761963394) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.083618 s, 25.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ed:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:16:36 (1761963396) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0517625 s, 40.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ef:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:17:01 (1761963421) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242776 s, 43.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ed:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:17:42 (1761963462) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0441683 s, 47.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:18:24 (1761963504) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. Creating new pool test_204f oleg337-server: Pool lustre.test_204f created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0498733 s, 42.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg337-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:19:20 (1761963560) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:19:22 (1761963562) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:19:25 (1761963565) oleg337-server: Pool lustre.test_206 created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg337-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:19:37 (1761963577) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284578 s, 36.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104477 s, 100 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x18e465f Mirror 3: 0x18e465f PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:19:40 (1761963580) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:19:41 (1761963581) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:19:42 (1761963582) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0198355 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xcd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1130000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1130000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1130000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ab0000 Read this (4 bytes): BBBB 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:19:58 (1761963598) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.003679 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:20:00 (1761963600) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013766 s, 76.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5ea lmm_fid: [0x200000404:0x5ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5ea lmm_fid: [0x200000404:0x5ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7fd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835434 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-31 22:20:01.000000000 -0400 Modify: 2025-10-31 22:20:01.000000000 -0400 Change: 2025-10-31 22:20:01.000000000 -0400 Birth: 2025-10-31 22:20:01.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:20:03 (1761963603) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:20:04 (1761963604) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.113822 s, 360 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0156146 s, 262 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 31 22:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5ec:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 31 22:20 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 777 sec ============ 22:20:06 (1761963606) === sanity-flr: start cleanup 22:20:07 (1761963607) === === sanity-flr: finish cleanup 22:20:08 (1761963608) ===