-----============= acceptance-small: sanity-flr ============----- Wed Sep 24 14:05:57 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg150-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:06:08 (1758737168) === oleg150-client.virtnet: executing check_config_client /mnt/lustre oleg150-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg150-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d8847d58000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d8847d58000.idle_timeout=debug disable quota as required oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:06:23 (1758737183) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7550] == sanity-flr test 0a: lfs mirror create with -N option == 14:06:24 (1758737184) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:06:33 (1758737193) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:06:34 (1758737194) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:06:36 (1758737196) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:06:50 (1758737210) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:06:52 (1758737212) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:06:54 (1758737214) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d8847d58000.stats=clear osc.lustre-OST0001-osc-ffff8d8847d58000.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:06:59 (1758737219) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:07:05 (1758737225) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:07:10 (1758737230) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:07:12 (1758737232) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 14:07:17 (1758737237) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:07:21 (1758737241) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:07:23 (1758737243) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:07:27 (1758737247) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:07:33 (1758737253) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:07:36 (1758737256) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.726377 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771256 s, 13.6 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:07:42 (1758737262) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0597836 s, 70.2 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:07:45 (1758737265) Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:08:24 (1758737304) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.537327 s, 17.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (56s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:09:20 (1758737360) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (55s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:10:15 (1758737415) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59567 s, 32.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 24 14:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.578s user 0m0.000s sys 0m0.144s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.697s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.793s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8847d58000.ost_server_uuid in FULL state after 0 sec PASS 33b (60s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:11:15 (1758737475) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:11:18 (1758737478) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:11:21 (1758737481) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:11:24 (1758737484) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 14:11:30 (1758737490) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.383786 s, 32.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:11:39 (1758737499) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313723 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178981 s, 58.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:11:46 (1758737506) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:11:53 (1758737513) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000786771 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00175759 s, 583 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012372 s, 828 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000734553 s, 1.4 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 14:12:01 (1758737521) create test files with size 15742853 .. 311a7edf40ac4caf94c5309fd703e454 - 5e6b43ca7727ae4668dee5b79bd57e54 - 5c31d442ae037591300cbe594f35e991 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 14:12:21 (1758737541) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:12:34 (1758737554) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:12:40 (1758737560) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875595 s, 12.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297025 s, 35.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 14:12:48 (1758737568) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0841854 s, 24.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.256575 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157079 s, 13.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 14:12:59 (1758737579) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:13:02 (1758737582) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:13:05 (1758737585) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207038 s, 50.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:13:13 (1758737593) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:13:17 (1758737597) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:13:27 (1758737607) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.339366 s, 30.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:13:40 (1758737620) llite.lustre-ffff8d8847d58000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325425 s, 32.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d884844f800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:13:55 (1758737635) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:14:32 (1758737672) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:15:12 (1758737712) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:15:15 (1758737715) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.371157 s, 8.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.277128 s, 11.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 0c6e606b014d6b06c90a8c6247e52d50 - mirror 1 checksum: 05bf4e71b5d9103c5dfc3119da35d9e3 - mirror 2 checksum: 0c6e606b014d6b06c90a8c6247e52d50 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Sep 24 14:15:18 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Sep 24 14:15:18 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0c6e606b014d6b06c90a8c6247e52d50 - mirror 1 checksum: 05bf4e71b5d9103c5dfc3119da35d9e3 - mirror 2 checksum: 05bf4e71b5d9103c5dfc3119da35d9e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Sep 24 14:15:18 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Sep 24 14:15:18 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:15:28 (1758737728) Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 14:15:34 (1758737734) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 14:16:04 (1758737764) targets are mounted 14:16:04 (1758737764) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034114 s, 30.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:16:18 (1758737778) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:16:21 (1758737781) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:16:24 (1758737784) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00656626 s, 2.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:16:31 (1758737791) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:16:34 (1758737794) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:16:37 (1758737797) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:16:57 (1758737817) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:17:24 (1758737844) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:17:47 (1758737867) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:18:11 (1758737891) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:18:19 (1758737899) mirror create pid 52253 mirror split pid 52255 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52253 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52255 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:19:27 (1758737967) Seed set to 7969 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1758737970.189430 mapwrite 0x000661 thru 0x00a405 (0x9da5 bytes) truncating to largest ever: 0x2d085 000003 1758737970.335058 trunc from 0x00a406 to 0x02d084 (0x22c7f bytes) 000004 1758737970.380913 MIRROR_EXTEND to 2 mirrors 000005 1758737971.050996 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000006 1758737971.270953 write 0x012cd3 thru 0x01e87d (0xbbab bytes) 000007 1758737971.289668 read 0x0248bf thru 0x029dd1 (0x5513 bytes) 000008 1758737971.301376 MIRROR_EXTEND to 2 mirrors 000009 1758737971.981838 mapread 0x002d68 thru 0x00de07 (0xb0a0 bytes) 000012 1758737972.028110 write 0x0172df thru 0x01c305 (0x5027 bytes) 000013 1758737972.122320 mapread 0x01e883 thru 0x01f8d1 (0x104f bytes) 000014 1758737972.136269 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000017 1758737972.302986 write 0x000007 thru 0x006da8 (0x6da2 bytes) 000018 1758737972.313107 mapwrite 0x027b04 thru 0x02da07 (0x5f04 bytes) 000019 1758737972.443446 mapwrite 0x036fe2 thru 0x03ffff (0x901e bytes) 000020 1758737972.606774 write 0x03032e thru 0x03dadb (0xd7ae bytes) 000022 1758737972.624566 mapread 0x03f146 thru 0x03ffff (0xeba bytes) 000025 1758737972.638138 MIRROR_EXTEND to 2 mirrors 000026 1758737973.309494 trunc from 0x040000 to 0x017768 (0x28897 bytes) 000027 1758737973.403012 read 0x00f4ae thru 0x010323 (0xe76 bytes) 000030 1758737973.411239 mapread 0x00268f thru 0x00c10b (0x9a7d bytes) truncating to largest ever: 0x349d5 000036 1758737973.430486 trunc from 0x017769 to 0x0349d4 (0x1d26c bytes) truncating to largest ever: 0x37444 000038 1758737973.465629 trunc from 0x0349d5 to 0x037443 (0x2a6f bytes) 000039 1758737973.493445 read 0x001751 thru 0x00d97a (0xc22a bytes) 000041 1758737973.509322 mapwrite 0x0198fc thru 0x01f13d (0x5842 bytes) 000046 1758737973.612954 write 0x034d00 thru 0x03f959 (0xac5a bytes) 000050 1758737973.624873 write 0x007db7 thru 0x00dfdd (0x6227 bytes) 000052 1758737973.644285 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000055 1758737973.845504 MIRROR_EXTEND to 2 mirrors 000056 1758737974.645122 write 0x0058e8 thru 0x00bae8 (0x6201 bytes) 000057 1758737974.776349 mapwrite 0x01e87c thru 0x01fbcb (0x1350 bytes) 000059 1758737974.907465 read 0x01c8ef thru 0x028a4b (0xc15d bytes) 000060 1758737974.924500 write 0x03957e thru 0x03bd66 (0x27e9 bytes) 000061 1758737974.943059 read 0x01aee4 thru 0x02053a (0x5657 bytes) 000062 1758737974.943659 mapwrite 0x0348bd thru 0x03e42a (0x9b6e bytes) 000064 1758737975.012861 read 0x011bff thru 0x019ce1 (0x80e3 bytes) 000065 1758737975.032738 read 0x031f3a thru 0x03f959 (0xda20 bytes) 000066 1758737975.034125 trunc from 0x03f95a to 0x021a55 (0x1df04 bytes) 000067 1758737975.060783 mapwrite 0x015fbf thru 0x01f35a (0x939c bytes) 000072 1758737975.130767 read 0x00c627 thru 0x014422 (0x7dfc bytes) 000074 1758737975.132782 mapread 0x00eb8a thru 0x0179d9 (0x8e50 bytes) 000075 1758737975.134682 write 0x03e423 thru 0x03fbde (0x17bc bytes) 000076 1758737975.135994 mapread 0x01a50a thru 0x01dd42 (0x3839 bytes) 000077 1758737975.137765 mapwrite 0x006f89 thru 0x0070d1 (0x149 bytes) 000078 1758737975.184951 write 0x0199bd thru 0x01f0e0 (0x5724 bytes) 000080 1758737975.186509 read 0x02d4c1 thru 0x035e25 (0x8965 bytes) 000081 1758737975.200329 write 0x030166 thru 0x03eaa8 (0xe943 bytes) 000082 1758737975.202519 write 0x03a1f4 thru 0x03ffff (0x5e0c bytes) 000084 1758737975.211402 MIRROR_RESYNC 2 mirrors 000085 1758737976.104247 write 0x028e5f thru 0x031e6d (0x900f bytes) 000086 1758737976.244997 read 0x00e8bc thru 0x01d83a (0xef7f bytes) 000087 1758737976.262116 MIRROR_RESYNC 2 mirrors 000088 1758737976.803873 trunc from 0x040000 to 0x02853b (0x17ac4 bytes) 000089 1758737976.966383 mapread 0x005f3d thru 0x010e46 (0xaf0a bytes) 000091 1758737977.029837 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000092 1758737977.234331 write 0x025719 thru 0x02ac01 (0x54e9 bytes) 000093 1758737977.246196 mapwrite 0x0230df thru 0x029067 (0x5f89 bytes) 000095 1758737977.318123 read 0x00bfbe thru 0x00c03e (0x81 bytes) 000096 1758737977.333333 mapwrite 0x02ae1c thru 0x03705e (0xc243 bytes) 000098 1758737977.554037 write 0x03ed65 thru 0x03ffff (0x129b bytes) 000101 1758737977.557997 mapwrite 0x016531 thru 0x0238c8 (0xd398 bytes) 000103 1758737977.654004 read 0x010325 thru 0x01edaa (0xea86 bytes) 000106 1758737977.655580 write 0x025c6a thru 0x027dcf (0x2166 bytes) truncating to largest ever: 0x3f6c3 000108 1758737977.656867 trunc from 0x040000 to 0x03f6c2 (0x93d bytes) 000110 1758737977.679908 mapwrite 0x030dfa thru 0x03ffff (0xf206 bytes) 000112 1758737977.756719 read 0x01a895 thru 0x02691f (0xc08b bytes) 000113 1758737977.757395 mapwrite 0x016a5e thru 0x01f9a8 (0x8f4b bytes) 000114 1758737977.799923 write 0x004e04 thru 0x006100 (0x12fd bytes) 000115 1758737977.801253 trunc from 0x040000 to 0x036cd6 (0x9329 bytes) 000116 1758737977.821281 write 0x03fd1f thru 0x03ffff (0x2e1 bytes) 000119 1758737977.823430 mapwrite 0x0023d3 thru 0x00d83f (0xb46d bytes) 000121 1758737977.899425 mapwrite 0x03d297 thru 0x03ffff (0x2d69 bytes) 000123 1758737977.961760 mapread 0x03cf62 thru 0x03ffff (0x309e bytes) 000124 1758737977.965992 trunc from 0x040000 to 0x0064b4 (0x39b4b bytes) 000125 1758737977.993081 mapwrite 0x02582a thru 0x030288 (0xaa5f bytes) 000126 1758737978.147142 read 0x01bf11 thru 0x01d87e (0x196e bytes) 000127 1758737978.162003 mapread 0x00e712 thru 0x0148d7 (0x61c6 bytes) 000128 1758737978.162949 mapread 0x0289d0 thru 0x030288 (0x78b9 bytes) 000130 1758737978.164154 write 0x02010f thru 0x02acb9 (0xabab bytes) 000131 1758737978.169893 read 0x003137 thru 0x003d01 (0xbcb bytes) 000133 1758737978.173073 write 0x01b1d5 thru 0x0230fb (0x7f27 bytes) 000135 1758737978.180478 mapread 0x026b27 thru 0x02e6d4 (0x7bae bytes) 000136 1758737978.188315 trunc from 0x030289 to 0x02b76f (0x4b19 bytes) 000137 1758737978.230132 mapread 0x01c604 thru 0x01c7e6 (0x1e3 bytes) 000138 1758737978.231061 write 0x004362 thru 0x008172 (0x3e11 bytes) 000140 1758737978.232481 write 0x02c3a2 thru 0x037f68 (0xbbc7 bytes) 000143 1758737978.242271 write 0x009d6e thru 0x00e673 (0x4906 bytes) 000147 1758737978.256235 trunc from 0x037f69 to 0x00a24e (0x2dd1a bytes) 000148 1758737978.291878 mapwrite 0x0000e1 thru 0x006063 (0x5f83 bytes) 000149 1758737978.385390 trunc from 0x00a24f to 0x004a01 (0x584d bytes) 000151 1758737978.405355 write 0x009108 thru 0x011013 (0x7f0c bytes) 000153 1758737978.407852 trunc from 0x011014 to 0x025147 (0x14134 bytes) 000155 1758737978.430273 write 0x00fff7 thru 0x01ba8c (0xba96 bytes) 000156 1758737978.437982 read 0x01f2b0 thru 0x0236a3 (0x43f4 bytes) 000157 1758737978.448787 mapread 0x004251 thru 0x00d0a9 (0x8e59 bytes) 000159 1758737978.460459 MIRROR_EXTEND to 2 mirrors 000161 1758737979.122251 write 0x011320 thru 0x020e97 (0xfb78 bytes) 000162 1758737979.204478 read 0x0098c6 thru 0x009ab5 (0x1f0 bytes) 000163 1758737979.211748 mapwrite 0x0272ec thru 0x0365ca (0xf2df bytes) 000164 1758737979.350690 trunc from 0x0365cb to 0x0209aa (0x15c20 bytes) 000165 1758737979.369894 write 0x02aebb thru 0x035a8a (0xabd0 bytes) 000166 1758737979.371319 write 0x00549c thru 0x00e852 (0x93b7 bytes) 000167 1758737979.378857 mapread 0x02538d thru 0x032c52 (0xd8c6 bytes) 000169 1758737979.392049 mapread 0x02a446 thru 0x02ce70 (0x2a2b bytes) 000170 1758737979.392956 read 0x00ff7b thru 0x01f820 (0xf8a6 bytes) 000171 1758737979.393583 mapread 0x02bb11 thru 0x03464d (0x8b3d bytes) 000172 1758737979.394915 mapwrite 0x0097ad thru 0x00f5bb (0x5e0f bytes) 000173 1758737979.467585 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000174 1758737979.634648 write 0x01c91a thru 0x0247ec (0x7ed3 bytes) 000175 1758737979.658797 mapwrite 0x01ba87 thru 0x01ffdb (0x4555 bytes) 000177 1758737979.723863 MIRROR_EXTEND to 2 mirrors 000179 1758737980.456277 mapread 0x024f2b thru 0x034cc6 (0xfd9c bytes) 000181 1758737980.534388 MIRROR_EXTEND to 3 mirrors 000185 1758737981.308863 read 0x02f0b1 thru 0x032a99 (0x39e9 bytes) 000187 1758737981.362489 mapwrite 0x037e93 thru 0x03a39c (0x250a bytes) 000189 1758737981.576975 mapwrite 0x034df7 thru 0x03ffff (0xb209 bytes) 000192 1758737981.715772 write 0x025221 thru 0x02fbd0 (0xa9b0 bytes) 000193 1758737981.736966 mapread 0x0152e1 thru 0x0221ba (0xceda bytes) 000194 1758737981.751930 MIRROR_EXTEND to 4 mirrors 000196 1758737982.327997 write 0x031635 thru 0x0346e6 (0x30b2 bytes) 000198 1758737982.408973 mapread 0x0275de thru 0x036dc6 (0xf7e9 bytes) 000199 1758737982.426899 trunc from 0x040000 to 0x019bc1 (0x2643e bytes) 000200 1758737982.446759 mapread 0x0046a6 thru 0x00a64a (0x5fa5 bytes) 000201 1758737982.455037 mapwrite 0x01ca72 thru 0x0297a4 (0xcd33 bytes) 000202 1758737982.581616 MIRROR_RESYNC 4 mirrors 000203 1758737983.452275 MIRROR_EXTEND to 5 mirrors 000204 1758737984.103795 mapread 0x027485 thru 0x0297a4 (0x2320 bytes) 000205 1758737984.147647 read 0x011c6f thru 0x01dffb (0xc38d bytes) 000206 1758737984.161877 mapread 0x021120 thru 0x0297a4 (0x8685 bytes) 000207 1758737984.167928 trunc from 0x0297a5 to 0x006009 (0x2379b bytes) 000210 1758737984.278124 mapwrite 0x03a63c thru 0x03d113 (0x2ad8 bytes) 000211 1758737984.417022 trunc from 0x03d114 to 0x009884 (0x3388f bytes) 000213 1758737984.438458 trunc from 0x009885 to 0x02a642 (0x20dbe bytes) 000214 1758737984.486221 write 0x011d8e thru 0x01b794 (0x9a07 bytes) 000215 1758737984.524989 MIRROR_EXTEND to 6 mirrors 000216 1758737985.286378 mapread 0x01ffb5 thru 0x021629 (0x1675 bytes) 000221 1758737985.369961 mapread 0x020e92 thru 0x027975 (0x6ae4 bytes) 000222 1758737985.385487 mapwrite 0x0016a4 thru 0x0060ea (0x4a47 bytes) 000224 1758737985.626126 write 0x02a7db thru 0x037057 (0xc87d bytes) 000226 1758737985.638013 mapread 0x00e70e thru 0x01cb2b (0xe41e bytes) 000227 1758737985.663697 trunc from 0x037058 to 0x00e1bd (0x28e9a bytes) 000229 1758737985.692134 MIRROR_EXTEND to 7 mirrors 000230 1758737986.363399 mapread 0x004fd7 thru 0x00933b (0x4365 bytes) 000231 1758737986.429149 mapread 0x005ead thru 0x00e1bd (0x8311 bytes) 000232 1758737986.459035 mapwrite 0x01f258 thru 0x0275ae (0x8357 bytes) 000233 1758737986.663618 mapread 0x01b586 thru 0x026397 (0xae12 bytes) 000235 1758737986.665028 write 0x01c234 thru 0x022e47 (0x6c14 bytes) 000237 1758737986.687971 MIRROR_EXTEND to 8 mirrors 000238 1758737987.471385 read 0x020f70 thru 0x021769 (0x7fa bytes) 000239 1758737987.492872 read 0x00d92f thru 0x01787e (0x9f50 bytes) 000240 1758737987.532816 write 0x030609 thru 0x036743 (0x613b bytes) 000241 1758737987.640120 write 0x00f005 thru 0x0154b0 (0x64ac bytes) 000243 1758737987.653270 trunc from 0x036744 to 0x0375a7 (0xe64 bytes) 000244 1758737987.682662 MIRROR_RESYNC 8 mirrors 000245 1758737989.248671 trunc from 0x0375a8 to 0x031842 (0x5d65 bytes) 000247 1758737989.324695 trunc from 0x031843 to 0x036f5f (0x571d bytes) 000248 1758737989.353155 mapread 0x035e31 thru 0x036f5f (0x112f bytes) 000251 1758737989.366954 write 0x006edc thru 0x00a119 (0x323e bytes) 000252 1758737989.379034 mapread 0x00f0fe thru 0x012223 (0x3126 bytes) 000253 1758737989.387245 mapwrite 0x0166dd thru 0x0199a2 (0x32c6 bytes) 000256 1758737989.432874 mapread 0x00fe1d thru 0x010126 (0x30a bytes) 000258 1758737989.434137 write 0x03f8ed thru 0x03ffff (0x713 bytes) 000259 1758737989.434973 write 0x00b4a9 thru 0x00fee6 (0x4a3e bytes) 000260 1758737989.435789 mapread 0x03a89e thru 0x03ffff (0x5762 bytes) 000262 1758737989.447013 write 0x0263af thru 0x02e132 (0x7d84 bytes) 000265 1758737989.469649 mapwrite 0x001a81 thru 0x008304 (0x6884 bytes) 000267 1758737989.521472 write 0x00a520 thru 0x019439 (0xef1a bytes) 000268 1758737989.532286 MIRROR_RESYNC 8 mirrors 000269 1758737990.843161 read 0x02daaf thru 0x02dead (0x3ff bytes) 000270 1758737990.849729 trunc from 0x040000 to 0x00216a (0x3de95 bytes) 000271 1758737990.926878 mapread 0x0013ac thru 0x00216a (0xdbf bytes) 000272 1758737990.940535 write 0x00651b thru 0x00fd4f (0x9835 bytes) 000275 1758737990.943877 write 0x029bf0 thru 0x034095 (0xa4a6 bytes) 000279 1758737990.947220 mapread 0x027657 thru 0x02f7d5 (0x817f bytes) 000283 1758737990.966728 mapread 0x012ca5 thru 0x01c86f (0x9bcb bytes) 000284 1758737991.004649 write 0x015390 thru 0x01f6ea (0xa35b bytes) 000286 1758737991.006312 write 0x031730 thru 0x03d9b7 (0xc288 bytes) 000287 1758737991.013059 write 0x03ec34 thru 0x03ffff (0x13cc bytes) 000288 1758737991.018317 write 0x038d5a thru 0x03b739 (0x29e0 bytes) 000289 1758737991.024723 MIRROR_RESYNC 8 mirrors 000290 1758737992.503247 mapread 0x0219e4 thru 0x02dd53 (0xc370 bytes) 000292 1758737992.539050 mapwrite 0x039ef4 thru 0x03ffff (0x610c bytes) 000293 1758737992.789035 trunc from 0x040000 to 0x0352a4 (0xad5b bytes) 000294 1758737992.818021 read 0x010d5f thru 0x01eb0b (0xddad bytes) 000296 1758737992.829386 read 0x01a7f0 thru 0x021d1e (0x752f bytes) 000297 1758737992.840502 mapread 0x02a2dc thru 0x0352a4 (0xafc9 bytes) 000298 1758737992.841723 mapread 0x01a1ea thru 0x02721d (0xd034 bytes) 000300 1758737992.843965 write 0x02b2a6 thru 0x0313fa (0x6155 bytes) 000301 1758737992.849260 mapread 0x01fd94 thru 0x02f507 (0xf774 bytes) 000304 1758737992.851131 mapwrite 0x0213f8 thru 0x021b59 (0x762 bytes) 000305 1758737992.900337 trunc from 0x0352a5 to 0x01d8b4 (0x179f0 bytes) 000307 1758737992.918039 read 0x010915 thru 0x01566a (0x4d56 bytes) 000308 1758737992.918682 read 0x007412 thru 0x015beb (0xe7da bytes) 000313 1758737992.923975 write 0x02b884 thru 0x03573b (0x9eb8 bytes) 000316 1758737992.929389 mapread 0x019b81 thru 0x019da8 (0x228 bytes) 000319 1758737992.951347 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000320 1758737993.215629 write 0x01d260 thru 0x027942 (0xa6e3 bytes) 000321 1758737993.242147 trunc from 0x03573c to 0x03ed4f (0x9614 bytes) 000322 1758737993.279815 MIRROR_EXTEND to 8 mirrors 000324 1758737994.066386 write 0x009fd6 thru 0x00a2e5 (0x310 bytes) 000326 1758737994.220651 trunc from 0x03ed50 to 0x02885c (0x164f3 bytes) 000327 1758737994.258926 read 0x012e5c thru 0x022775 (0xf91a bytes) 000330 1758737994.277147 read 0x00fe76 thru 0x014f22 (0x50ad bytes) 000333 1758737994.289990 MIRROR_RESYNC 8 mirrors 000334 1758737995.620681 MIRROR_RESYNC 8 mirrors 000335 1758737995.866589 write 0x019f05 thru 0x01c6e3 (0x27df bytes) 000337 1758737995.974845 read 0x005697 thru 0x00e1db (0x8b45 bytes) 000338 1758737995.988837 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000339 1758737996.323970 MIRROR_RESYNC 7 mirrors 000340 1758737997.476333 trunc from 0x02885d to 0x03b5fd (0x12da1 bytes) 000342 1758737997.562584 mapread 0x025daf thru 0x0281b8 (0x240a bytes) 000343 1758737997.582761 mapread 0x00a853 thru 0x0130a2 (0x8850 bytes) 000348 1758737997.604025 read 0x021cc3 thru 0x025835 (0x3b73 bytes) 000349 1758737997.609050 MIRROR_RESYNC 7 mirrors 000350 1758737998.748463 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000351 1758737999.205910 write 0x00d1d0 thru 0x01caa2 (0xf8d3 bytes) 000353 1758737999.343355 mapread 0x008d9a thru 0x00c6c6 (0x392d bytes) 000354 1758737999.354074 write 0x039193 thru 0x03b78a (0x25f8 bytes) 000355 1758737999.363990 mapread 0x013f61 thru 0x0232eb (0xf38b bytes) 000356 1758737999.385244 trunc from 0x03b78b to 0x01662a (0x25160 bytes) 000357 1758737999.421913 trunc from 0x01662b to 0x03e67a (0x28050 bytes) 000359 1758737999.448347 read 0x01350d thru 0x018f02 (0x59f6 bytes) 000360 1758737999.455498 mapread 0x018202 thru 0x01b4af (0x32ae bytes) 000361 1758737999.465763 trunc from 0x03e67b to 0x020347 (0x1e333 bytes) 000362 1758737999.483266 trunc from 0x020348 to 0x01bf80 (0x43c7 bytes) 000365 1758737999.512792 mapread 0x00fd4a thru 0x01bf80 (0xc237 bytes) 000366 1758737999.515134 mapwrite 0x02110f thru 0x02c21c (0xb10e bytes) 000368 1758737999.631321 trunc from 0x02c21d to 0x03ddc6 (0x11baa bytes) 000369 1758737999.668301 mapwrite 0x02f004 thru 0x03dac0 (0xeabd bytes) 000370 1758737999.813970 read 0x0094cd thru 0x00adb1 (0x18e5 bytes) 000371 1758737999.827592 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000373 1758738000.128401 read 0x0380c7 thru 0x03ddc6 (0x5d00 bytes) 000375 1758738000.142704 mapwrite 0x024bcf thru 0x02bfd1 (0x7403 bytes) 000376 1758738000.305222 trunc from 0x03ddc7 to 0x00da77 (0x3034f bytes) 000378 1758738000.336099 write 0x012f48 thru 0x01f2dd (0xc396 bytes) 000382 1758738000.344976 write 0x026410 thru 0x03118e (0xad7f bytes) 000383 1758738000.348912 read 0x006297 thru 0x0121c7 (0xbf31 bytes) 000384 1758738000.369137 mapread 0x00843c thru 0x01142b (0x8ff0 bytes) 000385 1758738000.370587 trunc from 0x03118f to 0x0146cd (0x1cac1 bytes) 000386 1758738000.401096 trunc from 0x0146ce to 0x02bc53 (0x17586 bytes) 000387 1758738000.423797 mapwrite 0x022c7c thru 0x02b2dd (0x8662 bytes) 000388 1758738000.582020 write 0x03c970 thru 0x03ffff (0x3690 bytes) 000391 1758738000.587524 write 0x017a62 thru 0x022e70 (0xb40f bytes) 000394 1758738000.607999 mapread 0x039ad6 thru 0x03ffff (0x652a bytes) 000395 1758738000.608821 mapwrite 0x02a4b6 thru 0x02c7f3 (0x233e bytes) 000400 1758738000.664341 write 0x015a9f thru 0x020717 (0xac79 bytes) 000404 1758738000.671872 mapread 0x0303de thru 0x035b5d (0x5780 bytes) 000406 1758738000.686500 write 0x005164 thru 0x00b8a2 (0x673f bytes) 000408 1758738000.692876 MIRROR_RESYNC 5 mirrors 000409 1758738002.092743 mapread 0x015323 thru 0x023f93 (0xec71 bytes) 000410 1758738002.121834 mapread 0x0270b6 thru 0x036e3f (0xfd8a bytes) 000412 1758738002.141387 mapwrite 0x004567 thru 0x005bd7 (0x1671 bytes) 000413 1758738002.359361 mapwrite 0x035b61 thru 0x03e0f6 (0x8596 bytes) 000414 1758738002.430985 write 0x014965 thru 0x023ee5 (0xf581 bytes) 000416 1758738002.449286 mapread 0x00fe16 thru 0x01adeb (0xafd6 bytes) 000418 1758738002.450650 mapread 0x01e139 thru 0x026822 (0x86ea bytes) 000419 1758738002.466888 trunc from 0x040000 to 0x03be49 (0x41b6 bytes) 000420 1758738002.498627 trunc from 0x03be4a to 0x03e34b (0x2502 bytes) 000422 1758738002.522864 trunc from 0x03e34c to 0x02c71b (0x11c30 bytes) 000424 1758738002.550937 mapwrite 0x009971 thru 0x00e161 (0x47f1 bytes) 000427 1758738002.639027 write 0x039e63 thru 0x03ffff (0x619d bytes) 000430 1758738002.645680 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000431 1758738002.918829 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000432 1758738003.168730 mapwrite 0x00dd46 thru 0x011b04 (0x3dbf bytes) 000433 1758738003.247790 MIRROR_EXTEND to 4 mirrors 000435 1758738004.000307 mapread 0x0355d7 thru 0x03ffff (0xaa29 bytes) 000440 1758738004.044776 mapread 0x0274a1 thru 0x0364fa (0xf05a bytes) 000443 1758738004.069101 trunc from 0x040000 to 0x02a5fc (0x15a03 bytes) 000445 1758738004.190827 mapread 0x014a29 thru 0x0197ff (0x4dd7 bytes) 000446 1758738004.202585 read 0x00ce31 thru 0x0103fc (0x35cc bytes) 000447 1758738004.213277 trunc from 0x02a5fd to 0x001524 (0x290d8 bytes) 000448 1758738004.247673 read 0x000113 thru 0x001524 (0x1412 bytes) 000451 1758738004.255957 mapwrite 0x03c9c7 thru 0x03f74f (0x2d89 bytes) 000453 1758738004.354980 write 0x03562e thru 0x03ffff (0xa9d2 bytes) 000454 1758738004.356140 write 0x036b93 thru 0x03dfee (0x745c bytes) 000455 1758738004.357111 trunc from 0x040000 to 0x029a0a (0x165f5 bytes) 000457 1758738004.383050 read 0x009f54 thru 0x00bd70 (0x1e1d bytes) 000458 1758738004.383954 trunc from 0x029a0b to 0x029440 (0x5ca bytes) 000459 1758738004.410802 write 0x031069 thru 0x03feb7 (0xee4f bytes) 000460 1758738004.412318 trunc from 0x03feb8 to 0x0304a8 (0xfa0f bytes) 000464 1758738004.438195 read 0x02613a thru 0x026538 (0x3ff bytes) 000466 1758738004.443220 MIRROR_EXTEND to 5 mirrors 000467 1758738005.057019 MIRROR_RESYNC 5 mirrors 000475 1758738006.214707 trunc from 0x0304a9 to 0x00670e (0x29d9a bytes) 000477 1758738006.344728 trunc from 0x00670f to 0x03d47a (0x36d6c bytes) 000479 1758738006.400524 trunc from 0x03d47b to 0x03b29b (0x21df bytes) 000480 1758738006.420097 read 0x01987b thru 0x027b7f (0xe305 bytes) 000481 1758738006.430623 trunc from 0x03b29c to 0x00cdf2 (0x2e4a9 bytes) 000482 1758738006.449272 write 0x027fe0 thru 0x030822 (0x8843 bytes) 000485 1758738006.453807 mapwrite 0x00611d thru 0x0068b3 (0x797 bytes) 000487 1758738006.517267 write 0x033251 thru 0x0391f5 (0x5fa5 bytes) 000488 1758738006.519103 mapread 0x02edee thru 0x03196e (0x2b81 bytes) 000489 1758738006.530987 MIRROR_EXTEND to 6 mirrors 000490 1758738007.239910 mapwrite 0x0253cb thru 0x0303bb (0xaff1 bytes) 000491 1758738007.552509 mapwrite 0x03fa93 thru 0x03ffff (0x56d bytes) 000493 1758738007.635268 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000494 1758738007.976513 mapwrite 0x01e36e thru 0x01f964 (0x15f7 bytes) 000495 1758738008.041242 trunc from 0x040000 to 0x02eb97 (0x11468 bytes) 000499 1758738008.080510 write 0x02b4c9 thru 0x0390f5 (0xdc2d bytes) 000501 1758738008.107446 MIRROR_EXTEND to 6 mirrors 000502 1758738008.851310 MIRROR_RESYNC 6 mirrors 000503 1758738009.877073 read 0x0342c4 thru 0x0383f5 (0x4132 bytes) 000504 1758738009.890365 MIRROR_EXTEND to 7 mirrors 000506 1758738010.870441 write 0x006169 thru 0x00d982 (0x781a bytes) 000507 1758738010.987910 write 0x03fa60 thru 0x03ffff (0x5a0 bytes) 000508 1758738010.988877 write 0x0368a1 thru 0x03ba10 (0x5170 bytes) 000509 1758738011.001897 write 0x010bc9 thru 0x019c41 (0x9079 bytes) 000510 1758738011.026116 read 0x01ccd1 thru 0x0288e0 (0xbc10 bytes) 000511 1758738011.043621 mapwrite 0x0122df thru 0x01e775 (0xc497 bytes) 000512 1758738011.172313 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000513 1758738011.533285 MIRROR_EXTEND to 7 mirrors 000515 1758738012.111387 read 0x02d760 thru 0x0389eb (0xb28c bytes) 000516 1758738012.128576 mapread 0x00fda1 thru 0x010592 (0x7f2 bytes) 000517 1758738012.150776 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000518 1758738012.384952 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000519 1758738012.658478 read 0x03d717 thru 0x03ffff (0x28e9 bytes) 000520 1758738012.675453 read 0x03bed5 thru 0x03cc07 (0xd33 bytes) 000522 1758738012.701091 read 0x03b767 thru 0x03ffff (0x4899 bytes) 000523 1758738012.703189 trunc from 0x040000 to 0x031b01 (0xe4fe bytes) 000524 1758738012.765749 mapread 0x01d814 thru 0x01f58e (0x1d7b bytes) 000525 1758738012.777625 write 0x006530 thru 0x006a33 (0x504 bytes) 000528 1758738012.779530 mapread 0x027326 thru 0x02c025 (0x4d00 bytes) 000529 1758738012.781382 trunc from 0x031b02 to 0x00a492 (0x2766f bytes) 000530 1758738012.804332 read 0x0011e5 thru 0x00518e (0x3faa bytes) 000536 1758738012.812688 read 0x006eca thru 0x00a492 (0x35c9 bytes) 000537 1758738012.814158 mapread 0x005647 thru 0x00a492 (0x4e4c bytes) 000542 1758738012.824212 MIRROR_RESYNC 5 mirrors 000543 1758738013.976636 read 0x001af7 thru 0x00a492 (0x899c bytes) 000544 1758738013.985158 mapwrite 0x00c909 thru 0x014a33 (0x812b bytes) 000545 1758738014.162932 write 0x0095f8 thru 0x0141bf (0xabc8 bytes) 000546 1758738014.164158 mapwrite 0x0186cd thru 0x0232ce (0xac02 bytes) 000549 1758738014.272221 write 0x0136e9 thru 0x0176a6 (0x3fbe bytes) 000551 1758738014.276023 read 0x003c3c thru 0x00db65 (0x9f2a bytes) 000554 1758738014.294294 trunc from 0x0232cf to 0x005cae (0x1d620 bytes) 000556 1758738014.320909 mapwrite 0x034800 thru 0x037fe0 (0x37e1 bytes) 000557 1758738014.392692 trunc from 0x037fe1 to 0x02bb64 (0xc47c bytes) 000559 1758738014.414990 mapwrite 0x030fd0 thru 0x03ffdd (0xf00e bytes) 000562 1758738014.543686 mapwrite 0x00db26 thru 0x0139ac (0x5e87 bytes) 000563 1758738014.618182 mapwrite 0x00794d thru 0x010536 (0x8bea bytes) 000564 1758738014.729467 read 0x00eabf thru 0x01dc0c (0xf14e bytes) 000567 1758738014.741646 mapread 0x038285 thru 0x03ffdd (0x7d59 bytes) 000568 1758738014.744306 read 0x02d2c2 thru 0x030030 (0x2d6f bytes) 000569 1758738014.746844 mapread 0x0276ad thru 0x02c906 (0x525a bytes) 000572 1758738014.752691 mapread 0x03df40 thru 0x03f598 (0x1659 bytes) 000573 1758738014.754795 mapread 0x01d8af thru 0x027903 (0xa055 bytes) 000575 1758738014.758940 trunc from 0x03ffde to 0x02c37e (0x13c5f bytes) 000576 1758738014.783456 read 0x01df03 thru 0x02a3f9 (0xc4f7 bytes) 000577 1758738014.784660 trunc from 0x02c37f to 0x02f6cb (0x334d bytes) 000579 1758738014.827724 write 0x03c793 thru 0x03ffff (0x386d bytes) 000580 1758738014.830205 read 0x029344 thru 0x02adf8 (0x1ab5 bytes) 000581 1758738014.830813 read 0x00bf80 thru 0x017d18 (0xbd99 bytes) 000582 1758738014.831383 read 0x02cd9f thru 0x035be8 (0x8e4a bytes) 000586 1758738014.847293 write 0x00eebd thru 0x019c65 (0xada9 bytes) 000587 1758738014.853158 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000589 1758738015.126926 mapread 0x008746 thru 0x00b98b (0x3246 bytes) 000592 1758738015.157887 mapwrite 0x03e2ff thru 0x03ffff (0x1d01 bytes) 000593 1758738015.256025 MIRROR_EXTEND to 5 mirrors 000594 1758738015.859088 read 0x015e04 thru 0x0240ff (0xe2fc bytes) 000596 1758738015.867772 write 0x03af09 thru 0x03ffff (0x50f7 bytes) 000599 1758738015.978362 read 0x00814b thru 0x01742a (0xf2e0 bytes) 000600 1758738015.993516 mapread 0x032108 thru 0x036982 (0x487b bytes) 000601 1758738015.995359 trunc from 0x040000 to 0x02714e (0x18eb1 bytes) 000604 1758738016.057520 trunc from 0x02714f to 0x02f21d (0x80cf bytes) 000605 1758738016.083019 write 0x028a2e thru 0x02e151 (0x5724 bytes) 000606 1758738016.096477 mapwrite 0x022df0 thru 0x02558c (0x279d bytes) 000607 1758738016.164198 write 0x01a5ae thru 0x023ee6 (0x9939 bytes) 000608 1758738016.165430 trunc from 0x02f21e to 0x02a07d (0x51a0 bytes) 000612 1758738016.200524 MIRROR_EXTEND to 6 mirrors 000614 1758738016.911978 trunc from 0x02a07e to 0x0335b2 (0x9535 bytes) 000617 1758738017.002423 MIRROR_EXTEND to 7 mirrors 000618 1758738017.669731 read 0x00dd85 thru 0x0199e6 (0xbc62 bytes) 000619 1758738017.684288 mapwrite 0x03b67a thru 0x03ffff (0x4986 bytes) 000620 1758738017.905347 mapread 0x014a4f thru 0x01725a (0x280c bytes) 000621 1758738017.924415 trunc from 0x040000 to 0x03841e (0x7be1 bytes) 000624 1758738017.969885 trunc from 0x03841f to 0x036bee (0x1830 bytes) 000625 1758738018.011475 trunc from 0x036bef to 0x017b79 (0x1f075 bytes) 000627 1758738018.050184 read 0x009558 thru 0x011eb6 (0x895f bytes) 000629 1758738018.068722 write 0x035589 thru 0x03d199 (0x7c11 bytes) 000630 1758738018.071176 mapread 0x0262ec thru 0x02644e (0x163 bytes) 000631 1758738018.081914 trunc from 0x03d19a to 0x00f46b (0x2dd2e bytes) 000632 1758738018.109583 MIRROR_EXTEND to 8 mirrors 000633 1758738018.773718 read 0x0062df thru 0x0079fb (0x171d bytes) 000635 1758738018.788888 read 0x00782f thru 0x00f46b (0x7c3d bytes) 000637 1758738018.806229 trunc from 0x00f46c to 0x01bca7 (0xc83c bytes) 000638 1758738018.926769 trunc from 0x01bca8 to 0x0210bd (0x5416 bytes) 000639 1758738018.968991 read 0x00807b thru 0x0108b1 (0x8837 bytes) 000640 1758738018.984519 mapread 0x00ee6b thru 0x01d124 (0xe2ba bytes) 000642 1758738018.989952 read 0x013df2 thru 0x01ffb5 (0xc1c4 bytes) 000644 1758738019.006389 MIRROR_EXTEND to 9 mirrors 000647 1758738019.808890 trunc from 0x0210be to 0x02007b (0x1042 bytes) 000651 1758738019.915659 read 0x008974 thru 0x018038 (0xf6c5 bytes) 000652 1758738019.927274 mapwrite 0x0086ef thru 0x00c9dc (0x42ee bytes) 000655 1758738019.984803 trunc from 0x02007c to 0x024ec5 (0x4e4a bytes) 000658 1758738020.017213 mapread 0x00f062 thru 0x01420d (0x51ac bytes) 000659 1758738020.018065 read 0x021f1f thru 0x024ec5 (0x2fa7 bytes) 000661 1758738020.029463 read 0x01ab89 thru 0x023c91 (0x9109 bytes) 000663 1758738020.030751 trunc from 0x024ec6 to 0x023616 (0x18af bytes) 000665 1758738020.055237 mapread 0x00cb7f thru 0x01ae81 (0xe303 bytes) 000666 1758738020.056167 mapwrite 0x00578f thru 0x00bc84 (0x64f6 bytes) 000667 1758738020.145972 write 0x004533 thru 0x01013d (0xbc0b bytes) 000671 1758738020.150340 write 0x035737 thru 0x037666 (0x1f30 bytes) 000672 1758738020.151424 write 0x02d412 thru 0x02e18b (0xd7a bytes) 000673 1758738020.164877 trunc from 0x037667 to 0x017d1d (0x1f949 bytes) 000674 1758738020.200677 mapwrite 0x00a7e1 thru 0x00e7ee (0x400e bytes) 000675 1758738020.264765 mapread 0x005bf6 thru 0x009809 (0x3c14 bytes) 000676 1758738020.266321 read 0x0126ef thru 0x017d1d (0x562f bytes) 000677 1758738020.267841 mapread 0x00406c thru 0x00ce5c (0x8df1 bytes) 000678 1758738020.269500 mapread 0x0162ce thru 0x017d1d (0x1a50 bytes) 000680 1758738020.282236 MIRROR_RESYNC 9 mirrors 000681 1758738021.848985 trunc from 0x017d1e to 0x003e16 (0x13f07 bytes) 000682 1758738021.947334 read 0x000f71 thru 0x003e16 (0x2ea6 bytes) 000683 1758738021.953438 trunc from 0x003e17 to 0x01193c (0xdb26 bytes) 000684 1758738021.977755 write 0x000ebc thru 0x00a9c2 (0x9b07 bytes) 000686 1758738021.987123 write 0x02c4cc thru 0x036704 (0xa239 bytes) 000689 1758738021.990106 trunc from 0x036705 to 0x030bf0 (0x5b14 bytes) 000690 1758738022.007850 read 0x02ca4d thru 0x030bf0 (0x41a4 bytes) truncating to largest ever: 0x3fa33 000692 1758738022.008957 trunc from 0x030bf1 to 0x03fa32 (0xee42 bytes) 000695 1758738022.031657 write 0x00daef thru 0x014094 (0x65a6 bytes) 000696 1758738022.044990 read 0x03d2f8 thru 0x03fa32 (0x273b bytes) 000698 1758738022.046071 write 0x007391 thru 0x015927 (0xe597 bytes) 000700 1758738022.047883 mapwrite 0x031351 thru 0x032384 (0x1034 bytes) 000701 1758738022.102250 write 0x013d51 thru 0x018daf (0x505f bytes) 000702 1758738022.104731 trunc from 0x03fa33 to 0x03f7a3 (0x28f bytes) 000703 1758738022.120942 write 0x025ec5 thru 0x029d07 (0x3e43 bytes) 000704 1758738022.125662 write 0x01c642 thru 0x0215fc (0x4fbb bytes) 000705 1758738022.128903 write 0x038bf2 thru 0x03ee52 (0x6261 bytes) 000706 1758738022.131719 mapread 0x03732b thru 0x03f7a3 (0x8479 bytes) 000707 1758738022.134998 write 0x0157ce thru 0x015dc9 (0x5fc bytes) 000708 1758738022.136175 mapread 0x02989a thru 0x036059 (0xc7c0 bytes) 000710 1758738022.138164 trunc from 0x03f7a4 to 0x035478 (0xa32b bytes) 000711 1758738022.162112 MIRROR_RESYNC 9 mirrors 000713 1758738023.448256 trunc from 0x035479 to 0x02bb42 (0x9936 bytes) 000714 1758738023.579231 mapread 0x00db25 thru 0x0132bd (0x5799 bytes) 000715 1758738023.637285 read 0x0034ad thru 0x00aed4 (0x7a28 bytes) 000718 1758738023.647889 trunc from 0x02bb43 to 0x03cf09 (0x113c7 bytes) 000719 1758738023.697881 write 0x037d67 thru 0x03d84c (0x5ae6 bytes) 000720 1758738023.708870 read 0x00f449 thru 0x011070 (0x1c28 bytes) 000721 1758738023.714289 MIRROR_EXTEND to 10 mirrors 000722 1758738024.489789 mapwrite 0x00d506 thru 0x01b36e (0xde69 bytes) 000724 1758738024.710891 mapwrite 0x02f67c thru 0x031ee0 (0x2865 bytes) 000725 1758738024.770047 trunc from 0x03d84d to 0x016113 (0x27739 bytes) 000729 1758738024.797194 read 0x011380 thru 0x016113 (0x4d94 bytes) 000730 1758738024.800898 MIRROR_EXTEND to 11 mirrors 000731 1758738025.388761 read 0x0084be thru 0x016113 (0xdc56 bytes) 000732 1758738025.409164 write 0x0249fd thru 0x033ba5 (0xf1a9 bytes) 000733 1758738025.494279 trunc from 0x033ba6 to 0x037f35 (0x4390 bytes) 000735 1758738025.525204 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000737 1758738025.809871 read 0x0375e9 thru 0x037f35 (0x94d bytes) 000738 1758738025.817374 mapwrite 0x00a364 thru 0x00c7dc (0x2479 bytes) 000739 1758738025.929098 MIRROR_RESYNC 10 mirrors 000740 1758738027.402673 mapwrite 0x023ef0 thru 0x03215a (0xe26b bytes) 000741 1758738027.587053 write 0x00a60d thru 0x010982 (0x6376 bytes) 000743 1758738027.596767 mapread 0x01499e thru 0x01c63e (0x7ca1 bytes) 000745 1758738027.606577 write 0x000507 thru 0x00bc6b (0xb765 bytes) 000749 1758738027.614600 mapwrite 0x018b4a thru 0x02549f (0xc956 bytes) 000750 1758738027.669763 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000752 1758738028.001591 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000753 1758738028.247171 write 0x003cb6 thru 0x005710 (0x1a5b bytes) 000757 1758738028.266670 MIRROR_EXTEND to 9 mirrors 000758 1758738028.855047 trunc from 0x037f36 to 0x03225f (0x5cd6 bytes) 000759 1758738028.964732 trunc from 0x032260 to 0x02ca12 (0x584d bytes) 000760 1758738028.988464 trunc from 0x02ca13 to 0x00b898 (0x2117a bytes) 000761 1758738029.010624 mapread 0x007ced thru 0x00b898 (0x3bac bytes) 000762 1758738029.033134 read 0x00259e thru 0x0055e5 (0x3048 bytes) 000763 1758738029.037896 MIRROR_EXTEND to 10 mirrors 000764 1758738029.852232 mapread 0x0011c4 thru 0x00b898 (0xa6d5 bytes) 000765 1758738029.881150 mapread 0x007027 thru 0x007852 (0x82c bytes) 000768 1758738029.907664 mapread 0x002ac4 thru 0x0086f4 (0x5c31 bytes) 000769 1758738029.927767 mapread 0x009329 thru 0x00a284 (0xf5c bytes) 000772 1758738029.973148 mapread 0x005334 thru 0x00b4c5 (0x6192 bytes) 000774 1758738030.018138 trunc from 0x00b899 to 0x009d14 (0x1b84 bytes) 000775 1758738030.134192 trunc from 0x009d15 to 0x026767 (0x1ca53 bytes) 000776 1758738030.167721 mapread 0x01a40d thru 0x026767 (0xc35b bytes) 000778 1758738030.199491 trunc from 0x026768 to 0x009478 (0x1d2ef bytes) 000779 1758738030.216783 trunc from 0x009479 to 0x03e845 (0x353cd bytes) 000780 1758738030.236087 write 0x026c07 thru 0x0331d1 (0xc5cb bytes) 000782 1758738030.257875 write 0x02e2de thru 0x033a9a (0x57bd bytes) 000783 1758738030.258859 read 0x034dec thru 0x038d19 (0x3f2e bytes) 000784 1758738030.282984 trunc from 0x03e846 to 0x023319 (0x1b52c bytes) 000785 1758738030.307683 read 0x00e864 thru 0x00ebdf (0x37c bytes) 000786 1758738030.326453 mapwrite 0x01a37d thru 0x020fb1 (0x6c35 bytes) 000788 1758738030.455965 MIRROR_RESYNC 10 mirrors 000790 1758738032.182930 mapwrite 0x007ee3 thru 0x0096f5 (0x1813 bytes) 000794 1758738032.386565 read 0x01d05c thru 0x023319 (0x62be bytes) 000798 1758738032.408252 write 0x000fb2 thru 0x0047ec (0x383b bytes) 000800 1758738032.428574 mapwrite 0x01732f thru 0x01e3e9 (0x70bb bytes) 000801 1758738032.517578 mapread 0x0195c8 thru 0x02002b (0x6a64 bytes) 000802 1758738032.524493 MIRROR_EXTEND to 11 mirrors 000803 1758738033.176922 write 0x00eaf7 thru 0x00faaa (0xfb4 bytes) 000804 1758738033.306477 mapwrite 0x0155b5 thru 0x01c7ed (0x7239 bytes) 000805 1758738033.366240 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000807 1758738033.737767 mapwrite 0x0075d1 thru 0x00fe1f (0x884f bytes) 000808 1758738033.848530 trunc from 0x02331a to 0x00e818 (0x14b01 bytes) 000809 1758738033.896303 write 0x03dc44 thru 0x03ffff (0x23bc bytes) 000811 1758738033.898103 write 0x00f2aa thru 0x015c28 (0x697f bytes) 000813 1758738033.907884 write 0x037ad0 thru 0x03ffff (0x8530 bytes) 000817 1758738033.917211 read 0x00a9f2 thru 0x018eae (0xe4bd bytes) 000818 1758738033.926042 MIRROR_EXTEND to 11 mirrors 000819 1758738034.675625 read 0x006288 thru 0x0115f5 (0xb36e bytes) 000821 1758738034.724774 mapwrite 0x02fc5f thru 0x03a7ad (0xab4f bytes) 000822 1758738034.976262 mapread 0x02b749 thru 0x036dab (0xb663 bytes) 000824 1758738034.977672 trunc from 0x040000 to 0x00f6ce (0x30931 bytes) 000825 1758738035.001382 trunc from 0x00f6cf to 0x00c1db (0x34f3 bytes) 000826 1758738035.015793 trunc from 0x00c1dc to 0x023da0 (0x17bc5 bytes) 000829 1758738035.039867 write 0x026dfb thru 0x02da5c (0x6c62 bytes) 000831 1758738035.041490 mapwrite 0x026610 thru 0x029b97 (0x3588 bytes) 000834 1758738035.096169 read 0x026a0a thru 0x029ca7 (0x329e bytes) 000835 1758738035.096780 write 0x02aa91 thru 0x02ac3c (0x1ac bytes) 000836 1758738035.097597 read 0x018984 thru 0x01d207 (0x4884 bytes) 000837 1758738035.115991 mapwrite 0x027adc thru 0x0316a0 (0x9bc5 bytes) 000838 1758738035.205313 read 0x01162a thru 0x019cb3 (0x868a bytes) 000839 1758738035.205979 write 0x03e895 thru 0x03ffff (0x176b bytes) 000841 1758738035.212051 read 0x03970d thru 0x03ffff (0x68f3 bytes) 000844 1758738035.222718 trunc from 0x040000 to 0x01ae19 (0x251e6 bytes) 000845 1758738035.255294 mapread 0x008992 thru 0x00cb9c (0x420b bytes) 000846 1758738035.256194 mapwrite 0x0011d9 thru 0x00c6b6 (0xb4de bytes) 000848 1758738035.354579 trunc from 0x01ae1a to 0x01034b (0xaace bytes) 000849 1758738035.390041 read 0x001c17 thru 0x002d3a (0x1124 bytes) 000850 1758738035.390884 mapread 0x005d8f thru 0x00df1e (0x8190 bytes) 000851 1758738035.392241 read 0x004af1 thru 0x0071a5 (0x26b5 bytes) 000854 1758738035.394674 trunc from 0x01034c to 0x009545 (0x6e06 bytes) 000855 1758738035.413624 mapwrite 0x01b96f thru 0x02abdc (0xf26e bytes) 000857 1758738035.519717 read 0x016a11 thru 0x02007a (0x966a bytes) 000858 1758738035.520354 write 0x01ac01 thru 0x01d1e1 (0x25e1 bytes) 000859 1758738035.521233 write 0x011079 thru 0x01b4ab (0xa433 bytes) 000860 1758738035.546951 MIRROR_EXTEND to 12 mirrors 000861 1758738036.129792 trunc from 0x02abdd to 0x0388db (0xdcff bytes) 000862 1758738036.258938 trunc from 0x0388dc to 0x02e125 (0xa7b6 bytes) 000865 1758738036.284470 trunc from 0x02e126 to 0x038f4d (0xae28 bytes) 000866 1758738036.306967 mapread 0x009569 thru 0x0140ed (0xab85 bytes) 000867 1758738036.334502 mapread 0x029413 thru 0x032358 (0x8f46 bytes) 000868 1758738036.350502 write 0x035da1 thru 0x03ffff (0xa25f bytes) 000869 1758738036.351904 write 0x0058db thru 0x00a6ed (0x4e13 bytes) 000875 1758738036.355029 trunc from 0x040000 to 0x019729 (0x268d6 bytes) 000876 1758738036.377586 mapwrite 0x03b641 thru 0x03ffff (0x49bf bytes) 000879 1758738036.483209 mapread 0x02c3d0 thru 0x039db6 (0xd9e7 bytes) 000882 1758738036.493373 mapread 0x021753 thru 0x025135 (0x39e3 bytes) 000883 1758738036.503894 write 0x019a29 thru 0x01abfa (0x11d2 bytes) 000885 1758738036.505473 read 0x01f27a thru 0x024dae (0x5b35 bytes) 000886 1758738036.507342 mapwrite 0x02a6c8 thru 0x034a4b (0xa384 bytes) 000887 1758738036.579408 read 0x02435a thru 0x02ec79 (0xa920 bytes) 000888 1758738036.580074 write 0x000ec0 thru 0x0089f4 (0x7b35 bytes) 000889 1758738036.592042 MIRROR_EXTEND to 13 mirrors 000890 1758738037.255865 mapread 0x00e72a thru 0x012e89 (0x4760 bytes) 000891 1758738037.288959 mapwrite 0x02e762 thru 0x030b74 (0x2413 bytes) 000894 1758738037.509955 trunc from 0x040000 to 0x01ce27 (0x231d8 bytes) 000896 1758738037.556050 MIRROR_RESYNC 13 mirrors 000897 1758738039.478851 trunc from 0x01ce28 to 0x031cfa (0x14ed3 bytes) 000904 1758738039.597881 MIRROR_RESYNC 13 mirrors 000905 1758738041.275918 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000906 1758738041.798777 MIRROR_RESYNC 12 mirrors 000910 1758738042.098068 mapwrite 0x017548 thru 0x01ea3b (0x74f4 bytes) 000911 1758738042.316322 write 0x0294c5 thru 0x032ad1 (0x960d bytes) 000912 1758738042.331462 mapread 0x02e3b8 thru 0x032ad1 (0x471a bytes) 000914 1758738042.333879 read 0x0007e6 thru 0x00769c (0x6eb7 bytes) 000916 1758738042.342410 read 0x0239fb thru 0x02b838 (0x7e3e bytes) 000918 1758738042.348793 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000919 1758738042.642559 mapread 0x01629c thru 0x01f691 (0x93f6 bytes) 000920 1758738042.685478 mapread 0x015ebb thru 0x0160f8 (0x23e bytes) 000921 1758738042.690296 mapread 0x01b766 thru 0x0247cb (0x9066 bytes) 000922 1758738042.702175 read 0x0134a8 thru 0x01abca (0x7723 bytes) 000926 1758738042.705150 mapread 0x00df27 thru 0x01b454 (0xd52e bytes) 000928 1758738042.717528 trunc from 0x032ad2 to 0x011d29 (0x20da8 bytes) 000929 1758738042.753069 MIRROR_RESYNC 11 mirrors 000931 1758738044.383425 read 0x00b998 thru 0x011d29 (0x6392 bytes) 000932 1758738044.393636 read 0x007ee1 thru 0x011d29 (0x9e49 bytes) 000933 1758738044.406610 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000934 1758738044.803003 read 0x00ab77 thru 0x00b5cd (0xa57 bytes) 000937 1758738044.811215 write 0x029583 thru 0x029616 (0x94 bytes) 000938 1758738044.941953 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000941 1758738045.391174 read 0x02511c thru 0x029616 (0x44fb bytes) 000942 1758738045.406159 trunc from 0x029617 to 0x02a60a (0xff4 bytes) 000945 1758738045.448049 trunc from 0x02a60b to 0x00d3fd (0x1d20d bytes) 000947 1758738045.474815 trunc from 0x00d3fe to 0x00acb6 (0x2747 bytes) 000949 1758738045.494974 read 0x008c02 thru 0x00acb6 (0x20b5 bytes) 000951 1758738045.507451 MIRROR_EXTEND to 10 mirrors 000952 1758738046.072852 MIRROR_EXTEND to 11 mirrors 000953 1758738046.796533 trunc from 0x00acb7 to 0x025156 (0x1a4a0 bytes) 000954 1758738046.987152 mapwrite 0x02ff64 thru 0x037570 (0x760d bytes) 000955 1758738047.103377 mapread 0x01b7ae thru 0x027336 (0xbb89 bytes) 000956 1758738047.124644 write 0x02b8d5 thru 0x0304bb (0x4be7 bytes) 000958 1758738047.126029 write 0x0214ea thru 0x0298eb (0x8402 bytes) 000959 1758738047.127088 write 0x02e449 thru 0x037bfc (0x97b4 bytes) 000962 1758738047.129057 mapread 0x01cabc thru 0x01e7ca (0x1d0f bytes) 000963 1758738047.129908 trunc from 0x037bfd to 0x02d490 (0xa76c bytes) 000966 1758738047.160123 mapread 0x01ea5c thru 0x027e56 (0x93fb bytes) 000968 1758738047.163046 mapread 0x0222f9 thru 0x02d490 (0xb198 bytes) 000969 1758738047.165056 trunc from 0x02d491 to 0x02ed4e (0x18be bytes) 000970 1758738047.187680 write 0x0229c1 thru 0x0306cc (0xdd0c bytes) 000971 1758738047.190457 write 0x0032f5 thru 0x00e371 (0xb07d bytes) 000972 1758738047.201627 mapwrite 0x0359c6 thru 0x0362f6 (0x931 bytes) 000974 1758738047.292986 MIRROR_EXTEND to 12 mirrors 000975 1758738047.908208 write 0x01cfca thru 0x028876 (0xb8ad bytes) 000976 1758738047.996483 mapwrite 0x01db74 thru 0x023114 (0x55a1 bytes) 000977 1758738048.053398 mapwrite 0x015a63 thru 0x01bda2 (0x6340 bytes) 000979 1758738048.125671 MIRROR_RESYNC 12 mirrors 000987 1758738049.735778 mapwrite 0x03a9aa thru 0x03ffff (0x5656 bytes) 000988 1758738049.959731 trunc from 0x040000 to 0x03ee4e (0x11b1 bytes) 000989 1758738049.993306 write 0x018db8 thru 0x01f772 (0x69bb bytes) 000990 1758738050.008837 trunc from 0x03ee4f to 0x01bb46 (0x23308 bytes) 000991 1758738050.053244 MIRROR_RESYNC 12 mirrors 000993 1758738051.874638 read 0x00a49b thru 0x015eb3 (0xba19 bytes) 000994 1758738051.896452 write 0x029330 thru 0x032ab6 (0x9787 bytes) 000995 1758738052.001122 MIRROR_EXTEND to 13 mirrors 000996 1758738052.734656 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) 000997 1758738053.203488 mapread 0x01bac6 thru 0x02a15d (0xe698 bytes) 000998 1758738053.286372 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18b:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 14:21:00 (1758738060) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 14:21:03 (1758738063) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1753088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5593920 .. failed Extending file size to 8088192 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8311072 .. Extending file size to 9186400 .. Extending file size to 5770400 .. Extending file size to 6529248 .. Extending file size to 7250528 .. Extending file size to 8779360 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 54679 54680 54683 54686 54688 failed 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:22:20 (1758738140) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8949216 .. Extending file size to 2423840 .. Extending file size to 2519552 .. Extending file size to 5045888 .. Extending file size to 8719872 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8858176 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56676 56677 56679 56681 56683 done 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:23:36 (1758738216) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109248 s, 19.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304959 s, 6.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:23:52 (1758738232) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0611579 s, 34.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 14:23:58 (1758738238) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:24:04 (1758738244) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224119 s, 9.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:24:13 (1758738253) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132283 s, 15.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x260:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x260:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:24:21 (1758738261) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128897 s, 16.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:24:54 (1758738294) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140827 s, 7.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:25:49 (1758738349) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0943607 s, 22.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:26:42 (1758738402) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. Creating new pool test_204f oleg150-server: Pool lustre.test_204f created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202761 s, 10.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg150-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:27:53 (1758738473) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:27:59 (1758738479) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:28:06 (1758738486) oleg150-server: Pool lustre.test_206 created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg150-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:28:27 (1758738507) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499485 s, 21.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261737 s, 40.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xc40f3f34 Mirror 3: 0xc40f3f34 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:28:36 (1758738516) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:28:39 (1758738519) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:28:43 (1758738523) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.135309 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2480000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a00000 Read this (4 bytes): BBBB 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:29:20 (1758738560) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0727993 s, 113 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:29:28 (1758738568) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417909 s, 25.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834428 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:29:29.000000000 -0400 Modify: 2025-09-24 14:29:29.000000000 -0400 Change: 2025-09-24 14:29:30.000000000 -0400 Birth: 2025-09-24 14:29:29.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:29:35 (1758738575) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:29:38 (1758738578) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.584378 s, 70.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.101043 s, 40.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 24 14:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 24 14:29 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test complete, duration 1429 sec =========== 14:29:47 (1758738587) === sanity-flr: start cleanup 14:29:49 (1758738589) === === sanity-flr: finish cleanup 14:29:55 (1758738595) ===