-----============= acceptance-small: sanity-flr ============----- Wed Mar 11 08:40:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg447-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:40:34 (1773232834) === oleg447-client.virtnet: executing check_config_client /mnt/lustre oleg447-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg447-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9df710fe0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9df710fe0000.idle_timeout=debug disable quota as required oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:40:46 (1773232846) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7553] == sanity-flr test 0a: lfs mirror create with -N option == 08:40:47 (1773232847) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:40:53 (1773232853) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:40:55 (1773232855) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:40:56 (1773232856) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:41:07 (1773232867) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:41:08 (1773232868) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 08:41:10 (1773232870) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9df710fe0000.stats=clear osc.lustre-OST0001-osc-ffff9df710fe0000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:41:14 (1773232874) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:41:18 (1773232878) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:41:22 (1773232882) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:41:23 (1773232883) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 08:41:27 (1773232887) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:41:30 (1773232890) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:41:32 (1773232892) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:41:35 (1773232895) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:41:40 (1773232900) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:41:43 (1773232903) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897036 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658097 s, 15.9 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:41:48 (1773232908) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.14389 s, 102 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:41:51 (1773232911) Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:42:24 (1773232944) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.13502 s, 46.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:42:39 (1773232959) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:43:02 (1773232982) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4883 s, 35.2 MB/s -rw-r--r-- 1 root root 52428800 Mar 11 08:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.318s user 0m0.001s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.357s user 0m0.000s sys 0m0.159s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.451s user 0m0.002s sys 0m0.159s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9df710fe0000.ost_server_uuid in FULL state after 0 sec PASS 33b (29s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:43:31 (1773233011) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:43:32 (1773233012) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:43:34 (1773233014) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 08:43:35 (1773233015) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 08:43:38 (1773233018) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.177148 s, 71.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:43:42 (1773233022) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163928 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145598 s, 72.0 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:43:45 (1773233025) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:43:49 (1773233029) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000938701 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000962085 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000743773 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000792797 s, 1.3 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 08:43:54 (1773233034) create test files with size 15738579 .. 59159c363b708f33be2268402b75c034 - 664ba4118995e13e300fc60408fc5356 - b6fc0b62c72dd676e8a215734d3ea732 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 08:44:01 (1773233041) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:44:08 (1773233048) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:44:11 (1773233051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345709 s, 30.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140452 s, 74.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 08:44:14 (1773233054) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0636977 s, 32.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.143847 s, 29.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0627311 s, 33.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 08:44:20 (1773233060) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 08:44:22 (1773233062) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:44:23 (1773233063) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131092 s, 80.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 08:44:25 (1773233065) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:44:27 (1773233067) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:44:30 (1773233070) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118943 s, 88.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:44:49 (1773233089) llite.lustre-ffff9df703fe3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.100073 s, 105 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9df7031af800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:44:54 (1773233094) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:45:09 (1773233109) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:45:22 (1773233122) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 08:45:23 (1773233123) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0809767 s, 38.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.070844 s, 44.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaa53f3ce (stale) Mirror 2: 0xaa53f3ce (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 62f6598c1b13e785f7db7e9771200d52 - mirror 1 checksum: 11f6b35c44bb15886c54bbfa858994f7 - mirror 2 checksum: 62f6598c1b13e785f7db7e9771200d52 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Mar 11 08:45:25 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Mar 11 08:45:25 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 62f6598c1b13e785f7db7e9771200d52 - mirror 1 checksum: 11f6b35c44bb15886c54bbfa858994f7 - mirror 2 checksum: 11f6b35c44bb15886c54bbfa858994f7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Mar 11 08:45:25 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Mar 11 08:45:25 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:45:27 (1773233127) Failing mds1 on oleg447-server Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 08:45:30 (1773233130) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server Failover mds1 to oleg447-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 08:45:44 (1773233144) targets are mounted 08:45:44 (1773233144) facet_failover done oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02065 s, 50.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:45:51 (1773233151) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:45:52 (1773233152) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:45:53 (1773233153) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00298997 s, 5.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:45:56 (1773233156) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:45:57 (1773233157) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:45:59 (1773233159) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:46:07 (1773233167) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:46:27 (1773233187) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:46:45 (1773233205) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:47:03 (1773233223) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 08:47:05 (1773233225) mirror create pid 52207 mirror split pid 52208 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52207 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52208 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 08:48:08 (1773233288) Seed set to 3288 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1773233288.769113 write 0x032470 thru 0x03415f (0x1cf0 bytes) truncating to largest ever: 0x262e3 000002 1773233288.772643 trunc from 0x034160 to 0x0262e2 (0xde7d bytes) 000006 1773233288.778602 mapread 0x006c47 thru 0x0125e4 (0xb99e bytes) 000007 1773233288.787126 read 0x011d1f thru 0x01bd83 (0xa065 bytes) 000011 1773233288.790293 write 0x029aff thru 0x03054a (0x6a4c bytes) 000013 1773233288.791611 trunc from 0x03054b to 0x0150db (0x1b46f bytes) 000014 1773233288.796399 mapwrite 0x01eb03 thru 0x01fe1b (0x1319 bytes) 000018 1773233288.831093 read 0x0070c3 thru 0x015d5b (0xec99 bytes) 000020 1773233288.832420 mapread 0x01195f thru 0x01e138 (0xc7da bytes) 000022 1773233288.839242 read 0x01c9ec thru 0x01fe1b (0x3430 bytes) 000023 1773233288.839901 read 0x0061f3 thru 0x0134d3 (0xd2e1 bytes) truncating to largest ever: 0x3e003 000024 1773233288.840479 trunc from 0x01fe1c to 0x03e002 (0x1e1e7 bytes) 000027 1773233288.851308 mapread 0x0239b5 thru 0x03267d (0xecc9 bytes) 000030 1773233288.861934 mapread 0x0275fa thru 0x0335bb (0xbfc2 bytes) 000031 1773233288.863074 mapread 0x0302cc thru 0x037c73 (0x79a8 bytes) 000032 1773233288.864192 write 0x01b282 thru 0x029736 (0xe4b5 bytes) 000033 1773233288.865602 write 0x019bf7 thru 0x0271e5 (0xd5ef bytes) 000036 1773233288.868362 mapwrite 0x035b83 thru 0x0390a2 (0x3520 bytes) 000039 1773233288.888347 read 0x032631 thru 0x03e002 (0xb9d2 bytes) 000041 1773233288.889458 read 0x01d1cd thru 0x0247df (0x7613 bytes) 000042 1773233288.890013 mapread 0x02b310 thru 0x02ca2c (0x171d bytes) 000043 1773233288.890934 mapread 0x014a19 thru 0x01f62d (0xac15 bytes) 000045 1773233288.892066 trunc from 0x03e003 to 0x021066 (0x1cf9c bytes) 000047 1773233288.900422 write 0x00a268 thru 0x00bce4 (0x1a7d bytes) 000050 1773233288.902222 mapread 0x01e261 thru 0x021066 (0x2e06 bytes) 000051 1773233288.903060 read 0x000557 thru 0x0020dd (0x1b87 bytes) 000052 1773233288.903486 trunc from 0x021067 to 0x0101b9 (0x10ead bytes) 000058 1773233288.915964 write 0x022032 thru 0x02e253 (0xc222 bytes) 000065 1773233288.920017 write 0x00bacf thru 0x0116ee (0x5c20 bytes) 000067 1773233288.924209 write 0x036a89 thru 0x03ffff (0x9577 bytes) 000068 1773233288.927711 MIRROR_EXTEND to 2 mirrors 000070 1773233289.162015 mapread 0x019a72 thru 0x0257ba (0xbd49 bytes) 000073 1773233289.172344 trunc from 0x040000 to 0x02c3be (0x13c41 bytes) 000075 1773233289.199901 trunc from 0x02c3bf to 0x02b5e6 (0xdd8 bytes) 000076 1773233289.208269 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b5e7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b5e7): Mirror 2: 0xbd452d68 (stale) CRC-32 checksum value for chunk [0, 0x2b5e7): Mirror 1: 0xbd452d68 Mirror 3: 0xbd452d68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b5e7: skipped 000077 1773233289.494728 trunc from 0x02b5e7 to 0x02675a (0x4e8c bytes) truncating to largest ever: 0x3f969 000082 1773233289.520223 trunc from 0x02675b to 0x03f968 (0x1920e bytes) 000084 1773233289.528541 mapwrite 0x01b28e thru 0x0203e9 (0x515c bytes) 000087 1773233289.555655 trunc from 0x03f969 to 0x00154d (0x3e41b bytes) 000089 1773233289.562491 trunc from 0x00154e to 0x0123d3 (0x10e86 bytes) 000095 1773233289.572430 MIRROR_RESYNC 3 mirrors 000096 1773233289.796531 trunc from 0x0123d4 to 0x02ccf3 (0x1a920 bytes) 000097 1773233289.816139 write 0x028e0b thru 0x02ec52 (0x5e48 bytes) 000099 1773233289.819715 read 0x00d4f6 thru 0x0148ba (0x73c5 bytes) 000100 1773233289.823313 mapwrite 0x020863 thru 0x02a90b (0xa0a9 bytes) 000102 1773233289.849858 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ec53): Mirror 2: 0xa9bea16c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec53): Mirror 2: 0xa9bea16c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec53: skipped 000103 1773233289.933093 trunc from 0x02ec53 to 0x01a5e9 (0x14669 bytes) 000104 1773233289.938207 mapread 0x01452e thru 0x01a5e9 (0x60bc bytes) 000108 1773233289.945487 mapwrite 0x028faa thru 0x02c334 (0x338b bytes) 000111 1773233289.970004 read 0x00a195 thru 0x0109dd (0x6849 bytes) 000113 1773233289.973673 read 0x0037a4 thru 0x01081b (0xd078 bytes) 000114 1773233289.976725 mapwrite 0x00f6a7 thru 0x01c2e2 (0xcc3c bytes) 000116 1773233289.992936 trunc from 0x02c335 to 0x00c179 (0x201bb bytes) 000117 1773233290.001043 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc17a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc17a): Mirror 2: 0x842ed83f (stale) CRC-32 checksum value for chunk [0, 0xc17a): Mirror 1: 0x842ed83f Mirror 3: 0x842ed83f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc17a: skipped 000119 1773233290.307549 read 0x00a4f9 thru 0x00c179 (0x1c81 bytes) 000120 1773233290.311385 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc17a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc17a): Mirror 2: 0x842ed83f (stale) CRC-32 checksum value for chunk [0, 0xc17a): Mirror 1: 0x842ed83f Mirror 3: 0x842ed83f Mirror 4: 0x842ed83f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc17a: skipped 000122 1773233290.581806 write 0x011121 thru 0x01c6fb (0xb5db bytes) 000123 1773233290.600910 read 0x018527 thru 0x01c6fb (0x41d5 bytes) 000124 1773233290.603121 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c6fc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c6fc): Mirror 2: 0xf51d0b91 (stale) Mirror 3: 0xf51d0b91 (stale) Mirror 4: 0xf51d0b91 (stale) CRC-32 checksum value for chunk [0, 0x1c6fc): Mirror 1: 0x71fc6098 Mirror 5: 0x71fc6098 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c6fc: skipped 000125 1773233290.849131 write 0x02957f thru 0x02dacc (0x454e bytes) 000127 1773233290.868798 read 0x02397f thru 0x0293c3 (0x5a45 bytes) 000132 1773233290.875139 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dacd): Mirror 3: 0xaaab85ef (stale) Mirror 4: 0xaaab85ef (stale) Mirror 5: 0x37678af0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dacd): Mirror 3: 0xaaab85ef (stale) Mirror 4: 0xaaab85ef (stale) Mirror 5: 0x37678af0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dacd: skipped 000133 1773233290.973202 write 0x02d53c thru 0x036f84 (0x9a49 bytes) 000134 1773233290.975261 read 0x026cbe thru 0x02e16c (0x74af bytes) 000136 1773233290.978694 mapread 0x01e4ae thru 0x02bda6 (0xd8f9 bytes) 000139 1773233290.983989 write 0x0034bb thru 0x0124c7 (0xf00d bytes) 000140 1773233290.988321 write 0x029a10 thru 0x031a1e (0x800f bytes) 000141 1773233290.989149 read 0x0351e5 thru 0x036f84 (0x1da0 bytes) 000142 1773233290.991017 MIRROR_RESYNC 4 mirrors 000145 1773233291.245822 read 0x00ae91 thru 0x01759d (0xc70d bytes) 000146 1773233291.248800 write 0x03ae31 thru 0x03ffff (0x51cf bytes) 000147 1773233291.264966 read 0x02918f thru 0x0378f8 (0xe76a bytes) 000150 1773233291.271766 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2137e571 (stale) Mirror 4: 0x2137e571 (stale) Mirror 5: 0x2137e571 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b7d30b Mirror 6: 0x8b7d30b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000151 1773233291.534946 read 0x020de9 thru 0x02c765 (0xb97d bytes) 000153 1773233291.538023 mapwrite 0x0389ae thru 0x03ffff (0x7652 bytes) 000154 1773233291.592858 mapwrite 0x013e17 thru 0x01480b (0x9f5 bytes) 000156 1773233291.607564 mapwrite 0x01aa26 thru 0x022af5 (0x80d0 bytes) 000157 1773233291.625465 write 0x02bc3c thru 0x02e387 (0x274c bytes) 000158 1773233291.630982 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2137e571 (stale) Mirror 5: 0x2137e571 (stale) Mirror 6: 0x8b7d30b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2137e571 (stale) Mirror 5: 0x2137e571 (stale) Mirror 6: 0x8b7d30b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000159 1773233291.769303 mapwrite 0x022433 thru 0x02561a (0x31e8 bytes) 000162 1773233291.792234 write 0x00dc19 thru 0x018124 (0xa50c bytes) 000164 1773233291.799655 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2137e571 (stale) Mirror 5: 0x2137e571 (stale) Mirror 6: 0x8b7d30b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a287998 Mirror 7: 0x8a287998 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000165 1773233292.050309 mapwrite 0x00acbd thru 0x0123dc (0x7720 bytes) 000166 1773233292.107776 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2137e571 (stale) Mirror 5: 0x2137e571 (stale) Mirror 6: 0x8b7d30b (stale) Mirror 7: 0x8a287998 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa7333fe Mirror 8: 0xa7333fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000167 1773233292.387034 write 0x0302f1 thru 0x03c1f0 (0xbf00 bytes) 000168 1773233292.411619 mapwrite 0x027e3d thru 0x034f0b (0xd0cf bytes) 000169 1773233292.434032 MIRROR_RESYNC 6 mirrors 000171 1773233292.713759 mapwrite 0x009163 thru 0x00adee (0x1c8c bytes) 000174 1773233292.762276 mapread 0x001a5f thru 0x003392 (0x1934 bytes) 000175 1773233292.765431 mapwrite 0x00309c thru 0x00dd70 (0xacd5 bytes) 000176 1773233292.781712 trunc from 0x040000 to 0x03d968 (0x2697 bytes) 000178 1773233292.793440 write 0x00b359 thru 0x0126ae (0x7356 bytes) 000179 1773233292.794465 mapwrite 0x00797e thru 0x00e685 (0x6d08 bytes) 000181 1773233292.812559 mapwrite 0x027ea4 thru 0x02d3e8 (0x5545 bytes) 000182 1773233292.832638 trunc from 0x03d969 to 0x03cb3c (0xe2c bytes) 000185 1773233292.839154 mapwrite 0x01df81 thru 0x02996d (0xb9ed bytes) 000186 1773233292.859640 mapwrite 0x0197ac thru 0x0244d6 (0xad2b bytes) 000188 1773233292.876055 trunc from 0x03cb3d to 0x020654 (0x1c4e8 bytes) 000190 1773233292.881723 trunc from 0x020655 to 0x03448d (0x13e39 bytes) 000193 1773233292.886675 trunc from 0x03448e to 0x0197fc (0x1ac91 bytes) 000194 1773233292.890988 read 0x00e315 thru 0x0142a5 (0x5f91 bytes) 000197 1773233292.893980 trunc from 0x0197fd to 0x01906d (0x78f bytes) 000200 1773233292.899628 read 0x0098d6 thru 0x015e3e (0xc569 bytes) 000201 1773233292.905175 write 0x021fa7 thru 0x025def (0x3e49 bytes) 000202 1773233292.906097 write 0x023a72 thru 0x030bad (0xd13c bytes) 000203 1773233292.908579 MIRROR_RESYNC 6 mirrors 000204 1773233293.190913 read 0x01bcd4 thru 0x029e3b (0xe168 bytes) 000208 1773233293.194579 trunc from 0x030bae to 0x038969 (0x7dbc bytes) 000209 1773233293.214048 read 0x02183c thru 0x02baaf (0xa274 bytes) 000212 1773233293.218546 write 0x016101 thru 0x016297 (0x197 bytes) 000213 1773233293.219402 mapwrite 0x0194c7 thru 0x028554 (0xf08e bytes) 000214 1773233293.234323 read 0x016278 thru 0x021d75 (0xbafe bytes) 000216 1773233293.235222 mapread 0x00fd86 thru 0x0160e6 (0x6361 bytes) 000219 1773233293.236580 mapwrite 0x02f845 thru 0x036de0 (0x759c bytes) 000221 1773233293.250081 mapread 0x0238bb thru 0x030474 (0xcbba bytes) 000222 1773233293.250939 read 0x0225c9 thru 0x03160d (0xf045 bytes) 000224 1773233293.251671 write 0x02c761 thru 0x03835e (0xbbfe bytes) 000225 1773233293.252679 write 0x029b4f thru 0x038acb (0xef7d bytes) 000226 1773233293.253886 read 0x019eb4 thru 0x02543a (0xb587 bytes) 000227 1773233293.254459 write 0x003205 thru 0x00bb4e (0x894a bytes) 000228 1773233293.255314 read 0x01e6f6 thru 0x02264a (0x3f55 bytes) 000229 1773233293.257667 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: 0xfbb71b30 (stale) Mirror 5: 0xfbb71b30 (stale) Mirror 6: 0xfbb71b30 (stale) Mirror 7: 0xfbb71b30 (stale) Mirror 8: 0xfbb71b30 (stale) CRC-32 checksum value for chunk [0, 0x38acc): Mirror 1: 0xcc9c133 Mirror 9: 0xcc9c133 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38acc: skipped 000231 1773233293.533909 mapread 0x0355b2 thru 0x038acb (0x351a bytes) 000232 1773233293.542890 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: 0xfbb71b30 (stale) Mirror 5: 0xfbb71b30 (stale) Mirror 6: 0xfbb71b30 (stale) Mirror 7: 0xfbb71b30 (stale) Mirror 8: 0xfbb71b30 (stale) CRC-32 checksum value for chunk [0, 0x38acc): Mirror 1: 0xcc9c133 Mirror 9: 0xcc9c133 Mirror 10: 0xcc9c133 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38acc: skipped 000234 1773233293.806119 mapread 0x036de8 thru 0x038acb (0x1ce4 bytes) 000235 1773233293.815933 read 0x0065ad thru 0x00b309 (0x4d5d bytes) 000239 1773233293.825362 read 0x00fb12 thru 0x01e6ee (0xebdd bytes) 000240 1773233293.830524 mapwrite 0x0307ab thru 0x034e85 (0x46db bytes) 000242 1773233293.890966 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: 0xfbb71b30 (stale) Mirror 6: 0xfbb71b30 (stale) Mirror 7: 0xfbb71b30 (stale) Mirror 8: 0xfbb71b30 (stale) Mirror 9: 0xcc9c133 (stale) Mirror 10: 0xcc9c133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38acc): Mirror 3: 0xfbb71b30 (stale) Mirror 6: 0xfbb71b30 (stale) Mirror 7: 0xfbb71b30 (stale) Mirror 8: 0xfbb71b30 (stale) Mirror 9: 0xcc9c133 (stale) Mirror 10: 0xcc9c133 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38acc: skipped 000245 1773233294.035417 mapread 0x02c649 thru 0x02dfd8 (0x1990 bytes) 000246 1773233294.039355 mapread 0x004ca3 thru 0x0053a4 (0x702 bytes) 000247 1773233294.042640 read 0x01eb84 thru 0x023306 (0x4783 bytes) 000248 1773233294.046533 mapwrite 0x03d4c2 thru 0x03ffff (0x2b3e bytes) 000249 1773233294.073352 read 0x003a0a thru 0x0061eb (0x27e2 bytes) 000250 1773233294.073951 mapwrite 0x031f76 thru 0x03281a (0x8a5 bytes) 000252 1773233294.091278 MIRROR_RESYNC 7 mirrors 000253 1773233294.401516 read 0x00dd73 thru 0x0194fb (0xb789 bytes) 000255 1773233294.404940 trunc from 0x040000 to 0x030f66 (0xf099 bytes) 000256 1773233294.431096 trunc from 0x030f67 to 0x010c7f (0x202e7 bytes) 000257 1773233294.437017 mapwrite 0x027e6c thru 0x02dcca (0x5e5f bytes) 000259 1773233294.460702 mapread 0x0040f6 thru 0x010ed7 (0xcde2 bytes) 000260 1773233294.464094 write 0x02c681 thru 0x02e36a (0x1cea bytes) 000261 1773233294.464901 write 0x025cfa thru 0x027f90 (0x2297 bytes) 000264 1773233294.466130 read 0x006507 thru 0x00c031 (0x5b2b bytes) 000265 1773233294.466498 mapread 0x0188da thru 0x021828 (0x8f4f bytes) 000266 1773233294.470018 read 0x001eac thru 0x007a83 (0x5bd8 bytes) 000269 1773233294.471167 trunc from 0x02e36b to 0x0394bc (0xb152 bytes) 000271 1773233294.477067 trunc from 0x0394bd to 0x00ca82 (0x2ca3a bytes) 000272 1773233294.482642 read 0x004a87 thru 0x00ca82 (0x7ffc bytes) 000274 1773233294.484820 MIRROR_RESYNC 7 mirrors 000276 1773233294.786483 write 0x0220a3 thru 0x0306c7 (0xe625 bytes) 000277 1773233294.804593 read 0x01e05c thru 0x02239c (0x4341 bytes) 000278 1773233294.807044 write 0x02fc08 thru 0x037849 (0x7c42 bytes) 000279 1773233294.808207 read 0x0251f4 thru 0x02b970 (0x677d bytes) 000284 1773233294.810271 read 0x017db2 thru 0x0182b8 (0x507 bytes) 000285 1773233294.816043 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3784a): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3784a): Mirror 3: 0xa86879b4 (stale) Mirror 6: 0xa86879b4 (stale) Mirror 7: 0xa86879b4 (stale) Mirror 8: 0xa86879b4 (stale) Mirror 9: 0xa86879b4 (stale) Mirror 10: 0xa86879b4 (stale) CRC-32 checksum value for chunk [0, 0x3784a): Mirror 1: 0x966ec191 Mirror 11: 0x966ec191 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3784a: skipped 000287 1773233295.096290 write 0x02fc23 thru 0x0337c2 (0x3ba0 bytes) 000289 1773233295.120520 trunc from 0x03784a to 0x002f0c (0x3493d bytes) 000290 1773233295.132804 write 0x01d566 thru 0x023856 (0x62f1 bytes) 000293 1773233295.135355 mapread 0x021752 thru 0x023856 (0x2105 bytes) 000294 1773233295.136445 write 0x030ba2 thru 0x034b28 (0x3f87 bytes) 000297 1773233295.138789 mapread 0x0214dc thru 0x025ae9 (0x460e bytes) 000298 1773233295.146357 mapread 0x01cee4 thru 0x02ca7a (0xfb97 bytes) 000300 1773233295.151339 trunc from 0x034b29 to 0x034c08 (0xe0 bytes) 000301 1773233295.156946 read 0x017def thru 0x0244e1 (0xc6f3 bytes) 000302 1773233295.157493 mapwrite 0x00ef6d thru 0x0185ad (0x9641 bytes) 000303 1773233295.178078 read 0x010413 thru 0x01598e (0x557c bytes) 000304 1773233295.180405 MIRROR_RESYNC 8 mirrors 000308 1773233295.530503 read 0x01f7d7 thru 0x02d216 (0xda40 bytes) 000312 1773233295.535983 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000313 1773233295.687376 MIRROR_RESYNC 7 mirrors 000314 1773233295.889146 read 0x0288a0 thru 0x034c08 (0xc369 bytes) 000315 1773233295.892480 mapwrite 0x0022c3 thru 0x006d32 (0x4a70 bytes) 000317 1773233295.952753 trunc from 0x034c09 to 0x016665 (0x1e5a3 bytes) 000318 1773233295.966188 write 0x01a890 thru 0x029084 (0xe7f5 bytes) 000319 1773233295.967937 write 0x01387d thru 0x01bbc5 (0x8349 bytes) 000320 1773233295.972332 trunc from 0x029085 to 0x03409d (0xb019 bytes) 000322 1773233295.979321 mapread 0x0032d3 thru 0x011820 (0xe54e bytes) 000323 1773233295.980767 write 0x02e41e thru 0x0312de (0x2ec1 bytes) 000324 1773233295.987683 MIRROR_RESYNC 7 mirrors 000325 1773233296.398390 write 0x03ec98 thru 0x03f13e (0x4a7 bytes) 000326 1773233296.420744 mapwrite 0x020095 thru 0x023b76 (0x3ae2 bytes) 000327 1773233296.442697 write 0x0268eb thru 0x02b54e (0x4c64 bytes) 000328 1773233296.445620 mapread 0x001043 thru 0x004241 (0x31ff bytes) 000329 1773233296.450936 write 0x002a57 thru 0x00ff44 (0xd4ee bytes) 000330 1773233296.452077 trunc from 0x03f13f to 0x038bbe (0x6580 bytes) 000332 1773233296.459855 trunc from 0x038bbf to 0x03f6bf (0x6b01 bytes) 000333 1773233296.465236 mapread 0x027d44 thru 0x0286a5 (0x962 bytes) 000339 1773233296.468274 mapwrite 0x01487f thru 0x020304 (0xba86 bytes) 000341 1773233296.492541 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f6c0): Mirror 3: 0xb25f3d4c (stale) Mirror 6: 0xb25f3d4c (stale) Mirror 7: 0xb25f3d4c (stale) Mirror 8: 0xb25f3d4c (stale) Mirror 10: 0xb25f3d4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6c0): Mirror 3: 0xb25f3d4c (stale) Mirror 6: 0xb25f3d4c (stale) Mirror 7: 0xb25f3d4c (stale) Mirror 8: 0xb25f3d4c (stale) Mirror 10: 0xb25f3d4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6c0: skipped 000343 1773233296.616284 mapread 0x01ba09 thru 0x02191c (0x5f14 bytes) 000344 1773233296.628416 write 0x03b340 thru 0x03ffff (0x4cc0 bytes) 000345 1773233296.632810 read 0x020a00 thru 0x02cd27 (0xc328 bytes) 000346 1773233296.637335 read 0x03b668 thru 0x03ffff (0x4998 bytes) 000347 1773233296.638091 write 0x00b858 thru 0x00d88a (0x2033 bytes) 000348 1773233296.642719 write 0x03961b thru 0x03ffff (0x69e5 bytes) 000349 1773233296.646411 MIRROR_RESYNC 6 mirrors 000350 1773233296.982927 write 0x0180a4 thru 0x0261ef (0xe14c bytes) 000352 1773233297.008791 MIRROR_RESYNC 6 mirrors 000357 1773233297.326062 mapread 0x033b96 thru 0x03f29d (0xb708 bytes) 000358 1773233297.335191 mapwrite 0x029edb thru 0x02e400 (0x4526 bytes) 000359 1773233297.393899 write 0x01c463 thru 0x027d16 (0xb8b4 bytes) 000360 1773233297.400969 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd03dfadb (stale) Mirror 7: 0xd03dfadb (stale) Mirror 8: 0xd03dfadb (stale) Mirror 10: 0xd03dfadb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd03dfadb (stale) Mirror 7: 0xd03dfadb (stale) Mirror 8: 0xd03dfadb (stale) Mirror 10: 0xd03dfadb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000361 1773233297.546678 write 0x025fb0 thru 0x035410 (0xf461 bytes) 000362 1773233297.551984 write 0x03d32e thru 0x03ffff (0x2cd2 bytes) 000367 1773233297.558933 trunc from 0x040000 to 0x01c07c (0x23f83 bytes) 000370 1773233297.566176 read 0x019ac4 thru 0x01c07c (0x25b9 bytes) 000371 1773233297.568936 trunc from 0x01c07d to 0x00e0fd (0xdf7f bytes) 000372 1773233297.574266 write 0x01c626 thru 0x020aa1 (0x447c bytes) 000373 1773233297.575594 read 0x018c13 thru 0x020aa1 (0x7e8f bytes) 000374 1773233297.580452 mapwrite 0x0131ea thru 0x01e357 (0xb16e bytes) 000375 1773233297.601730 read 0x0026a9 thru 0x009a52 (0x73aa bytes) 000376 1773233297.602507 trunc from 0x020aa2 to 0x03ea26 (0x1df85 bytes) 000377 1773233297.607937 mapwrite 0x01173a thru 0x01678b (0x5052 bytes) 000379 1773233297.625385 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea27): Mirror 7: 0xef0c1e6d (stale) Mirror 8: 0xef0c1e6d (stale) Mirror 10: 0xef0c1e6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea27): Mirror 7: 0xef0c1e6d (stale) Mirror 8: 0xef0c1e6d (stale) Mirror 10: 0xef0c1e6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea27: skipped 000380 1773233297.741817 read 0x00e1bc thru 0x01c848 (0xe68d bytes) 000381 1773233297.745881 trunc from 0x03ea27 to 0x026619 (0x1840d bytes) 000382 1773233297.753793 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2661a): Mirror 7: 0x7be73793 (stale) Mirror 8: 0x7be73793 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2661a): Mirror 7: 0x7be73793 (stale) Mirror 8: 0x7be73793 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2661a: skipped 000383 1773233297.845039 read 0x00793e thru 0x00b72c (0x3def bytes) 000384 1773233297.848426 mapwrite 0x031c3a thru 0x035ec6 (0x428d bytes) 000388 1773233297.876925 trunc from 0x035ec7 to 0x00cb81 (0x29345 bytes) 000389 1773233297.883359 mapwrite 0x03805d thru 0x03a8fc (0x28a0 bytes) 000390 1773233297.905462 read 0x007a8f thru 0x012210 (0xa782 bytes) 000392 1773233297.912681 MIRROR_RESYNC 3 mirrors 000393 1773233298.157246 read 0x03397c thru 0x03a8fc (0x6f81 bytes) 000394 1773233298.161025 read 0x037324 thru 0x037958 (0x635 bytes) 000398 1773233298.162306 mapread 0x013779 thru 0x01fd16 (0xc59e bytes) 000400 1773233298.167564 mapread 0x0184f7 thru 0x0241d6 (0xbce0 bytes) 000401 1773233298.168320 trunc from 0x03a8fd to 0x028b74 (0x11d88 bytes) 000403 1773233298.193790 read 0x005afc thru 0x0146c6 (0xebcb bytes) 000405 1773233298.199047 mapread 0x00bb0f thru 0x00eb1c (0x300e bytes) 000406 1773233298.200064 trunc from 0x028b75 to 0x00783c (0x21338 bytes) 000407 1773233298.209195 trunc from 0x00783d to 0x032854 (0x2b018 bytes) 000411 1773233298.217376 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32855): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32855): Mirror 7: 0x703860b0 (stale) Mirror 8: 0x703860b0 (stale) CRC-32 checksum value for chunk [0, 0x32855): Mirror 1: 0x27f5664a Mirror 9: 0x27f5664a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32855: skipped 000412 1773233298.494221 mapread 0x0031eb thru 0x00c2b6 (0x90cc bytes) 000413 1773233298.509868 mapwrite 0x02cc31 thru 0x032c64 (0x6034 bytes) 000414 1773233298.582785 MIRROR_RESYNC 4 mirrors 000416 1773233298.929424 write 0x017db8 thru 0x01a199 (0x23e2 bytes) 000417 1773233298.961337 trunc from 0x032c65 to 0x02bef2 (0x6d72 bytes) 000419 1773233298.975383 mapread 0x01b9a2 thru 0x029607 (0xdc66 bytes) 000420 1773233298.988536 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bef3): Mirror 8: 0xe53a912e (stale) Mirror 9: 0xe53a912e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bef3): Mirror 8: 0xe53a912e (stale) Mirror 9: 0xe53a912e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bef3: skipped 000423 1773233299.116996 write 0x02b009 thru 0x0321f3 (0x71eb bytes) 000425 1773233299.126133 trunc from 0x0321f4 to 0x01b0c4 (0x1712f bytes) 000426 1773233299.136951 write 0x018a77 thru 0x0206ce (0x7c58 bytes) 000427 1773233299.149697 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206cf): Mirror 9: 0x5c2e43df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206cf): Mirror 9: 0x5c2e43df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206cf: skipped 000429 1773233299.267964 mapread 0x00c8f9 thru 0x01b3c8 (0xead0 bytes) 000431 1773233299.279958 write 0x018a29 thru 0x019d0f (0x12e7 bytes) 000433 1773233299.281258 mapread 0x01a985 thru 0x0206ce (0x5d4a bytes) 000436 1773233299.285568 trunc from 0x0206cf to 0x019118 (0x75b6 bytes) 000438 1773233299.293086 trunc from 0x019119 to 0x011630 (0x7ae8 bytes) 000439 1773233299.299154 mapwrite 0x02a142 thru 0x02e62c (0x44eb bytes) 000440 1773233299.323776 read 0x0246ec thru 0x02e62c (0x9f41 bytes) 000442 1773233299.324776 mapread 0x02b2b0 thru 0x02e62c (0x337d bytes) 000443 1773233299.325610 mapread 0x012671 thru 0x01b972 (0x9302 bytes) 000444 1773233299.332425 mapread 0x018123 thru 0x0248de (0xc7bc bytes) 000446 1773233299.336484 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e62d): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e62d): Mirror 9: 0xdf199bde (stale) CRC-32 checksum value for chunk [0, 0x2e62d): Mirror 1: 0xde028d6e Mirror 10: 0xde028d6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e62d: skipped 000447 1773233299.669417 trunc from 0x02e62d to 0x032f20 (0x48f4 bytes) 000448 1773233299.723788 read 0x00312c thru 0x005d2b (0x2c00 bytes) 000449 1773233299.729274 MIRROR_RESYNC 3 mirrors 000450 1773233299.978507 read 0x026668 thru 0x02d7d1 (0x716a bytes) 000452 1773233299.983587 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000453 1773233300.123670 read 0x0235e2 thru 0x027543 (0x3f62 bytes) 000454 1773233300.129054 mapread 0x017ccb thru 0x0255a0 (0xd8d6 bytes) 000455 1773233300.142061 trunc from 0x032f21 to 0x02e430 (0x4af0 bytes) 000456 1773233300.171339 trunc from 0x02e431 to 0x039896 (0xb466 bytes) 000459 1773233300.180483 read 0x00af2e thru 0x01194f (0x6a22 bytes) 000462 1773233300.188042 read 0x00bb9d thru 0x00c1ff (0x663 bytes) 000463 1773233300.191493 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39897): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39897): Mirror 10: 0xed06dfef (stale) CRC-32 checksum value for chunk [0, 0x39897): Mirror 1: 0xc351daa1 Mirror 11: 0xc351daa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39897: skipped 000464 1773233300.550495 mapwrite 0x03c774 thru 0x03f162 (0x29ef bytes) 000465 1773233300.633607 write 0x00254c thru 0x00b401 (0x8eb6 bytes) 000466 1773233300.638283 trunc from 0x03f163 to 0x0123ec (0x2cd76 bytes) 000467 1773233300.646112 write 0x007b06 thru 0x00b487 (0x3982 bytes) 000469 1773233300.647549 mapwrite 0x01672c thru 0x02477c (0xe051 bytes) 000470 1773233300.682631 trunc from 0x02477d to 0x035d63 (0x115e7 bytes) 000471 1773233300.688328 mapwrite 0x02e9be thru 0x03518d (0x67d0 bytes) 000472 1773233300.708376 read 0x02733c thru 0x02da54 (0x6719 bytes) 000475 1773233300.710135 trunc from 0x035d64 to 0x0179d9 (0x1e38a bytes) 000476 1773233300.720109 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x179da): Mirror 11: 0xc8dc3db0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179da): Mirror 11: 0xc8dc3db0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179da: skipped 000477 1773233300.824965 trunc from 0x0179da to 0x0117f7 (0x61e2 bytes) 000479 1773233300.831294 mapwrite 0x00ba75 thru 0x0163fa (0xa986 bytes) 000480 1773233300.868821 read 0x0034d1 thru 0x007942 (0x4472 bytes) 000481 1773233300.872216 trunc from 0x0163fb to 0x01a294 (0x3e9a bytes) 000482 1773233300.879004 write 0x00bb33 thru 0x00e6ef (0x2bbd bytes) 000486 1773233300.881193 mapwrite 0x014479 thru 0x01e65b (0xa1e3 bytes) 000492 1773233300.912844 mapwrite 0x0026ba thru 0x010a72 (0xe3b9 bytes) 000494 1773233300.941430 trunc from 0x01e65c to 0x03d608 (0x1efad bytes) 000496 1773233300.950325 mapread 0x00eed7 thru 0x010d3e (0x1e68 bytes) 000500 1773233300.952645 mapwrite 0x035525 thru 0x03b0e1 (0x5bbd bytes) 000501 1773233300.976302 read 0x016eda thru 0x01a1f8 (0x331f bytes) 000502 1773233300.977044 mapread 0x009e5a thru 0x018e75 (0xf01c bytes) 000503 1773233300.978110 trunc from 0x03d609 to 0x02c0ed (0x1151b bytes) 000505 1773233300.987261 read 0x002278 thru 0x00e3f3 (0xc17c bytes) 000508 1773233300.990021 write 0x03b47c thru 0x03ffff (0x4b84 bytes) 000509 1773233300.991163 read 0x03807d thru 0x03ffff (0x7f83 bytes) 000511 1773233300.999155 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000512 1773233301.058229 mapwrite 0x00fac6 thru 0x01a0f7 (0xa632 bytes) 000513 1773233301.089144 mapwrite 0x0336be thru 0x039531 (0x5e74 bytes) 000516 1773233301.112093 trunc from 0x040000 to 0x024d66 (0x1b299 bytes) 000517 1773233301.119783 write 0x0347a8 thru 0x039283 (0x4adc bytes) 000518 1773233301.121055 write 0x02e674 thru 0x0340f7 (0x5a84 bytes) 000520 1773233301.125449 mapread 0x034b56 thru 0x0386f0 (0x3b9b bytes) 000522 1773233301.126702 write 0x02d82a thru 0x037bca (0xa3a1 bytes) 000527 1773233301.129411 mapread 0x0070cb thru 0x00bd3f (0x4c75 bytes) 000530 1773233301.134055 trunc from 0x039284 to 0x03a252 (0xfcf bytes) 000534 1773233301.140968 trunc from 0x03a253 to 0x035dff (0x4453 bytes) 000535 1773233301.146668 mapread 0x00832c thru 0x00de11 (0x5ae6 bytes) 000537 1773233301.149513 read 0x010ae2 thru 0x020888 (0xfda7 bytes) 000539 1773233301.150453 write 0x00b06f thru 0x01265c (0x75ee bytes) 000540 1773233301.151467 mapread 0x001da4 thru 0x010ca9 (0xef06 bytes) 000541 1773233301.152675 trunc from 0x035e00 to 0x02cfe4 (0x8e1b bytes) 000545 1773233301.161420 write 0x03e572 thru 0x03ffff (0x1a8e bytes) 000546 1773233301.162303 mapread 0x02c80e thru 0x038483 (0xbc76 bytes) 000547 1773233301.171280 mapread 0x020bf7 thru 0x02156c (0x976 bytes) 000550 1773233301.173517 mapwrite 0x019a58 thru 0x01c345 (0x28ee bytes) 000552 1773233301.204391 read 0x028aaf thru 0x035338 (0xc88a bytes) 000553 1773233301.204946 mapread 0x01b9cb thru 0x027e31 (0xc467 bytes) 000561 1773233301.216468 mapread 0x0067a7 thru 0x0075fa (0xe54 bytes) 000562 1773233301.217728 trunc from 0x040000 to 0x038f68 (0x7097 bytes) 000567 1773233301.228487 trunc from 0x038f69 to 0x00e3fc (0x2ab6c bytes) 000568 1773233301.246160 mapread 0x00afae thru 0x00e3fc (0x344f bytes) 000569 1773233301.247169 mapwrite 0x0154fa thru 0x01c271 (0x6d78 bytes) 000571 1773233301.323182 read 0x0120b0 thru 0x01c271 (0xa1c2 bytes) 000572 1773233301.327188 read 0x017848 thru 0x01bee7 (0x46a0 bytes) 000573 1773233301.327799 read 0x003cec thru 0x00adbd (0x70d2 bytes) 000575 1773233301.329015 read 0x00de8b thru 0x00e328 (0x49e bytes) 000576 1773233301.329603 write 0x021cf0 thru 0x030de9 (0xf0fa bytes) 000577 1773233301.331137 mapwrite 0x00e9ab thru 0x011b9c (0x31f2 bytes) 000578 1773233301.354921 MIRROR_EXTEND to 2 mirrors 000579 1773233301.543537 write 0x032845 thru 0x0365d8 (0x3d94 bytes) 000580 1773233301.566098 MIRROR_RESYNC 2 mirrors 000586 1773233301.785988 trunc from 0x0365d9 to 0x0172f1 (0x1f2e7 bytes) 000587 1773233301.822498 mapread 0x010618 thru 0x015a53 (0x543c bytes) 000588 1773233301.831453 mapwrite 0x03e3f5 thru 0x03ffff (0x1c0b bytes) 000589 1773233301.862703 trunc from 0x040000 to 0x0096d9 (0x36926 bytes) 000590 1773233301.868580 mapwrite 0x01d6cc thru 0x025049 (0x797e bytes) 000592 1773233301.893946 write 0x01dfea thru 0x0260ab (0x80c2 bytes) 000594 1773233301.895095 mapwrite 0x0317ab thru 0x03ffff (0xe855 bytes) 000595 1773233301.921342 write 0x0000e9 thru 0x001de0 (0x1cf8 bytes) 000597 1773233301.925519 write 0x00b37d thru 0x0192ed (0xdf71 bytes) 000598 1773233301.926981 read 0x0326be thru 0x032971 (0x2b4 bytes) 000599 1773233301.927633 trunc from 0x040000 to 0x014b65 (0x2b49a bytes) 000602 1773233301.936715 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000603 1773233301.999168 mapwrite 0x03b615 thru 0x03ffff (0x49eb bytes) 000605 1773233302.027282 read 0x022e12 thru 0x02601c (0x320b bytes) 000606 1773233302.031520 mapread 0x026402 thru 0x031d1d (0xb91c bytes) 000609 1773233302.041086 write 0x02fc48 thru 0x035213 (0x55cc bytes) 000612 1773233302.042967 read 0x034cf1 thru 0x03ffff (0xb30f bytes) 000613 1773233302.043477 mapwrite 0x000c6e thru 0x00bea4 (0xb237 bytes) 000616 1773233302.082431 write 0x00fbca thru 0x0180c7 (0x84fe bytes) 000618 1773233302.084568 write 0x017161 thru 0x020328 (0x91c8 bytes) 000619 1773233302.085785 read 0x02c3c2 thru 0x02f271 (0x2eb0 bytes) 000620 1773233302.086408 mapwrite 0x006c51 thru 0x013682 (0xca32 bytes) 000621 1773233302.107288 write 0x0019cb thru 0x00eea6 (0xd4dc bytes) 000622 1773233302.108626 trunc from 0x040000 to 0x012212 (0x2dded bytes) 000623 1773233302.114941 write 0x00cc0a thru 0x0135e8 (0x69df bytes) 000630 1773233302.119901 mapread 0x010d80 thru 0x0135e8 (0x2869 bytes) 000631 1773233302.120958 mapread 0x01283b thru 0x0135e8 (0xdae bytes) 000635 1773233302.123915 read 0x0014ea thru 0x01006a (0xeb81 bytes) 000636 1773233302.124581 write 0x01d5d2 thru 0x02549f (0x7ece bytes) 000638 1773233302.126387 read 0x01f78b thru 0x02549f (0x5d15 bytes) 000639 1773233302.127010 trunc from 0x0254a0 to 0x03160a (0xc16b bytes) 000640 1773233302.132713 read 0x02c8bb thru 0x03160a (0x4d50 bytes) 000641 1773233302.136763 mapwrite 0x02c615 thru 0x034429 (0x7e15 bytes) 000643 1773233302.161940 read 0x0315b2 thru 0x034429 (0x2e78 bytes) 000644 1773233302.162488 write 0x03f796 thru 0x03ffff (0x86a bytes) 000645 1773233302.163217 mapwrite 0x026acd thru 0x028c01 (0x2135 bytes) 000646 1773233302.178690 mapread 0x03a335 thru 0x03f29c (0x4f68 bytes) 000647 1773233302.183392 read 0x0323ca thru 0x039a51 (0x7688 bytes) 000648 1773233302.184136 mapread 0x00c019 thru 0x0195fe (0xd5e6 bytes) 000650 1773233302.186053 mapwrite 0x027ddd thru 0x02a0e0 (0x2304 bytes) 000651 1773233302.201798 write 0x03c352 thru 0x03ffff (0x3cae bytes) 000657 1773233302.205605 mapread 0x018bd3 thru 0x025d74 (0xd1a2 bytes) 000658 1773233302.206843 mapwrite 0x02d899 thru 0x037f06 (0xa66e bytes) 000659 1773233302.228308 mapread 0x00317b thru 0x0055ac (0x2432 bytes) 000660 1773233302.229114 trunc from 0x040000 to 0x012b34 (0x2d4cb bytes) 000661 1773233302.234773 read 0x0032b8 thru 0x012b34 (0xf87d bytes) 000663 1773233302.235797 mapwrite 0x021465 thru 0x021594 (0x130 bytes) 000667 1773233302.262114 trunc from 0x021595 to 0x00a14b (0x17449 bytes) 000668 1773233302.267189 trunc from 0x00a14c to 0x0362ba (0x2c16f bytes) 000672 1773233302.273268 read 0x026c97 thru 0x02af4e (0x42b8 bytes) 000675 1773233302.278321 trunc from 0x0362bb to 0x02cc28 (0x9692 bytes) 000676 1773233302.284188 read 0x00e300 thru 0x00fac4 (0x17c5 bytes) 000677 1773233302.284881 mapwrite 0x012d69 thru 0x01f3ff (0xc697 bytes) 000678 1773233302.307796 mapread 0x01cfeb thru 0x02761e (0xa634 bytes) 000679 1773233302.308663 write 0x013f97 thru 0x017e12 (0x3e7c bytes) 000680 1773233302.309394 mapread 0x00ff15 thru 0x019a2f (0x9b1b bytes) 000681 1773233302.310491 read 0x01d478 thru 0x02cc28 (0xf7b1 bytes) 000682 1773233302.313547 MIRROR_EXTEND to 2 mirrors 000684 1773233302.532070 trunc from 0x02cc29 to 0x03d711 (0x10ae9 bytes) 000685 1773233302.562877 mapwrite 0x030102 thru 0x03a02b (0x9f2a bytes) 000686 1773233302.587394 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d712): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d712): Mirror 2: 0xf1d099c8 (stale) CRC-32 checksum value for chunk [0, 0x3d712): Mirror 1: 0xc62d36a8 Mirror 3: 0xc62d36a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d712: skipped 000687 1773233302.847393 trunc from 0x03d712 to 0x016152 (0x275bf bytes) 000688 1773233302.873456 write 0x016628 thru 0x016bc2 (0x59b bytes) 000690 1773233302.876280 mapread 0x00e4f5 thru 0x016bc2 (0x86ce bytes) 000695 1773233302.884016 mapwrite 0x01a74a thru 0x01aaa0 (0x357 bytes) 000697 1773233302.906985 read 0x00db65 thru 0x00f287 (0x1723 bytes) 000699 1773233302.908289 read 0x017715 thru 0x01aaa0 (0x338c bytes) 000704 1773233302.910890 write 0x03727f thru 0x03eb1d (0x789f bytes) 000706 1773233302.914036 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3eb1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb1e): Mirror 2: 0xce442e67 (stale) Mirror 3: 0xa1a99beb (stale) CRC-32 checksum value for chunk [0, 0x3eb1e): Mirror 1: 0xf40d31de Mirror 4: 0xf40d31de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb1e: skipped 000707 1773233303.233187 mapwrite 0x02096f thru 0x02440a (0x3a9c bytes) 000708 1773233303.311169 trunc from 0x03eb1e to 0x012083 (0x2ca9a bytes) 000709 1773233303.323415 trunc from 0x012084 to 0x02d036 (0x1afb3 bytes) 000711 1773233303.334303 trunc from 0x02d037 to 0x03b9c7 (0xe991 bytes) 000715 1773233303.349047 trunc from 0x03b9c8 to 0x004d2b (0x36c9c bytes) 000716 1773233303.358207 trunc from 0x004d2c to 0x00994b (0x4c20 bytes) 000717 1773233303.367688 read 0x004442 thru 0x00994b (0x550a bytes) 000718 1773233303.371267 mapwrite 0x00b9d2 thru 0x017e8e (0xc4bd bytes) 000721 1773233303.399573 mapread 0x002033 thru 0x00c8c1 (0xa88f bytes) 000722 1773233303.402145 write 0x02348e thru 0x02bc48 (0x87bb bytes) 000723 1773233303.403144 mapwrite 0x034b75 thru 0x0372fc (0x2788 bytes) 000724 1773233303.426300 mapwrite 0x01790d thru 0x01f46f (0x7b63 bytes) 000725 1773233303.447745 write 0x01620e thru 0x01b37f (0x5172 bytes) 000726 1773233303.448999 read 0x00e506 thru 0x01ad3d (0xc838 bytes) 000732 1773233303.452362 mapwrite 0x018149 thru 0x025236 (0xd0ee bytes) 000734 1773233303.480984 mapread 0x02f097 thru 0x02f5fc (0x566 bytes) 000735 1773233303.481895 mapread 0x0347f7 thru 0x0372fc (0x2b06 bytes) 000742 1773233303.486359 read 0x02e7fa thru 0x030297 (0x1a9e bytes) 000745 1773233303.488057 read 0x0317ba thru 0x034eb8 (0x36ff bytes) 000746 1773233303.488700 read 0x02d47e thru 0x0372fc (0x9e7f bytes) 000747 1773233303.489373 read 0x02a49f thru 0x033178 (0x8cda bytes) 000748 1773233303.492454 MIRROR_RESYNC 4 mirrors 000751 1773233303.813199 trunc from 0x0372fd to 0x02ecb1 (0x864b bytes) 000752 1773233303.843204 mapread 0x00339d thru 0x00c84a (0x94ae bytes) 000753 1773233303.851239 trunc from 0x02ecb2 to 0x032756 (0x3aa5 bytes) 000755 1773233303.862563 MIRROR_RESYNC 4 mirrors 000757 1773233304.160260 write 0x02a295 thru 0x02efab (0x4d17 bytes) 000758 1773233304.186916 write 0x033dc0 thru 0x035124 (0x1365 bytes) 000759 1773233304.188457 read 0x025c84 thru 0x030295 (0xa612 bytes) 000761 1773233304.195029 mapread 0x01bde7 thru 0x02acec (0xef06 bytes) 000762 1773233304.206806 trunc from 0x035125 to 0x039f1c (0x4df8 bytes) 000763 1773233304.215555 read 0x035065 thru 0x039f1c (0x4eb8 bytes) 000767 1773233304.224350 read 0x028498 thru 0x031917 (0x9480 bytes) 000769 1773233304.225790 write 0x02b627 thru 0x0372dc (0xbcb6 bytes) 000770 1773233304.227226 mapread 0x01e98e thru 0x025f85 (0x75f8 bytes) 000774 1773233304.229918 trunc from 0x039f1d to 0x034113 (0x5e09 bytes) 000775 1773233304.240713 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34114): Mirror 2: 0xd16e83f5 (stale) Mirror 4: 0xd16e83f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34114): Mirror 2: 0xd16e83f5 (stale) Mirror 4: 0xd16e83f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34114: skipped 000776 1773233304.364402 mapread 0x009ebf thru 0x00ef49 (0x508b bytes) 000777 1773233304.373088 mapwrite 0x02681e thru 0x02c8e0 (0x60c3 bytes) 000778 1773233304.395566 mapwrite 0x014f35 thru 0x0171c6 (0x2292 bytes) 000779 1773233304.418371 read 0x017bcd thru 0x01ba12 (0x3e46 bytes) 000781 1773233304.419443 read 0x02c299 thru 0x02fe9b (0x3c03 bytes) 000782 1773233304.424213 write 0x00d4fe thru 0x013d58 (0x685b bytes) 000783 1773233304.425217 mapread 0x011747 thru 0x01d322 (0xbbdc bytes) 000784 1773233304.426258 mapwrite 0x01264c thru 0x0215af (0xef64 bytes) 000785 1773233304.445350 mapwrite 0x00d976 thru 0x01aef5 (0xd580 bytes) 000786 1773233304.476075 mapwrite 0x0394b2 thru 0x03ffff (0x6b4e bytes) 000787 1773233304.511760 mapread 0x005e62 thru 0x012047 (0xc1e6 bytes) 000788 1773233304.512903 write 0x011cae thru 0x01d56a (0xb8bd bytes) 000790 1773233304.516025 write 0x018482 thru 0x019645 (0x11c4 bytes) 000792 1773233304.518009 read 0x03cd34 thru 0x03ffff (0x32cc bytes) 000797 1773233304.521193 trunc from 0x040000 to 0x02c6be (0x13941 bytes) 000799 1773233304.539297 trunc from 0x02c6bf to 0x024025 (0x8699 bytes) 000800 1773233304.545599 trunc from 0x024026 to 0x028127 (0x4102 bytes) 000802 1773233304.556520 read 0x0145c8 thru 0x01f7f1 (0xb22a bytes) 000803 1773233304.557072 write 0x00237c thru 0x011fe0 (0xfc65 bytes) 000804 1773233304.560333 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28128): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28128): Mirror 2: 0xa74384f3 (stale) Mirror 4: 0xa74384f3 (stale) CRC-32 checksum value for chunk [0, 0x28128): Mirror 1: 0x96dd1e6c Mirror 5: 0x96dd1e6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28128: skipped 000806 1773233304.859614 write 0x033473 thru 0x036b11 (0x369f bytes) 000807 1773233304.880712 read 0x02e25d thru 0x02f313 (0x10b7 bytes) 000810 1773233304.883732 mapread 0x027d70 thru 0x02b819 (0x3aaa bytes) 000811 1773233304.891240 read 0x030121 thru 0x036b11 (0x69f1 bytes) 000812 1773233304.894888 write 0x010a75 thru 0x01b98c (0xaf18 bytes) 000813 1773233304.896147 write 0x01c9a6 thru 0x025601 (0x8c5c bytes) 000815 1773233304.900016 MIRROR_RESYNC 4 mirrors 000817 1773233305.240041 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000819 1773233305.404069 mapread 0x025926 thru 0x030597 (0xac72 bytes) 000821 1773233305.421934 mapwrite 0x00aa0f thru 0x00bbee (0x11e0 bytes) 000822 1773233305.493299 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b12): Mirror 4: 0x3a94f91b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b12): Mirror 4: 0x3a94f91b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b12: skipped 000823 1773233305.617486 trunc from 0x036b12 to 0x032ac5 (0x404c bytes) 000824 1773233305.623132 mapread 0x01784c thru 0x025629 (0xddde bytes) 000827 1773233305.638246 MIRROR_RESYNC 2 mirrors 000829 1773233305.863248 mapread 0x0208f2 thru 0x02bf61 (0xb670 bytes) 000830 1773233305.879668 MIRROR_EXTEND to 3 mirrors 000831 1773233306.104919 read 0x02d062 thru 0x032194 (0x5133 bytes) 000834 1773233306.109199 write 0x037f0d thru 0x03ca09 (0x4afd bytes) 000835 1773233306.133479 mapread 0x0061fd thru 0x00ee00 (0x8c04 bytes) 000836 1773233306.141371 mapwrite 0x00f2fd thru 0x017335 (0x8039 bytes) 000837 1773233306.164544 read 0x023f6c thru 0x02b03f (0x70d4 bytes) 000838 1773233306.169244 mapwrite 0x011c19 thru 0x01bac1 (0x9ea9 bytes) 000844 1773233306.188538 trunc from 0x03ca0a to 0x02a743 (0x122c6 bytes) 000845 1773233306.201800 trunc from 0x02a744 to 0x014e8b (0x158b8 bytes) 000846 1773233306.207972 trunc from 0x014e8c to 0x02c611 (0x17786 bytes) 000847 1773233306.213423 mapread 0x00e33c thru 0x015d9c (0x7a61 bytes) 000848 1773233306.218085 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c612): Mirror 4: 0x2be631e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c612): Mirror 4: 0x2be631e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c612: skipped 000849 1773233306.312395 trunc from 0x02c612 to 0x038111 (0xbb00 bytes) 000850 1773233306.318372 read 0x00d8d0 thru 0x00eeff (0x1630 bytes) 000851 1773233306.323593 mapwrite 0x01203b thru 0x01ae48 (0x8e0e bytes) 000853 1773233306.354105 mapread 0x02ca25 thru 0x038111 (0xb6ed bytes) 000855 1773233306.367844 read 0x019c31 thru 0x01ea6d (0x4e3d bytes) 000858 1773233306.375609 write 0x036f07 thru 0x03c412 (0x550c bytes) 000861 1773233306.380745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000862 1773233306.437776 read 0x020770 thru 0x022a39 (0x22ca bytes) 000867 1773233306.443204 read 0x02bc3e thru 0x03ba0b (0xfdce bytes) 000868 1773233306.450367 mapwrite 0x009c38 thru 0x00b7b6 (0x1b7f bytes) 000869 1773233306.473903 mapwrite 0x004f59 thru 0x008feb (0x4093 bytes) 000870 1773233306.500363 mapwrite 0x01e131 thru 0x02d722 (0xf5f2 bytes) 000872 1773233306.525281 read 0x00d910 thru 0x0121ce (0x48bf bytes) 000873 1773233306.528887 MIRROR_EXTEND to 2 mirrors 000874 1773233306.750721 trunc from 0x03c413 to 0x020402 (0x1c010 bytes) 000876 1773233306.783923 MIRROR_RESYNC 2 mirrors 000877 1773233307.009902 trunc from 0x020403 to 0x004816 (0x1bbec bytes) 000878 1773233307.047581 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000879 1773233307.097693 read 0x001f60 thru 0x004816 (0x28b7 bytes) 000881 1773233307.100946 trunc from 0x004817 to 0x01002e (0xb818 bytes) 000883 1773233307.106451 trunc from 0x01002f to 0x03e2a1 (0x2e273 bytes) 000885 1773233307.112253 read 0x00ce7a thru 0x01cde5 (0xff6c bytes) 000886 1773233307.117350 mapread 0x0370a2 thru 0x03e2a1 (0x7200 bytes) 000887 1773233307.118395 trunc from 0x03e2a2 to 0x00751b (0x36d86 bytes) 000888 1773233307.124604 mapwrite 0x02759a thru 0x031002 (0x9a69 bytes) 000889 1773233307.169348 mapwrite 0x009054 thru 0x0158a0 (0xc84d bytes) 000891 1773233307.228898 trunc from 0x031003 to 0x033d61 (0x2d5f bytes) 000892 1773233307.236399 mapread 0x0338f5 thru 0x033d61 (0x46d bytes) 000896 1773233307.243407 mapwrite 0x023b82 thru 0x029754 (0x5bd3 bytes) 000897 1773233307.270898 write 0x030d8f thru 0x036b74 (0x5de6 bytes) 000898 1773233307.271981 trunc from 0x036b75 to 0x036177 (0x9fd bytes) 000899 1773233307.278095 mapwrite 0x004de3 thru 0x0143ff (0xf61d bytes) 000907 1773233307.305332 mapwrite 0x03a7b9 thru 0x03ffff (0x5847 bytes) 000908 1773233307.341320 read 0x00881f thru 0x010895 (0x8077 bytes) 000910 1773233307.342367 write 0x03aa8d thru 0x03ffff (0x5573 bytes) 000911 1773233307.343459 write 0x027473 thru 0x028b8c (0x171a bytes) 000912 1773233307.344353 mapwrite 0x03f9a4 thru 0x03ffff (0x65c bytes) 000916 1773233307.360894 mapread 0x026e15 thru 0x032430 (0xb61c bytes) 000919 1773233307.363849 mapread 0x002f66 thru 0x00bd0c (0x8da7 bytes) 000920 1773233307.367116 MIRROR_EXTEND to 2 mirrors 000925 1773233307.592097 mapread 0x035dda thru 0x03fe9d (0xa0c4 bytes) 000929 1773233307.605667 mapwrite 0x030b97 thru 0x030dab (0x215 bytes) 000930 1773233307.664459 trunc from 0x040000 to 0x00899a (0x37665 bytes) 000932 1773233307.679481 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000936 1773233307.724176 trunc from 0x00899b to 0x00fc6d (0x72d3 bytes) 000937 1773233307.729942 read 0x006919 thru 0x00d1a0 (0x6888 bytes) 000939 1773233307.734585 mapwrite 0x039c18 thru 0x03ffff (0x63e8 bytes) 000940 1773233307.765459 mapread 0x00c1de thru 0x01193d (0x5760 bytes) 000941 1773233307.769506 read 0x03e307 thru 0x03ffff (0x1cf9 bytes) 000942 1773233307.771738 MIRROR_EXTEND to 2 mirrors 000944 1773233307.968473 trunc from 0x040000 to 0x01565b (0x2a9a4 bytes) 000948 1773233308.002845 MIRROR_RESYNC 2 mirrors 000949 1773233308.238657 mapread 0x00d55c thru 0x01565b (0x8100 bytes) 000950 1773233308.248579 trunc from 0x01565c to 0x01f507 (0x9eac bytes) 000956 1773233308.277249 mapwrite 0x02a71b thru 0x02fad9 (0x53bf bytes) 000957 1773233308.314100 trunc from 0x02fada to 0x029eef (0x5bea bytes) 000958 1773233308.320914 mapwrite 0x021b6e thru 0x026d6a (0x51fd bytes) 000959 1773233308.347916 read 0x00b7cc thru 0x0122bc (0x6af1 bytes) 000960 1773233308.352989 mapread 0x010882 thru 0x013376 (0x2af5 bytes) 000965 1773233308.359763 mapwrite 0x00a415 thru 0x00c75c (0x2348 bytes) 000966 1773233308.379520 trunc from 0x029ef0 to 0x0275ed (0x2902 bytes) 000967 1773233308.386075 trunc from 0x0275ee to 0x032128 (0xab3b bytes) 000974 1773233308.396633 MIRROR_RESYNC 2 mirrors 000976 1773233308.619555 read 0x004792 thru 0x013d4c (0xf5bb bytes) 000977 1773233308.628679 write 0x01520a thru 0x01f788 (0xa57f bytes) 000978 1773233308.665576 mapwrite 0x01e446 thru 0x024e66 (0x6a21 bytes) 000979 1773233308.692514 trunc from 0x032129 to 0x0085e8 (0x29b40 bytes) 000980 1773233308.714889 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x558:0x0]: Invalid argument (22) 000981 1773233308.776777 mapwrite 0x0352f2 thru 0x03ffff (0xad0e bytes) 000983 1773233308.817784 mapread 0x03ef56 thru 0x03ffff (0x10aa bytes) 000984 1773233308.818784 write 0x005352 thru 0x00b304 (0x5fb3 bytes) 000988 1773233308.825742 read 0x008a98 thru 0x00fd71 (0x72da bytes) 000989 1773233308.826385 mapwrite 0x02aef4 thru 0x036913 (0xba20 bytes) 000990 1773233308.851493 read 0x01a0af thru 0x020c7e (0x6bd0 bytes) 000994 1773233308.858617 read 0x01c8bb thru 0x01c8f3 (0x39 bytes) 000996 1773233308.859755 write 0x035b48 thru 0x03912d (0x35e6 bytes) 000997 1773233308.860500 mapread 0x006724 thru 0x015ec0 (0xf79d bytes) 000999 1773233308.861922 trunc from 0x040000 to 0x023655 (0x1c9aa bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 71: check mirror extend parent fid ==== 08:48:30 (1773233310) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 08:48:32 (1773233312) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875513 s, 24.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569665 s, 18.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7b6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 08:48:35 (1773233315) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8221312 .. Extending file size to 6486944 .. done Extending file size to 7017888 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7333888 .. Extending file size to 8263200 .. failed Extending file size to 8584992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8585280 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8601984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8713696 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8645344 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9007392 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9170272 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9006144 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 57933 57934 57936 57938 57939 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:49:41 (1773233381) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6763712 .. Extending file size to 2475360 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5959296 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6111840 .. Extending file size to 6533344 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7932640 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7409504 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8804096 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9022496 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8817536 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8889152 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8896928 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61141 61142 61143 61146 61148 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:50:46 (1773233446) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0430853 s, 48.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0798288 s, 26.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x62c23bbd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:50:52 (1773233452) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0301523 s, 69.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 08:50:54 (1773233454) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x597:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x7cb:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:50:57 (1773233457) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0453336 s, 46.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:50:59 (1773233459) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0472662 s, 44.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cf:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:51:02 (1773233462) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0388819 s, 53.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d1:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:51:26 (1773233486) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275713 s, 38.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ce:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:52:07 (1773233527) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0501232 s, 41.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:52:54 (1773233574) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. Creating new pool test_204f oleg447-server: Pool lustre.test_204f created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0448901 s, 46.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg447-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:53:50 (1773233630) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:53:53 (1773233633) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:53:56 (1773233636) oleg447-server: Pool lustre.test_206 created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg447-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:54:10 (1773233650) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281462 s, 37.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136971 s, 76.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x59cf1f60 Mirror 3: 0x59cf1f60 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:54:13 (1773233653) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:54:14 (1773233654) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:54:15 (1773233655) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0244335 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2360000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1780000 Read this (4 bytes): BBBB 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:54:31 (1773233671) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00387609 s, 2.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:54:34 (1773233674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145273 s, 72.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5bc lmm_fid: [0x200000404:0x5bc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5bc lmm_fid: [0x200000404:0x5bc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7df:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835388 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-11 08:54:34.000000000 -0400 Modify: 2026-03-11 08:54:34.000000000 -0400 Change: 2026-03-11 08:54:35.000000000 -0400 Birth: 2026-03-11 08:54:35.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:54:36 (1773233676) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:54:37 (1773233677) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.103691 s, 395 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128996 s, 318 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 11 08:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5be:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 11 08:54 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:54:40 (1773233680) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:54:43 (1773233683) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 858 sec ============ 08:54:45 (1773233685) === sanity-flr: start cleanup 08:54:46 (1773233686) === === sanity-flr: finish cleanup 08:54:47 (1773233687) ===