-----============= acceptance-small: sanity-flr ============----- Mon Mar 16 14:24:21 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg304-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:24:34 (1773685474) === oleg304-client.virtnet: executing check_config_client /mnt/lustre oleg304-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg304-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d0804e30800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d0804e30800.idle_timeout=debug disable quota as required oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:24:59 (1773685499) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8354] == sanity-flr test 0a: lfs mirror create with -N option == 14:25:01 (1773685501) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:25:13 (1773685513) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:25:16 (1773685516) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:25:18 (1773685518) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:25:37 (1773685537) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:25:40 (1773685540) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:25:42 (1773685542) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d0804e30800.stats=clear osc.lustre-OST0001-osc-ffff8d0804e30800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:25:48 (1773685548) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:25:57 (1773685557) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:26:04 (1773685564) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:26:07 (1773685567) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 14:26:14 (1773685574) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:26:19 (1773685579) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:26:24 (1773685584) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:26:30 (1773685590) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:26:37 (1773685597) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:26:42 (1773685602) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.988629 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099804 s, 10.5 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:26:50 (1773685610) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.332006 s, 63.2 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:26:56 (1773685616) Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (48s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:27:44 (1773685664) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.12647 s, 24.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:28:05 (1773685685) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (45s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:28:50 (1773685730) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.19594 s, 16.4 MB/s -rw-r--r-- 1 root root 52428800 Mar 16 14:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.645s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.977s user 0m0.001s sys 0m0.167s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.861s user 0m0.000s sys 0m0.146s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d0804e30800.ost_server_uuid in FULL state after 0 sec PASS 33b (56s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:29:46 (1773685786) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:29:48 (1773685788) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:29:50 (1773685790) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 14:29:52 (1773685792) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 14:29:57 (1773685797) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.436673 s, 28.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:30:05 (1773685805) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264742 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202313 s, 51.8 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:30:09 (1773685809) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:30:14 (1773685814) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166113 s, 616 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106715 s, 960 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00063906 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011722 s, 874 kB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 14:30:21 (1773685821) create test files with size 15751087 .. a092771efa9aa46206396808346db269 - ec875b4c15b2ba9024d9f822ae78b36a - ee793f89467ba840903a878de6ce7b9a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 14:30:34 (1773685834) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:30:42 (1773685842) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:30:46 (1773685846) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476008 s, 22.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175175 s, 59.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 14:30:50 (1773685850) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.091049 s, 23.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.196997 s, 21.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109968 s, 19.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 14:30:59 (1773685859) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 14:31:01 (1773685861) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:31:03 (1773685863) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106807 s, 98.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 14:31:07 (1773685867) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.197759 s, 15.9 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:31:13 (1773685873) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:31:19 (1773685879) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.128757 s, 81.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:31:27 (1773685887) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8d0804e33000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.1456 s, 72.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d0819e4b800.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:31:36 (1773685896) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (23s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:31:59 (1773685919) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (21s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:32:20 (1773685940) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:32:22 (1773685942) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171097 s, 18.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.109993 s, 28.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe407b483 (stale) Mirror 2: 0xe407b483 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 6e728b1b254d6095b4c73c937d7d22ed - mirror 1 checksum: 8f69748b7eb179be9fd45b883c28a5a7 - mirror 2 checksum: 6e728b1b254d6095b4c73c937d7d22ed - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 14:32:24 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 14:32:24 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6e728b1b254d6095b4c73c937d7d22ed - mirror 1 checksum: 8f69748b7eb179be9fd45b883c28a5a7 - mirror 2 checksum: 8f69748b7eb179be9fd45b883c28a5a7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 14:32:24 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 14:32:24 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:32:29 (1773685949) Failing mds1 on oleg304-server Stopping /mnt/lustre-mds1 (opts:) on oleg304-server 14:32:33 (1773685953) shut down facet: mds1 facet_host: oleg304-server facet_failover_host: oleg304-server Failover mds1 to oleg304-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 14:32:49 (1773685969) targets are mounted 14:32:49 (1773685969) facet_failover done oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222107 s, 47.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:32:59 (1773685979) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:33:01 (1773685981) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:33:03 (1773685983) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0061783 s, 2.7 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:33:07 (1773685987) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:33:09 (1773685989) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:33:11 (1773685991) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:33:22 (1773686002) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:33:44 (1773686024) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x40:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:34:04 (1773686044) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:34:24 (1773686064) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 14:34:28 (1773686068) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 54281 mirror split pid 54283 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 54281 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 54283 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 14:35:32 (1773686132) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6133 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000006 1773686133.361179 write 0x0320b9 thru 0x0392f4 (0x723c bytes) truncating to largest ever: 0x91ed 000007 1773686133.362802 trunc from 0x0392f5 to 0x0091ec (0x30108 bytes) truncating to largest ever: 0x9a28 000013 1773686133.378478 trunc from 0x0091ed to 0x009a27 (0x83b bytes) 000014 1773686133.393407 MIRROR_EXTEND to 2 mirrors 000016 1773686133.787395 read 0x000590 thru 0x0086df (0x8150 bytes) 000017 1773686133.794998 read 0x000cc3 thru 0x009a27 (0x8d65 bytes) 000018 1773686133.798680 read 0x0055c3 thru 0x009a27 (0x4465 bytes) truncating to largest ever: 0x23ba1 000019 1773686133.799339 trunc from 0x009a28 to 0x023ba0 (0x1a179 bytes) 000020 1773686133.895945 read 0x00d190 thru 0x01b198 (0xe009 bytes) 000021 1773686133.907621 mapwrite 0x038b1a thru 0x03d947 (0x4e2e bytes) 000023 1773686133.969608 mapwrite 0x01356c thru 0x020591 (0xd026 bytes) 000026 1773686134.010737 mapwrite 0x02b55a thru 0x036540 (0xafe7 bytes) 000027 1773686134.038896 write 0x026b99 thru 0x030e88 (0xa2f0 bytes) 000028 1773686134.040177 mapread 0x0212a4 thru 0x027b43 (0x68a0 bytes) 000033 1773686134.042738 write 0x0279eb thru 0x02d7ae (0x5dc4 bytes) 000035 1773686134.044168 mapread 0x0070d3 thru 0x012e60 (0xbd8e bytes) 000037 1773686134.049821 mapwrite 0x008543 thru 0x012f61 (0xaa1f bytes) 000038 1773686134.092973 mapwrite 0x004af4 thru 0x00d201 (0x870e bytes) 000039 1773686134.123781 trunc from 0x03d948 to 0x01b2bb (0x2268c bytes) 000040 1773686134.133532 mapread 0x0008b8 thru 0x00e0e5 (0xd82e bytes) 000045 1773686134.135932 write 0x01bb2e thru 0x0200a5 (0x4578 bytes) 000046 1773686134.141264 MIRROR_RESYNC 2 mirrors 000048 1773686134.550934 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000049 1773686134.629939 read 0x015d94 thru 0x0200a5 (0xa312 bytes) 000050 1773686134.635991 mapwrite 0x01b3e2 thru 0x020648 (0x5267 bytes) 000052 1773686134.743661 write 0x039422 thru 0x03ffff (0x6bde bytes) 000053 1773686134.747124 write 0x0217d4 thru 0x02fbfc (0xe429 bytes) truncating to largest ever: 0x3c5e4 000057 1773686134.764423 trunc from 0x040000 to 0x03c5e3 (0x3a1c bytes) 000058 1773686134.783446 read 0x0034c5 thru 0x00f40e (0xbf4a bytes) 000067 1773686134.796981 mapread 0x02c251 thru 0x03a32e (0xe0de bytes) 000068 1773686134.798688 read 0x003814 thru 0x00fde1 (0xc5ce bytes) 000069 1773686134.799469 mapread 0x015269 thru 0x01d99e (0x8736 bytes) 000072 1773686134.802008 read 0x015d9f thru 0x0209aa (0xac0c bytes) 000073 1773686134.806210 trunc from 0x03c5e4 to 0x0235f4 (0x18fef bytes) 000074 1773686134.816222 trunc from 0x0235f5 to 0x00131b (0x222d9 bytes) 000080 1773686134.842568 mapwrite 0x01bf8a thru 0x027207 (0xb27e bytes) 000086 1773686134.934910 mapwrite 0x028bfa thru 0x037714 (0xeb1b bytes) 000090 1773686135.032893 mapread 0x0046ea thru 0x00f09f (0xa9b6 bytes) 000091 1773686135.033943 mapwrite 0x03b6b2 thru 0x03ffff (0x494e bytes) 000092 1773686135.099237 MIRROR_EXTEND to 2 mirrors 000093 1773686135.434219 mapwrite 0x015b4c thru 0x01bce9 (0x619e bytes) 000095 1773686135.541221 mapwrite 0x03f220 thru 0x03ffff (0xde0 bytes) 000096 1773686135.572557 mapread 0x01256f thru 0x021732 (0xf1c4 bytes) 000097 1773686135.582597 write 0x016f81 thru 0x022501 (0xb581 bytes) 000099 1773686135.586152 write 0x023fbf thru 0x029cf8 (0x5d3a bytes) 000100 1773686135.591591 write 0x02427b thru 0x02d07a (0x8e00 bytes) 000102 1773686135.596834 mapread 0x01750b thru 0x018931 (0x1427 bytes) 000104 1773686135.598352 mapread 0x012f2a thru 0x018bc3 (0x5c9a bytes) 000105 1773686135.599329 write 0x01e962 thru 0x024b48 (0x61e7 bytes) 000106 1773686135.604250 MIRROR_RESYNC 2 mirrors 000108 1773686135.928794 write 0x00589f thru 0x00fabe (0xa220 bytes) 000109 1773686135.974951 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x634f68f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x193a7939 Mirror 3: 0x193a7939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000112 1773686136.419470 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x634f68f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x193a7939 Mirror 3: 0x193a7939 Mirror 4: 0x193a7939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000113 1773686136.909302 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000114 1773686137.150102 mapread 0x02bdf7 thru 0x03559c (0x97a6 bytes) 000115 1773686137.176169 read 0x01a4aa thru 0x01f2a8 (0x4dff bytes) 000116 1773686137.182823 mapwrite 0x013968 thru 0x01a7af (0x6e48 bytes) 000117 1773686137.289944 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x193a7939 (stale) Mirror 4: 0x193a7939 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef98d24b Mirror 5: 0xef98d24b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000118 1773686137.829344 mapread 0x010203 thru 0x015eb3 (0x5cb1 bytes) 000119 1773686137.860219 mapread 0x01521b thru 0x016b57 (0x193d bytes) 000120 1773686137.869512 mapread 0x007595 thru 0x010aba (0x9526 bytes) 000121 1773686137.880203 read 0x022787 thru 0x0252a2 (0x2b1c bytes) 000123 1773686137.889889 mapread 0x00f538 thru 0x00f72c (0x1f5 bytes) 000124 1773686137.895635 write 0x020421 thru 0x02947f (0x905f bytes) 000125 1773686137.941852 mapread 0x004c85 thru 0x00cb7a (0x7ef6 bytes) 000128 1773686137.952619 trunc from 0x040000 to 0x0334bd (0xcb42 bytes) 000133 1773686137.983993 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x334be): Mirror 3: 0x700996fd (stale) Mirror 5: 0xc298d8f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334be): Mirror 3: 0x700996fd (stale) Mirror 5: 0xc298d8f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334be: skipped 000135 1773686138.158452 write 0x01bdd3 thru 0x02514d (0x937b bytes) 000136 1773686138.165390 mapwrite 0x01cf13 thru 0x0225a3 (0x5691 bytes) 000138 1773686138.205818 mapwrite 0x01666f thru 0x01f951 (0x92e3 bytes) 000139 1773686138.242135 mapread 0x029373 thru 0x02fa8d (0x671b bytes) 000141 1773686138.258102 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x334be): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334be): Mirror 3: 0x700996fd (stale) Mirror 5: 0xc298d8f2 (stale) CRC-32 checksum value for chunk [0, 0x334be): Mirror 1: 0x3aa0916b Mirror 6: 0x3aa0916b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334be: skipped 000142 1773686138.717352 read 0x0331c6 thru 0x0334bd (0x2f8 bytes) 000143 1773686138.725411 mapwrite 0x037638 thru 0x03b6c3 (0x408c bytes) 000144 1773686138.817490 mapwrite 0x027a9c thru 0x031fc5 (0xa52a bytes) 000147 1773686138.864101 trunc from 0x03b6c4 to 0x0262b8 (0x1540b bytes) 000148 1773686138.888074 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x262b9): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262b9): Mirror 3: 0x594b36be (stale) Mirror 5: 0xca09907d (stale) Mirror 6: 0x423a2270 (stale) CRC-32 checksum value for chunk [0, 0x262b9): Mirror 1: 0x423a2270 Mirror 7: 0x423a2270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262b9: skipped 000150 1773686139.342343 mapwrite 0x004c65 thru 0x00b032 (0x63ce bytes) 000151 1773686139.473125 trunc from 0x0262b9 to 0x0099b0 (0x1c908 bytes) 000152 1773686139.487933 read 0x0094fa thru 0x0099b0 (0x4b7 bytes) 000155 1773686139.489810 read 0x0009aa thru 0x0099b0 (0x9007 bytes) 000156 1773686139.495705 write 0x0025db thru 0x00d4c5 (0xaeeb bytes) 000159 1773686139.497832 read 0x00a6da thru 0x00d4c5 (0x2dec bytes) 000161 1773686139.498973 trunc from 0x00d4c6 to 0x03415d (0x26c98 bytes) 000162 1773686139.507746 trunc from 0x03415e to 0x0259ff (0xe75e bytes) 000163 1773686139.516316 read 0x022e11 thru 0x0259ff (0x2bef bytes) 000165 1773686139.526242 read 0x022b57 thru 0x0259ff (0x2ea9 bytes) 000166 1773686139.526804 mapwrite 0x01096b thru 0x01113d (0x7d3 bytes) 000170 1773686139.568947 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25a00): Mirror 5: 0xa1163737 (stale) Mirror 6: 0xa062f31c (stale) Mirror 7: 0xa062f31c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a00): Mirror 5: 0xa1163737 (stale) Mirror 6: 0xa062f31c (stale) Mirror 7: 0xa062f31c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a00: skipped 000171 1773686139.767947 mapwrite 0x02b424 thru 0x030f7d (0x5b5a bytes) 000172 1773686139.810723 mapwrite 0x037e63 thru 0x03cf8e (0x512c bytes) 000173 1773686139.878246 trunc from 0x03cf8f to 0x00f36c (0x2dc22 bytes) truncating to largest ever: 0x3f6d0 000175 1773686139.899343 trunc from 0x00f36d to 0x03f6cf (0x30363 bytes) 000176 1773686139.911359 write 0x03af49 thru 0x03ffff (0x50b7 bytes) 000178 1773686139.921831 trunc from 0x040000 to 0x03c133 (0x3ecc bytes) 000180 1773686139.940481 write 0x00175d thru 0x006a29 (0x52cd bytes) 000181 1773686139.953156 trunc from 0x03c134 to 0x020063 (0x1c0d0 bytes) 000182 1773686139.969974 read 0x0015dd thru 0x0110be (0xfae2 bytes) 000183 1773686139.982094 write 0x02712e thru 0x02ae9c (0x3d6f bytes) 000185 1773686139.987217 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ae9d): Mirror 5: 0x94e8f265 (stale) Mirror 7: 0x12b4b11e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae9d): Mirror 5: 0x94e8f265 (stale) Mirror 7: 0x12b4b11e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae9d: skipped 000186 1773686140.157292 write 0x02bdd4 thru 0x0335e8 (0x7815 bytes) 000187 1773686140.161440 MIRROR_RESYNC 3 mirrors 000188 1773686140.584430 mapread 0x02f931 thru 0x0335e8 (0x3cb8 bytes) 000189 1773686140.598992 write 0x02df11 thru 0x03b939 (0xda29 bytes) 000190 1773686140.667915 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b93a): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b93a): Mirror 5: 0x91d255c6 (stale) Mirror 7: 0x91d255c6 (stale) CRC-32 checksum value for chunk [0, 0x3b93a): Mirror 1: 0x9b2f1eda Mirror 8: 0x9b2f1eda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b93a: skipped 000191 1773686141.262287 MIRROR_RESYNC 4 mirrors 000194 1773686141.747321 write 0x020d89 thru 0x02a594 (0x980c bytes) 000198 1773686141.806675 read 0x02a14a thru 0x02a6f5 (0x5ac bytes) 000199 1773686141.807410 read 0x0284e4 thru 0x031b76 (0x9693 bytes) 000200 1773686141.814862 read 0x01a7f8 thru 0x01a979 (0x182 bytes) 000202 1773686141.815828 trunc from 0x03b93a to 0x01106a (0x2a8cf bytes) 000203 1773686141.826468 trunc from 0x01106b to 0x02f1b6 (0x1e14c bytes) 000205 1773686141.835828 read 0x02b61c thru 0x02f1b6 (0x3b9b bytes) 000207 1773686141.845963 write 0x03fcc4 thru 0x03ffff (0x33c bytes) 000208 1773686141.850990 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a6c48fe (stale) Mirror 7: 0x3a6c48fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a6c48fe (stale) Mirror 7: 0x3a6c48fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000210 1773686142.037026 mapwrite 0x023bd6 thru 0x032162 (0xe58d bytes) 000211 1773686142.098757 mapwrite 0x02972c thru 0x035c0b (0xc4e0 bytes) 000212 1773686142.136909 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a6c48fe (stale) Mirror 7: 0x3a6c48fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf2199bf0 Mirror 8: 0xf2199bf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000213 1773686142.554825 mapwrite 0x02c330 thru 0x032854 (0x6525 bytes) 000214 1773686142.683087 mapwrite 0x029f06 thru 0x0364f0 (0xc5eb bytes) 000215 1773686142.709963 mapread 0x01354a thru 0x01ca33 (0x94ea bytes) 000216 1773686142.718563 read 0x0371aa thru 0x03df0b (0x6d62 bytes) 000218 1773686142.719365 trunc from 0x040000 to 0x001195 (0x3ee6a bytes) 000221 1773686142.737942 read 0x0008f4 thru 0x001195 (0x8a2 bytes) 000225 1773686142.744105 mapwrite 0x034ca3 thru 0x03ffff (0xb35d bytes) 000226 1773686142.791885 write 0x0252c4 thru 0x02fed2 (0xac0f bytes) 000227 1773686142.793120 trunc from 0x040000 to 0x013d9e (0x2c261 bytes) 000228 1773686142.803971 write 0x02aa81 thru 0x03607f (0xb5ff bytes) 000230 1773686142.806182 write 0x019eab thru 0x0225d8 (0x872e bytes) 000232 1773686142.812228 write 0x00a48c thru 0x00cff3 (0x2b68 bytes) 000234 1773686142.813382 trunc from 0x036080 to 0x01eae1 (0x1759e bytes) 000235 1773686142.822268 trunc from 0x01eae2 to 0x0387cf (0x19cee bytes) 000237 1773686142.831770 trunc from 0x0387d0 to 0x0246b8 (0x14117 bytes) 000238 1773686142.845063 MIRROR_RESYNC 4 mirrors 000240 1773686143.427584 read 0x0132ea thru 0x0179ed (0x4704 bytes) 000241 1773686143.432914 trunc from 0x0246b9 to 0x0298d0 (0x5218 bytes) 000242 1773686143.491723 mapwrite 0x00f810 thru 0x013ba8 (0x4399 bytes) 000244 1773686143.553661 write 0x00b504 thru 0x017707 (0xc204 bytes) 000247 1773686143.555890 mapread 0x01b9bb thru 0x023c32 (0x8278 bytes) 000249 1773686143.571231 write 0x03b861 thru 0x03e4d7 (0x2c77 bytes) 000250 1773686143.573127 mapwrite 0x0323da thru 0x039839 (0x7460 bytes) 000251 1773686143.613999 mapread 0x028a97 thru 0x02b4aa (0x2a14 bytes) 000252 1773686143.620845 read 0x014c00 thru 0x01b6c4 (0x6ac5 bytes) 000253 1773686143.621581 mapwrite 0x01534b thru 0x01bcde (0x6994 bytes) 000254 1773686143.654052 read 0x027256 thru 0x032000 (0xadab bytes) 000255 1773686143.655066 trunc from 0x03e4d8 to 0x02107f (0x1d458 bytes) 000256 1773686143.675541 mapwrite 0x033033 thru 0x03ffff (0xcfcd bytes) 000257 1773686143.729836 write 0x03f33a thru 0x03ffff (0xcc6 bytes) 000259 1773686143.731227 mapwrite 0x018cca thru 0x023d49 (0xb080 bytes) 000260 1773686143.777995 trunc from 0x040000 to 0x029605 (0x169fa bytes) 000262 1773686143.799799 mapwrite 0x001d2a thru 0x00e34f (0xc626 bytes) 000263 1773686143.854312 trunc from 0x029606 to 0x033af7 (0xa4f2 bytes) 000265 1773686143.880082 mapwrite 0x03fe93 thru 0x03ffff (0x16d bytes) 000266 1773686143.927268 read 0x02ed48 thru 0x02f5cd (0x886 bytes) 000267 1773686143.936870 read 0x024ff6 thru 0x0304fe (0xb509 bytes) 000268 1773686143.937654 mapread 0x002401 thru 0x0043d1 (0x1fd1 bytes) 000270 1773686143.939812 trunc from 0x040000 to 0x02e1c6 (0x11e39 bytes) 000271 1773686143.951578 read 0x00c0d7 thru 0x00de3a (0x1d64 bytes) 000273 1773686143.953578 mapread 0x007b58 thru 0x0131e8 (0xb691 bytes) 000279 1773686143.959192 write 0x02a125 thru 0x038a04 (0xe8e0 bytes) 000280 1773686143.964960 trunc from 0x038a05 to 0x01b9a7 (0x1d05d bytes) 000281 1773686143.991062 mapread 0x01aaca thru 0x01b3bf (0x8f6 bytes) 000282 1773686143.992181 mapwrite 0x024115 thru 0x029681 (0x556d bytes) 000283 1773686144.044396 mapread 0x012059 thru 0x01a91f (0x88c7 bytes) 000285 1773686144.046151 write 0x02fc34 thru 0x03b011 (0xb3de bytes) 000287 1773686144.050391 mapread 0x00aa3c thru 0x00c2d7 (0x189c bytes) 000289 1773686144.052547 read 0x033de7 thru 0x03a34b (0x6565 bytes) 000290 1773686144.057462 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b012): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b012): Mirror 5: 0x8b8845a9 (stale) Mirror 7: 0x8b8845a9 (stale) Mirror 8: 0x8b8845a9 (stale) CRC-32 checksum value for chunk [0, 0x3b012): Mirror 1: 0x3b35e702 Mirror 9: 0x3b35e702 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b012: skipped 000291 1773686144.621333 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b012): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b012): Mirror 5: 0x8b8845a9 (stale) Mirror 7: 0x8b8845a9 (stale) Mirror 8: 0x8b8845a9 (stale) CRC-32 checksum value for chunk [0, 0x3b012): Mirror 1: 0x3b35e702 Mirror 9: 0x3b35e702 Mirror 10: 0x3b35e702 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b012: skipped 000292 1773686145.147804 mapwrite 0x0194c3 thru 0x0277c5 (0xe303 bytes) 000293 1773686145.285666 mapread 0x01e414 thru 0x024225 (0x5e12 bytes) 000296 1773686145.287480 trunc from 0x03b012 to 0x03a37c (0xc95 bytes) 000298 1773686145.345921 read 0x018570 thru 0x01d6eb (0x517c bytes) 000299 1773686145.348933 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 5: 0x57821197 (stale) Mirror 7: 0x57821197 (stale) Mirror 8: 0x57821197 (stale) Mirror 10: 0xa1b42a17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 5: 0x57821197 (stale) Mirror 7: 0x57821197 (stale) Mirror 8: 0x57821197 (stale) Mirror 10: 0xa1b42a17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37d: skipped 000302 1773686145.620793 mapread 0x012e5e thru 0x01c530 (0x96d3 bytes) 000303 1773686145.636997 mapread 0x0215e2 thru 0x022bae (0x15cd bytes) 000305 1773686145.650755 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 5: 0x57821197 (stale) Mirror 7: 0x57821197 (stale) Mirror 8: 0x57821197 (stale) Mirror 10: 0xa1b42a17 (stale) CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 1: 0x3e347c5e Mirror 11: 0x3e347c5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37d: skipped 000308 1773686146.163646 MIRROR_RESYNC 6 mirrors 000309 1773686146.825606 read 0x03a0b1 thru 0x03a37c (0x2cc bytes) 000313 1773686146.846914 MIRROR_EXTEND to 7 mirrors 000316 1773686147.317900 write 0x02932f thru 0x02fd1d (0x69ef bytes) 000317 1773686147.373850 mapread 0x0384fd thru 0x03a37c (0x1e80 bytes) 000319 1773686147.380151 write 0x00572d thru 0x00a543 (0x4e17 bytes) 000321 1773686147.387231 write 0x022d78 thru 0x02c0a8 (0x9331 bytes) 000322 1773686147.392559 write 0x00e5f4 thru 0x0162ae (0x7cbb bytes) 000323 1773686147.401440 MIRROR_RESYNC 7 mirrors 000324 1773686148.108987 read 0x02e068 thru 0x03a37c (0xc315 bytes) 000325 1773686148.119694 trunc from 0x03a37d to 0x008d74 (0x31608 bytes) 000326 1773686148.204544 mapwrite 0x00a138 thru 0x00ff64 (0x5e2d bytes) 000327 1773686148.263240 mapwrite 0x0324f4 thru 0x03ffff (0xdb0c bytes) 000330 1773686148.337411 mapread 0x0075b0 thru 0x00d1d0 (0x5c21 bytes) 000332 1773686148.339460 write 0x02768a thru 0x029522 (0x1e99 bytes) 000333 1773686148.350748 mapread 0x01a5d4 thru 0x01c989 (0x23b6 bytes) 000336 1773686148.358122 mapwrite 0x02ebd2 thru 0x03060d (0x1a3c bytes) 000337 1773686148.382679 mapwrite 0x018691 thru 0x02396e (0xb2de bytes) 000341 1773686148.449891 mapwrite 0x014cc4 thru 0x01a295 (0x55d2 bytes) 000342 1773686148.509387 read 0x0364d7 thru 0x03ffff (0x9b29 bytes) 000343 1773686148.510418 write 0x022448 thru 0x02b7a2 (0x935b bytes) 000344 1773686148.516186 write 0x025af1 thru 0x026b24 (0x1034 bytes) 000345 1773686148.522363 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe509787c (stale) Mirror 7: 0xe509787c (stale) Mirror 8: 0xe509787c (stale) Mirror 11: 0xe509787c (stale) Mirror 12: 0xe509787c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe509787c (stale) Mirror 7: 0xe509787c (stale) Mirror 8: 0xe509787c (stale) Mirror 11: 0xe509787c (stale) Mirror 12: 0xe509787c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000347 1773686148.799870 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe509787c (stale) Mirror 7: 0xe509787c (stale) Mirror 8: 0xe509787c (stale) Mirror 11: 0xe509787c (stale) Mirror 12: 0xe509787c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f38dc1e Mirror 13: 0x6f38dc1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000349 1773686149.331899 read 0x01f14e thru 0x029b88 (0xaa3b bytes) 000351 1773686149.346439 write 0x0281f7 thru 0x02a61b (0x2425 bytes) 000352 1773686149.438422 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe509787c (stale) Mirror 7: 0xe509787c (stale) Mirror 11: 0xe509787c (stale) Mirror 12: 0xe509787c (stale) Mirror 13: 0x6f38dc1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe509787c (stale) Mirror 7: 0xe509787c (stale) Mirror 11: 0xe509787c (stale) Mirror 12: 0xe509787c (stale) Mirror 13: 0x6f38dc1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000354 1773686149.712770 trunc from 0x040000 to 0x006f7c (0x39083 bytes) 000355 1773686149.719802 read 0x002baa thru 0x0053ed (0x2844 bytes) 000358 1773686149.729482 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x6f7d): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6f7d): Mirror 5: 0x61b8f317 (stale) Mirror 7: 0x61b8f317 (stale) Mirror 11: 0x61b8f317 (stale) Mirror 12: 0x61b8f317 (stale) Mirror 13: 0x61b8f317 (stale) CRC-32 checksum value for chunk [0, 0x6f7d): Mirror 1: 0x61b8f317 Mirror 14: 0x61b8f317 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6f7d: skipped 000359 1773686150.170389 mapread 0x0064d2 thru 0x006f7c (0xaab bytes) 000363 1773686150.186791 read 0x00311d thru 0x006f7c (0x3e60 bytes) 000364 1773686150.193776 trunc from 0x006f7d to 0x01a9c1 (0x13a45 bytes) 000367 1773686150.259313 write 0x03bfc0 thru 0x03da26 (0x1a67 bytes) 000369 1773686150.260636 trunc from 0x03da27 to 0x038713 (0x5313 bytes) 000372 1773686150.280351 mapwrite 0x025dd8 thru 0x027565 (0x178e bytes) 000373 1773686150.323221 read 0x006c7a thru 0x00b6fd (0x4a84 bytes) 000376 1773686150.338400 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38714): Mirror 5: 0x9f8369a8 (stale) Mirror 7: 0x9f8369a8 (stale) Mirror 11: 0x9f8369a8 (stale) Mirror 12: 0x9f8369a8 (stale) Mirror 13: 0x8d50a989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38714): Mirror 5: 0x9f8369a8 (stale) Mirror 7: 0x9f8369a8 (stale) Mirror 11: 0x9f8369a8 (stale) Mirror 12: 0x9f8369a8 (stale) Mirror 13: 0x8d50a989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38714: skipped 000377 1773686150.618988 mapwrite 0x02d882 thru 0x03cdb7 (0xf536 bytes) 000380 1773686150.681713 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3cdb8): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdb8): Mirror 5: 0x28ca0d70 (stale) Mirror 7: 0x28ca0d70 (stale) Mirror 11: 0x28ca0d70 (stale) Mirror 12: 0x28ca0d70 (stale) Mirror 13: 0x2b5af109 (stale) CRC-32 checksum value for chunk [0, 0x3cdb8): Mirror 1: 0x2b906646 Mirror 14: 0x2b906646 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdb8: skipped 000381 1773686151.149051 read 0x023c7e thru 0x026ac1 (0x2e44 bytes) 000383 1773686151.166046 mapread 0x0304d4 thru 0x0363e5 (0x5f12 bytes) 000384 1773686151.187586 mapwrite 0x023e43 thru 0x0322c6 (0xe484 bytes) 000386 1773686151.325393 read 0x006da6 thru 0x0168ab (0xfb06 bytes) 000387 1773686151.334522 mapread 0x0108cf thru 0x01acee (0xa420 bytes) 000389 1773686151.336737 trunc from 0x03cdb8 to 0x014a28 (0x2838f bytes) 000390 1773686151.381411 trunc from 0x014a29 to 0x017cdb (0x32b3 bytes) 000391 1773686151.406247 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x17cdc): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17cdc): Mirror 5: 0x4b421f3b (stale) Mirror 7: 0x4b421f3b (stale) Mirror 11: 0x4b421f3b (stale) Mirror 12: 0x4b421f3b (stale) Mirror 13: 0xe891c917 (stale) Mirror 14: 0x6e70c195 (stale) CRC-32 checksum value for chunk [0, 0x17cdc): Mirror 1: 0x6e70c195 Mirror 15: 0x6e70c195 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17cdc: skipped 000392 1773686151.932877 write 0x03f22f thru 0x03f5a1 (0x373 bytes) 000393 1773686151.979012 read 0x0278d0 thru 0x03067a (0x8dab bytes) 000395 1773686151.986149 write 0x018c71 thru 0x01d007 (0x4397 bytes) 000396 1773686151.993471 mapwrite 0x03879b thru 0x03a84b (0x20b1 bytes) 000397 1773686152.052594 mapwrite 0x02aaa0 thru 0x02f038 (0x4599 bytes) 000398 1773686152.106893 mapread 0x00dc9c thru 0x00fbc4 (0x1f29 bytes) 000399 1773686152.117102 read 0x017b24 thru 0x01e825 (0x6d02 bytes) 000400 1773686152.123715 write 0x03bb9b thru 0x03ffff (0x4465 bytes) 000403 1773686152.128914 read 0x023376 thru 0x02c29b (0x8f26 bytes) 000404 1773686152.130304 write 0x0293ee thru 0x0328f0 (0x9503 bytes) 000405 1773686152.132285 trunc from 0x040000 to 0x01208f (0x2df70 bytes) 000407 1773686152.160850 write 0x00b3b3 thru 0x0160a8 (0xacf6 bytes) 000408 1773686152.161865 read 0x01446f thru 0x0160a8 (0x1c3a bytes) 000411 1773686152.168729 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: 0x5b616066 (stale) Mirror 7: 0x5b616066 (stale) Mirror 11: 0x5b616066 (stale) Mirror 12: 0x5b616066 (stale) Mirror 13: 0x299077be (stale) Mirror 15: 0xb42eed7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: 0x5b616066 (stale) Mirror 7: 0x5b616066 (stale) Mirror 11: 0x5b616066 (stale) Mirror 12: 0x5b616066 (stale) Mirror 13: 0x299077be (stale) Mirror 15: 0xb42eed7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160a9: skipped 000413 1773686152.455703 read 0x01282b thru 0x013209 (0x9df bytes) 000415 1773686152.464147 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: 0x5b616066 (stale) Mirror 7: 0x5b616066 (stale) Mirror 11: 0x5b616066 (stale) Mirror 12: 0x5b616066 (stale) Mirror 13: 0x299077be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: 0x5b616066 (stale) Mirror 7: 0x5b616066 (stale) Mirror 11: 0x5b616066 (stale) Mirror 12: 0x5b616066 (stale) Mirror 13: 0x299077be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160a9: skipped 000416 1773686152.724351 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160a9): Mirror 5: 0x5b616066 (stale) Mirror 7: 0x5b616066 (stale) Mirror 11: 0x5b616066 (stale) Mirror 12: 0x5b616066 (stale) Mirror 13: 0x299077be (stale) CRC-32 checksum value for chunk [0, 0x160a9): Mirror 1: 0x2e55ed58 Mirror 14: 0x2e55ed58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160a9: skipped 000417 1773686153.210879 mapread 0x005cab thru 0x00692b (0xc81 bytes) 000418 1773686153.227679 trunc from 0x0160a9 to 0x02bed4 (0x15e2c bytes) 000419 1773686153.299135 read 0x017685 thru 0x025d78 (0xe6f4 bytes) 000420 1773686153.311018 mapread 0x0253ce thru 0x02bed4 (0x6b07 bytes) 000421 1773686153.320927 read 0x013c48 thru 0x018c0d (0x4fc6 bytes) 000422 1773686153.335157 write 0x005d10 thru 0x00f62e (0x991f bytes) 000425 1773686153.338673 trunc from 0x02bed5 to 0x00c5a5 (0x1f92f bytes) 000426 1773686153.364780 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc5a6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc5a6): Mirror 5: 0xd12a1426 (stale) Mirror 7: 0xd12a1426 (stale) Mirror 11: 0xd12a1426 (stale) Mirror 12: 0xd12a1426 (stale) Mirror 13: 0x5eff97a8 (stale) Mirror 14: 0x701d1661 (stale) CRC-32 checksum value for chunk [0, 0xc5a6): Mirror 1: 0x2d2f44b8 Mirror 15: 0x2d2f44b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc5a6: skipped 000427 1773686153.862425 mapread 0x00b225 thru 0x00c5a5 (0x1381 bytes) 000428 1773686153.885222 mapwrite 0x0019df thru 0x00d39d (0xb9bf bytes) 000430 1773686154.055607 write 0x0156ad thru 0x02188b (0xc1df bytes) 000431 1773686154.057122 mapwrite 0x0112ea thru 0x012ef0 (0x1c07 bytes) 000432 1773686154.110842 mapwrite 0x01f3ae thru 0x02cd35 (0xd988 bytes) 000433 1773686154.177922 write 0x03c72a thru 0x03ffff (0x38d6 bytes) 000434 1773686154.179238 read 0x034958 thru 0x03ffff (0xb6a8 bytes) 000436 1773686154.191020 read 0x030fcb thru 0x03990d (0x8943 bytes) 000437 1773686154.197369 read 0x01c6dd thru 0x024f65 (0x8889 bytes) 000438 1773686154.198168 write 0x01b38e thru 0x026716 (0xb389 bytes) 000439 1773686154.200740 read 0x022c76 thru 0x024198 (0x1523 bytes) 000440 1773686154.205405 MIRROR_RESYNC 8 mirrors 000442 1773686155.011143 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000443 1773686155.268216 mapread 0x008d91 thru 0x00faf8 (0x6d68 bytes) 000444 1773686155.277954 mapwrite 0x037649 thru 0x03ffff (0x89b7 bytes) 000447 1773686155.412266 write 0x03c664 thru 0x03ffff (0x399c bytes) 000448 1773686155.413606 mapread 0x03c02d thru 0x03ffff (0x3fd3 bytes) 000449 1773686155.415607 mapread 0x030d31 thru 0x03e01f (0xd2ef bytes) 000452 1773686155.419862 trunc from 0x040000 to 0x03c045 (0x3fba bytes) 000453 1773686155.456828 mapread 0x0108a1 thru 0x01325d (0x29bd bytes) 000455 1773686155.466193 mapread 0x013fb0 thru 0x015e7c (0x1ecd bytes) 000456 1773686155.468896 write 0x01e2bd thru 0x029b04 (0xb848 bytes) 000457 1773686155.480700 mapwrite 0x03c85e thru 0x03ffff (0x37a2 bytes) 000458 1773686155.537447 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xac3207e6 (stale) Mirror 7: 0xac3207e6 (stale) Mirror 12: 0xac3207e6 (stale) Mirror 14: 0xac3207e6 (stale) Mirror 15: 0xac3207e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xac3207e6 (stale) Mirror 7: 0xac3207e6 (stale) Mirror 12: 0xac3207e6 (stale) Mirror 14: 0xac3207e6 (stale) Mirror 15: 0xac3207e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1773686155.812895 mapwrite 0x010485 thru 0x01d606 (0xd182 bytes) 000460 1773686155.881897 read 0x032a0f thru 0x03c23e (0x9830 bytes) 000461 1773686155.892816 mapwrite 0x00d904 thru 0x018132 (0xa82f bytes) 000462 1773686155.931246 mapread 0x03eb89 thru 0x03ffff (0x1477 bytes) 000463 1773686155.939621 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xac3207e6 (stale) Mirror 12: 0xac3207e6 (stale) Mirror 14: 0xac3207e6 (stale) Mirror 15: 0xac3207e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xac3207e6 (stale) Mirror 12: 0xac3207e6 (stale) Mirror 14: 0xac3207e6 (stale) Mirror 15: 0xac3207e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000466 1773686156.167906 mapread 0x03cf8b thru 0x03f9af (0x2a25 bytes) 000468 1773686156.190175 MIRROR_RESYNC 5 mirrors 000469 1773686156.775930 write 0x02d5ec thru 0x03477c (0x7191 bytes) 000470 1773686156.828224 mapread 0x03127a thru 0x03feda (0xec61 bytes) 000471 1773686156.842263 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x731b4b01 (stale) Mirror 12: 0x731b4b01 (stale) Mirror 14: 0x731b4b01 (stale) Mirror 15: 0x731b4b01 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e47c8d1 Mirror 16: 0x9e47c8d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000472 1773686157.293197 write 0x0137ea thru 0x02347c (0xfc93 bytes) 000473 1773686157.338118 mapwrite 0x0189b4 thru 0x01a2fd (0x194a bytes) 000475 1773686157.373810 read 0x0386a6 thru 0x03c9d7 (0x4332 bytes) 000477 1773686157.380488 read 0x01182b thru 0x020790 (0xef66 bytes) 000479 1773686157.394343 mapwrite 0x03d6f9 thru 0x03ffff (0x2907 bytes) 000480 1773686157.461982 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x731b4b01 (stale) Mirror 14: 0x731b4b01 (stale) Mirror 15: 0x731b4b01 (stale) Mirror 16: 0x9e47c8d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x731b4b01 (stale) Mirror 14: 0x731b4b01 (stale) Mirror 15: 0x731b4b01 (stale) Mirror 16: 0x9e47c8d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1773686157.732353 MIRROR_RESYNC 5 mirrors 000482 1773686158.390936 trunc from 0x040000 to 0x017280 (0x28d7f bytes) 000483 1773686158.434894 mapread 0x00781d thru 0x0106a1 (0x8e85 bytes) 000484 1773686158.451124 mapwrite 0x0094a2 thru 0x00f274 (0x5dd3 bytes) 000488 1773686158.498267 MIRROR_RESYNC 5 mirrors 000491 1773686159.139121 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000493 1773686159.437574 MIRROR_RESYNC 4 mirrors 000495 1773686159.737115 write 0x03fc95 thru 0x03ffff (0x36b bytes) 000497 1773686159.796478 mapread 0x01b833 thru 0x02021d (0x49eb bytes) 000499 1773686159.810182 mapwrite 0x02a15a thru 0x02bb47 (0x19ee bytes) 000501 1773686159.860266 write 0x03a411 thru 0x03ffff (0x5bef bytes) 000502 1773686159.869551 read 0x02f4eb thru 0x0359dc (0x64f2 bytes) 000505 1773686159.871882 mapwrite 0x001ecd thru 0x00b643 (0x9777 bytes) 000506 1773686159.927452 mapread 0x024676 thru 0x032965 (0xe2f0 bytes) 000507 1773686159.929358 write 0x02aac5 thru 0x03a731 (0xfc6d bytes) 000508 1773686159.931963 write 0x02a6cf thru 0x02fd98 (0x56ca bytes) 000509 1773686159.933432 read 0x019743 thru 0x02664a (0xcf08 bytes) 000511 1773686159.935477 trunc from 0x040000 to 0x0331c6 (0xce39 bytes) 000512 1773686159.974471 trunc from 0x0331c7 to 0x036f4d (0x3d87 bytes) 000513 1773686159.994738 mapwrite 0x031e92 thru 0x03ffff (0xe16e bytes) 000515 1773686160.078191 trunc from 0x040000 to 0x01b148 (0x24eb7 bytes) 000517 1773686160.100123 mapread 0x01a7a2 thru 0x01b148 (0x9a7 bytes) 000518 1773686160.101571 trunc from 0x01b149 to 0x006204 (0x14f44 bytes) 000519 1773686160.116236 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x6205): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6205): Mirror 14: 0xdff2888f (stale) Mirror 15: 0xdff2888f (stale) Mirror 16: 0xdff2888f (stale) CRC-32 checksum value for chunk [0, 0x6205): Mirror 1: 0x33ba9f0 Mirror 17: 0x33ba9f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6205: skipped 000522 1773686160.789441 mapwrite 0x019e6f thru 0x01ec92 (0x4e24 bytes) 000523 1773686160.909473 mapwrite 0x011509 thru 0x01b98f (0xa487 bytes) 000524 1773686160.963366 mapwrite 0x03e6dc thru 0x03ffff (0x1924 bytes) 000526 1773686161.025986 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x15013c56 (stale) Mirror 15: 0x15013c56 (stale) Mirror 16: 0x15013c56 (stale) Mirror 17: 0xb8a8ad03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa11ff751 Mirror 18: 0xa11ff751 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000527 1773686161.684057 mapwrite 0x025e1e thru 0x027c87 (0x1e6a bytes) 000529 1773686161.829087 trunc from 0x040000 to 0x03f493 (0xb6c bytes) 000530 1773686161.853470 write 0x03de63 thru 0x03ffff (0x219d bytes) 000531 1773686161.867386 MIRROR_RESYNC 6 mirrors 000534 1773686162.709842 mapread 0x026fc9 thru 0x036900 (0xf938 bytes) 000535 1773686162.746168 mapread 0x00c194 thru 0x014bd1 (0x8a3e bytes) 000540 1773686162.758304 write 0x0153db thru 0x01a023 (0x4c49 bytes) 000541 1773686162.830135 read 0x014ace thru 0x01ef39 (0xa46c bytes) 000543 1773686162.846752 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xed83ccf8 (stale) Mirror 16: 0xed83ccf8 (stale) Mirror 17: 0xed83ccf8 (stale) Mirror 18: 0xed83ccf8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xed83ccf8 (stale) Mirror 16: 0xed83ccf8 (stale) Mirror 17: 0xed83ccf8 (stale) Mirror 18: 0xed83ccf8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000544 1773686163.163543 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xed83ccf8 (stale) Mirror 17: 0xed83ccf8 (stale) Mirror 18: 0xed83ccf8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xed83ccf8 (stale) Mirror 17: 0xed83ccf8 (stale) Mirror 18: 0xed83ccf8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1773686163.512210 mapread 0x03ab06 thru 0x03af50 (0x44b bytes) 000546 1773686163.516162 write 0x01636c thru 0x01dafc (0x7791 bytes) 000549 1773686163.524764 mapread 0x00237b thru 0x00e376 (0xbffc bytes) 000550 1773686163.530567 mapread 0x03e287 thru 0x03ffff (0x1d79 bytes) 000552 1773686163.537300 mapwrite 0x033283 thru 0x039675 (0x63f3 bytes) 000554 1773686163.574118 mapread 0x01b95e thru 0x01c229 (0x8cc bytes) 000556 1773686163.576376 mapread 0x03713a thru 0x03ffff (0x8ec6 bytes) 000558 1773686163.583838 MIRROR_RESYNC 4 mirrors 000559 1773686164.250228 mapwrite 0x001720 thru 0x010f8a (0xf86b bytes) 000560 1773686164.418305 mapwrite 0x024dab thru 0x02e108 (0x935e bytes) 000562 1773686164.471303 mapwrite 0x012d21 thru 0x018ba0 (0x5e80 bytes) 000564 1773686164.501534 trunc from 0x040000 to 0x02f044 (0x10fbb bytes) 000566 1773686164.516158 trunc from 0x02f045 to 0x03f5e2 (0x1059e bytes) 000568 1773686164.533324 MIRROR_RESYNC 4 mirrors 000569 1773686165.184815 MIRROR_EXTEND to 5 mirrors 000570 1773686165.611942 read 0x014f9e thru 0x01dbec (0x8c4f bytes) 000571 1773686165.645340 mapread 0x035d84 thru 0x03dff6 (0x8273 bytes) 000574 1773686165.716607 MIRROR_RESYNC 5 mirrors 000575 1773686166.117218 read 0x003fda thru 0x00af6e (0x6f95 bytes) 000578 1773686166.148199 write 0x02594a thru 0x026be9 (0x12a0 bytes) 000579 1773686166.206960 write 0x009ab4 thru 0x012134 (0x8681 bytes) 000581 1773686166.220760 trunc from 0x03f5e3 to 0x00d4b2 (0x32130 bytes) 000582 1773686166.238395 read 0x008361 thru 0x00a4df (0x217f bytes) 000584 1773686166.239368 mapread 0x00d1e6 thru 0x00d4b2 (0x2cd bytes) 000587 1773686166.245603 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd4b3): Mirror 17: 0x9203e1d5 (stale) Mirror 18: 0x9203e1d5 (stale) Mirror 19: 0x9203e1d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd4b3): Mirror 17: 0x9203e1d5 (stale) Mirror 18: 0x9203e1d5 (stale) Mirror 19: 0x9203e1d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd4b3: skipped 000588 1773686166.502692 mapwrite 0x00c94c thru 0x0194dc (0xcb91 bytes) 000589 1773686166.606828 write 0x016d1b thru 0x022395 (0xb67b bytes) 000591 1773686166.608916 read 0x000ac8 thru 0x005011 (0x454a bytes) 000593 1773686166.628340 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22396): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22396): Mirror 17: 0x66e3a960 (stale) Mirror 18: 0x66e3a960 (stale) Mirror 19: 0x66e3a960 (stale) CRC-32 checksum value for chunk [0, 0x22396): Mirror 1: 0x4281c0f2 Mirror 20: 0x4281c0f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22396: skipped 000595 1773686167.278805 mapwrite 0x03a728 thru 0x03bfe0 (0x18b9 bytes) 000596 1773686167.429357 write 0x004964 thru 0x00e55a (0x9bf7 bytes) 000600 1773686167.439648 trunc from 0x03bfe1 to 0x003b4d (0x38493 bytes) 000602 1773686167.459488 write 0x01d97f thru 0x0289eb (0xb06d bytes) 000603 1773686167.464782 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x289ec): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x289ec): Mirror 17: 0x64e583d2 (stale) Mirror 18: 0x64e583d2 (stale) Mirror 19: 0x64e583d2 (stale) Mirror 20: 0xb5364b1a (stale) CRC-32 checksum value for chunk [0, 0x289ec): Mirror 1: 0x10bb1709 Mirror 21: 0x10bb1709 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x289ec: skipped 000604 1773686168.078956 read 0x00006a thru 0x00e94b (0xe8e2 bytes) 000605 1773686168.085655 read 0x0241ff thru 0x0289eb (0x47ed bytes) 000606 1773686168.092452 trunc from 0x0289ec to 0x0377b0 (0xedc5 bytes) 000607 1773686168.174299 read 0x037516 thru 0x0377b0 (0x29b bytes) 000608 1773686168.182480 read 0x02cb4a thru 0x0377b0 (0xac67 bytes) 000609 1773686168.183286 mapread 0x034bda thru 0x0377b0 (0x2bd7 bytes) 000610 1773686168.189336 MIRROR_RESYNC 6 mirrors 000615 1773686168.877936 trunc from 0x0377b1 to 0x033a89 (0x3d27 bytes) 000616 1773686168.930956 mapread 0x033835 thru 0x033a89 (0x255 bytes) 000619 1773686168.939100 trunc from 0x033a8a to 0x037cc4 (0x423b bytes) 000620 1773686168.954284 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37cc5): Mirror 18: 0x13f4ed32 (stale) Mirror 19: 0x13f4ed32 (stale) Mirror 20: 0x13f4ed32 (stale) Mirror 21: 0x13f4ed32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cc5): Mirror 18: 0x13f4ed32 (stale) Mirror 19: 0x13f4ed32 (stale) Mirror 20: 0x13f4ed32 (stale) Mirror 21: 0x13f4ed32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cc5: skipped 000621 1773686169.178835 read 0x02728a thru 0x031001 (0x9d78 bytes) 000623 1773686169.187262 trunc from 0x037cc5 to 0x00c00e (0x2bcb6 bytes) 000624 1773686169.204970 mapread 0x0039aa thru 0x00a54f (0x6ba6 bytes) 000625 1773686169.222565 trunc from 0x00c00f to 0x01ddff (0x11df1 bytes) 000629 1773686169.253691 mapread 0x018a13 thru 0x01a29b (0x1889 bytes) 000630 1773686169.263921 mapwrite 0x018143 thru 0x01bc62 (0x3b20 bytes) 000633 1773686169.311407 mapread 0x0118c2 thru 0x01ddff (0xc53e bytes) 000637 1773686169.320168 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1de00): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1de00): Mirror 18: 0x688c3e6c (stale) Mirror 19: 0x688c3e6c (stale) Mirror 20: 0x688c3e6c (stale) Mirror 21: 0x688c3e6c (stale) CRC-32 checksum value for chunk [0, 0x1de00): Mirror 1: 0xef5b0500 Mirror 22: 0xef5b0500 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1de00: skipped 000639 1773686169.930137 mapread 0x00102c thru 0x00b4d3 (0xa4a8 bytes) 000640 1773686169.948551 write 0x022ae1 thru 0x024fef (0x250f bytes) 000641 1773686170.006314 write 0x003dbb thru 0x010215 (0xc45b bytes) 000643 1773686170.022296 trunc from 0x024ff0 to 0x03db08 (0x18b19 bytes) 000644 1773686170.091410 MIRROR_RESYNC 6 mirrors 000645 1773686170.812151 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000649 1773686171.033963 mapread 0x039d31 thru 0x03db08 (0x3dd8 bytes) 000650 1773686171.056307 mapwrite 0x00c94c thru 0x014285 (0x793a bytes) 000651 1773686171.156196 mapread 0x030883 thru 0x0351e9 (0x4967 bytes) 000653 1773686171.164063 read 0x007abd thru 0x00e9b4 (0x6ef8 bytes) 000654 1773686171.164549 trunc from 0x03db09 to 0x01b74a (0x223be bytes) 000657 1773686171.183948 mapread 0x017e5a thru 0x0190f4 (0x129b bytes) 000660 1773686171.195856 mapread 0x006285 thru 0x009841 (0x35bd bytes) 000663 1773686171.198587 write 0x01c5c4 thru 0x01ef6a (0x29a7 bytes) 000664 1773686171.200122 mapread 0x00b0ae thru 0x00b5b3 (0x506 bytes) 000668 1773686171.206469 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ef6b): Mirror 19: 0x3a965447 (stale) Mirror 20: 0x3a965447 (stale) Mirror 21: 0x3a965447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef6b): Mirror 19: 0x3a965447 (stale) Mirror 20: 0x3a965447 (stale) Mirror 21: 0x3a965447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef6b: skipped 000669 1773686171.428963 mapwrite 0x0101f8 thru 0x0140ed (0x3ef6 bytes) 000670 1773686171.466698 write 0x02849a thru 0x02cefe (0x4a65 bytes) 000672 1773686171.471110 mapwrite 0x03481b thru 0x036332 (0x1b18 bytes) 000673 1773686171.516657 read 0x0131db thru 0x02041f (0xd245 bytes) 000675 1773686171.522040 trunc from 0x036333 to 0x00898f (0x2d9a3 bytes) 000677 1773686171.533367 read 0x002fc3 thru 0x0076f9 (0x4737 bytes) 000678 1773686171.537478 trunc from 0x008990 to 0x030437 (0x27aa8 bytes) 000679 1773686171.543718 trunc from 0x030438 to 0x022765 (0xdcd2 bytes) 000681 1773686171.553836 read 0x0105c6 thru 0x01b3d3 (0xae0e bytes) 000682 1773686171.558989 mapread 0x01f1bf thru 0x022765 (0x35a7 bytes) 000683 1773686171.559916 mapwrite 0x018d31 thru 0x025d88 (0xd058 bytes) 000684 1773686171.614471 mapread 0x007c8b thru 0x013560 (0xb8d6 bytes) 000686 1773686171.617286 mapwrite 0x01c64b thru 0x0285fe (0xbfb4 bytes) 000687 1773686171.681129 MIRROR_RESYNC 4 mirrors 000690 1773686172.117242 write 0x02ffa0 thru 0x03bd2a (0xbd8b bytes) 000692 1773686172.171214 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd2b): Mirror 19: 0x49ff06f3 (stale) Mirror 20: 0x49ff06f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd2b): Mirror 19: 0x49ff06f3 (stale) Mirror 20: 0x49ff06f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd2b: skipped 000693 1773686172.394197 read 0x015d0b thru 0x016cb7 (0xfad bytes) 000694 1773686172.398387 write 0x01331d thru 0x0216f6 (0xe3da bytes) 000695 1773686172.406471 mapread 0x02e1dd thru 0x03a773 (0xc597 bytes) 000696 1773686172.418691 read 0x0120a7 thru 0x01831b (0x6275 bytes) 000698 1773686172.429169 MIRROR_RESYNC 3 mirrors 000699 1773686172.913060 trunc from 0x03bd2b to 0x03559f (0x678b bytes) 000702 1773686172.989625 write 0x0012fa thru 0x0086d4 (0x73db bytes) 000703 1773686173.005721 MIRROR_RESYNC 3 mirrors 000705 1773686173.337782 mapwrite 0x039fa0 thru 0x03d493 (0x34f4 bytes) 000706 1773686173.413185 trunc from 0x03d494 to 0x001cea (0x3b7a9 bytes) 000707 1773686173.436030 write 0x016c24 thru 0x0179d6 (0xdb3 bytes) 000708 1773686173.437124 mapread 0x016b40 thru 0x0179d6 (0xe97 bytes) 000709 1773686173.438121 write 0x03514f thru 0x038808 (0x36ba bytes) 000710 1773686173.442443 mapwrite 0x01785b thru 0x01bbf3 (0x4399 bytes) 000712 1773686173.497169 trunc from 0x038809 to 0x000375 (0x38493 bytes) 000714 1773686173.513985 mapread 0x000350 thru 0x000375 (0x26 bytes) 000715 1773686173.524076 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376): Mirror 20: 0x8d2e5c6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376): Mirror 20: 0x8d2e5c6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376: skipped 000716 1773686173.656346 mapwrite 0x00fa66 thru 0x0139dc (0x3f77 bytes) 000717 1773686173.738462 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x139dd): Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x139dd): Mirror 20: 0x8cc807f0 (stale) CRC-32 checksum value for chunk [0, 0x139dd): Mirror 1: 0x3d5701cb Mirror 21: 0x3d5701cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x139dd: skipped 000719 1773686174.131797 mapread 0x00a48c thru 0x00c36e (0x1ee3 bytes) 000724 1773686174.144260 write 0x00899c thru 0x009d9c (0x1401 bytes) 000727 1773686174.188650 mapwrite 0x00d51d thru 0x016f97 (0x9a7b bytes) 000728 1773686174.244101 mapwrite 0x011a0d thru 0x014cf3 (0x32e7 bytes) 000731 1773686174.277275 mapwrite 0x03c6d0 thru 0x03ffff (0x3930 bytes) 000732 1773686174.316990 write 0x01f18b thru 0x02bd02 (0xcb78 bytes) 000733 1773686174.324983 trunc from 0x040000 to 0x020a1f (0x1f5e0 bytes) 000734 1773686174.334148 MIRROR_RESYNC 3 mirrors 000735 1773686174.763030 write 0x03da35 thru 0x03e479 (0xa45 bytes) 000736 1773686174.801009 write 0x00d751 thru 0x0188ed (0xb19d bytes) 000737 1773686174.808701 mapread 0x019149 thru 0x024622 (0xb4da bytes) 000738 1773686174.818866 write 0x016cc0 thru 0x0218bf (0xac00 bytes) 000741 1773686174.821262 mapwrite 0x01e0bc thru 0x0220b0 (0x3ff5 bytes) 000742 1773686174.862968 mapread 0x015b5c thru 0x021e37 (0xc2dc bytes) 000743 1773686174.864223 mapwrite 0x0119c3 thru 0x0166b1 (0x4cef bytes) 000744 1773686174.909262 read 0x035b6f thru 0x03b25b (0x56ed bytes) 000745 1773686174.920161 mapwrite 0x0120f7 thru 0x0167e2 (0x46ec bytes) 000746 1773686174.950035 mapread 0x018b8c thru 0x019d68 (0x11dd bytes) 000747 1773686174.951139 mapread 0x028c33 thru 0x034f36 (0xc304 bytes) 000748 1773686174.966995 mapread 0x0156cc thru 0x016676 (0xfab bytes) 000750 1773686174.968399 trunc from 0x03e47a to 0x0049d2 (0x39aa7 bytes) 000751 1773686174.986818 write 0x00d459 thru 0x01c79b (0xf343 bytes) 000752 1773686174.993650 MIRROR_RESYNC 3 mirrors 000753 1773686175.451973 read 0x001b52 thru 0x010ccf (0xf17e bytes) 000755 1773686175.460228 write 0x024bfa thru 0x026afe (0x1f05 bytes) 000759 1773686175.495253 mapread 0x021e4e thru 0x026afe (0x4cb1 bytes) 000760 1773686175.512089 mapread 0x01049a thru 0x0125a7 (0x210e bytes) 000761 1773686175.517168 write 0x0200d0 thru 0x02d588 (0xd4b9 bytes) 000762 1773686175.518660 mapwrite 0x01e461 thru 0x021e1c (0x39bc bytes) 000763 1773686175.537788 trunc from 0x02d589 to 0x00e488 (0x1f100 bytes) 000765 1773686175.546141 trunc from 0x00e489 to 0x00dda7 (0x6e1 bytes) 000766 1773686175.554067 trunc from 0x00dda8 to 0x005e75 (0x7f32 bytes) 000768 1773686175.561882 write 0x02680c thru 0x02d6d5 (0x6eca bytes) 000770 1773686175.564652 mapread 0x020882 thru 0x0227db (0x1f5a bytes) 000775 1773686175.573241 mapwrite 0x014ec9 thru 0x019426 (0x455e bytes) 000776 1773686175.611769 trunc from 0x02d6d6 to 0x03ee4b (0x11776 bytes) 000778 1773686175.629081 mapread 0x03c4c6 thru 0x03ee4b (0x2986 bytes) 000779 1773686175.637739 write 0x015a43 thru 0x016359 (0x917 bytes) 000780 1773686175.640609 MIRROR_RESYNC 3 mirrors 000781 1773686176.009567 MIRROR_RESYNC 3 mirrors 000783 1773686176.206700 trunc from 0x03ee4c to 0x011e78 (0x2cfd3 bytes) 000784 1773686176.258537 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11e79): Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e79): Mirror 20: 0x5cb66c47 (stale) Mirror 21: 0x5cb66c47 (stale) CRC-32 checksum value for chunk [0, 0x11e79): Mirror 1: 0x5cb66c47 Mirror 22: 0x5cb66c47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e79: skipped 000785 1773686176.740305 trunc from 0x011e79 to 0x03cfbb (0x2b143 bytes) 000787 1773686176.796130 write 0x006c15 thru 0x00b98a (0x4d76 bytes) 000788 1773686176.807450 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfbc): Mirror 20: 0xeef9e76 (stale) Mirror 21: 0xeef9e76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfbc): Mirror 20: 0xeef9e76 (stale) Mirror 21: 0xeef9e76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfbc: skipped 000789 1773686177.053438 read 0x025fb7 thru 0x032f50 (0xcf9a bytes) 000790 1773686177.064070 read 0x009627 thru 0x015450 (0xbe2a bytes) 000792 1773686177.070463 trunc from 0x03cfbc to 0x011bda (0x2b3e1 bytes) 000793 1773686177.082696 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11bdb): Mirror 21: 0x966ecbdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bdb): Mirror 21: 0x966ecbdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bdb: skipped 000795 1773686177.243407 read 0x006d16 thru 0x011bda (0xaec5 bytes) 000796 1773686177.253935 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000797 1773686177.359388 read 0x000b94 thru 0x00ea8e (0xdefb bytes) 000800 1773686177.366143 read 0x00d07f thru 0x011bda (0x4b5c bytes) 000803 1773686177.371137 write 0x03684f thru 0x03a685 (0x3e37 bytes) 000804 1773686177.372492 read 0x003e41 thru 0x009e09 (0x5fc9 bytes) 000807 1773686177.374631 trunc from 0x03a686 to 0x030661 (0xa024 bytes) 000809 1773686177.384451 trunc from 0x030662 to 0x031fee (0x198d bytes) 000810 1773686177.397248 mapwrite 0x014a04 thru 0x01a47a (0x5a77 bytes) 000813 1773686177.434944 write 0x009f30 thru 0x00a2ee (0x3bf bytes) 000814 1773686177.435918 read 0x01335e thru 0x01aaa9 (0x774c bytes) 000816 1773686177.436981 mapwrite 0x02ae28 thru 0x02e9e0 (0x3bb9 bytes) 000818 1773686177.470116 mapwrite 0x0028e1 thru 0x0041a9 (0x18c9 bytes) 000822 1773686177.508852 mapread 0x01cdaf thru 0x0222c5 (0x5517 bytes) 000823 1773686177.511280 write 0x015e97 thru 0x01ce20 (0x6f8a bytes) 000824 1773686177.515155 mapwrite 0x03417d thru 0x03da2d (0x98b1 bytes) 000826 1773686177.562082 mapread 0x03ac88 thru 0x03da2d (0x2da6 bytes) 000827 1773686177.562870 read 0x008775 thru 0x008960 (0x1ec bytes) 000829 1773686177.568044 MIRROR_EXTEND to 2 mirrors 000830 1773686177.891631 mapwrite 0x00f5c9 thru 0x012347 (0x2d7f bytes) 000831 1773686177.991057 mapwrite 0x00f7c8 thru 0x01ba85 (0xc2be bytes) 000833 1773686178.036139 read 0x01858f thru 0x018ea0 (0x912 bytes) 000834 1773686178.036897 write 0x011a4b thru 0x01cdee (0xb3a4 bytes) 000837 1773686178.040004 mapwrite 0x03037d thru 0x035653 (0x52d7 bytes) 000838 1773686178.086937 read 0x03c487 thru 0x03da2d (0x15a7 bytes) 000839 1773686178.094973 trunc from 0x03da2e to 0x01f2c8 (0x1e765 bytes) 000840 1773686178.134702 mapwrite 0x01166a thru 0x012cd9 (0x1670 bytes) 000841 1773686178.163178 mapread 0x01b28f thru 0x01beef (0xc61 bytes) 000843 1773686178.164560 write 0x02211a thru 0x02b713 (0x95fa bytes) 000844 1773686178.165793 mapwrite 0x00750d thru 0x00e4ed (0x6fe1 bytes) 000845 1773686178.199160 mapwrite 0x021965 thru 0x030c42 (0xf2de bytes) 000847 1773686178.278454 write 0x0270f7 thru 0x0302d2 (0x91dc bytes) 000848 1773686178.279532 read 0x00d142 thru 0x014e5a (0x7d19 bytes) 000849 1773686178.280168 mapwrite 0x03790a thru 0x03ffff (0x86f6 bytes) 000850 1773686178.341662 write 0x008845 thru 0x00fe6e (0x762a bytes) 000852 1773686178.343037 trunc from 0x040000 to 0x023953 (0x1c6ac bytes) 000854 1773686178.356722 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23954): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23954): Mirror 2: 0x33e2bad8 (stale) CRC-32 checksum value for chunk [0, 0x23954): Mirror 1: 0xcbf5b473 Mirror 3: 0xcbf5b473 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23954: skipped 000855 1773686178.749669 mapread 0x0001ee thru 0x00a544 (0xa357 bytes) 000856 1773686178.769966 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000857 1773686178.942156 read 0x00df40 thru 0x01b6e7 (0xd7a8 bytes) 000860 1773686178.955144 MIRROR_RESYNC 2 mirrors 000863 1773686179.158323 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000864 1773686179.243029 mapwrite 0x023bbe thru 0x02b67f (0x7ac2 bytes) 000870 1773686179.353425 write 0x039383 thru 0x03ffff (0x6c7d bytes) 000873 1773686179.357137 trunc from 0x040000 to 0x03e3ab (0x1c54 bytes) 000877 1773686179.374905 trunc from 0x03e3ac to 0x02e8ae (0xfafd bytes) 000880 1773686179.391572 trunc from 0x02e8af to 0x027c91 (0x6c1d bytes) 000881 1773686179.414463 read 0x016040 thru 0x018350 (0x2311 bytes) 000887 1773686179.423163 mapread 0x00cd2b thru 0x01544d (0x8723 bytes) 000891 1773686179.441075 mapread 0x0059ea thru 0x0089d0 (0x2fe7 bytes) 000894 1773686179.443333 read 0x015312 thru 0x022a1c (0xd70b bytes) 000901 1773686179.450094 MIRROR_EXTEND to 2 mirrors 000902 1773686179.851230 mapwrite 0x03048b thru 0x0338b0 (0x3426 bytes) 000903 1773686179.959679 read 0x02ee79 thru 0x0338b0 (0x4a38 bytes) 000904 1773686179.964985 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x338b1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338b1): Mirror 2: 0x6d75ca03 (stale) CRC-32 checksum value for chunk [0, 0x338b1): Mirror 1: 0x9cc3e518 Mirror 3: 0x9cc3e518 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338b1: skipped 000906 1773686180.478161 mapwrite 0x02ed42 thru 0x03d270 (0xe52f bytes) 000907 1773686180.595641 mapread 0x02e0e9 thru 0x03d270 (0xf188 bytes) 000908 1773686180.596997 mapread 0x02a839 thru 0x03607a (0xb842 bytes) 000910 1773686180.601455 mapwrite 0x03f501 thru 0x03ffff (0xaff bytes) 000911 1773686180.636970 write 0x03f1ed thru 0x03ffff (0xe13 bytes) 000912 1773686180.637773 write 0x0038da thru 0x00af97 (0x76be bytes) 000914 1773686180.644338 trunc from 0x040000 to 0x00d2f0 (0x32d0f bytes) 000915 1773686180.654112 mapread 0x0022e6 thru 0x00d2f0 (0xb00b bytes) 000916 1773686180.655438 trunc from 0x00d2f1 to 0x017b2d (0xa83d bytes) 000917 1773686180.665448 mapwrite 0x012919 thru 0x021296 (0xe97e bytes) 000919 1773686180.745127 mapwrite 0x00e5cb thru 0x01550e (0x6f44 bytes) 000922 1773686180.781980 read 0x017da1 thru 0x01a61c (0x287c bytes) 000923 1773686180.783007 mapwrite 0x022531 thru 0x025734 (0x3204 bytes) 000925 1773686180.821243 mapread 0x010796 thru 0x019eeb (0x9756 bytes) 000926 1773686180.822438 mapwrite 0x01dece thru 0x0296ee (0xb821 bytes) 000929 1773686180.869905 trunc from 0x0296ef to 0x00d9d1 (0x1bd1d bytes) 000930 1773686180.888731 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd9d2): Mirror 2: 0x50043107 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd9d2): Mirror 2: 0x50043107 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd9d2: skipped 000931 1773686181.036364 mapwrite 0x0270a0 thru 0x029fe7 (0x2f48 bytes) 000934 1773686181.091106 read 0x016311 thru 0x01c619 (0x6309 bytes) 000939 1773686181.101040 mapread 0x000908 thru 0x006f51 (0x664a bytes) 000941 1773686181.110978 write 0x00c965 thru 0x0144c2 (0x7b5e bytes) 000942 1773686181.117694 mapread 0x00fc0f thru 0x01ec0b (0xeffd bytes) 000948 1773686181.139316 mapread 0x010c78 thru 0x01ab72 (0x9efb bytes) 000951 1773686181.140969 write 0x002a85 thru 0x00df76 (0xb4f2 bytes) 000952 1773686181.142056 mapwrite 0x01b47d thru 0x01f3bb (0x3f3f bytes) 000954 1773686181.174026 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000957 1773686181.241214 read 0x01f571 thru 0x0239bc (0x444c bytes) 000958 1773686181.248180 mapwrite 0x027e70 thru 0x036a49 (0xebda bytes) 000959 1773686181.318662 trunc from 0x036a4a to 0x01d806 (0x19243 bytes) 000960 1773686181.328562 write 0x00628c thru 0x012104 (0xbe79 bytes) 000961 1773686181.339024 mapread 0x000767 thru 0x00b46f (0xad09 bytes) 000962 1773686181.348446 mapread 0x0034e3 thru 0x006512 (0x3030 bytes) 000964 1773686181.349945 read 0x014114 thru 0x01d806 (0x96f3 bytes) 000965 1773686181.359097 trunc from 0x01d807 to 0x025a04 (0x81fe bytes) 000967 1773686181.369080 trunc from 0x025a05 to 0x029698 (0x3c94 bytes) 000968 1773686181.379735 mapwrite 0x011b77 thru 0x01abe5 (0x906f bytes) 000969 1773686181.434971 MIRROR_EXTEND to 2 mirrors 000971 1773686181.748999 trunc from 0x029699 to 0x024359 (0x533f bytes) 000974 1773686181.827172 read 0x002d0e thru 0x00825f (0x5552 bytes) 000975 1773686181.833896 trunc from 0x02435a to 0x00de98 (0x164c1 bytes) 000976 1773686181.848372 mapwrite 0x0033f0 thru 0x01043f (0xd050 bytes) 000978 1773686181.903123 write 0x03a639 thru 0x03ffff (0x59c7 bytes) 000980 1773686181.905426 mapwrite 0x00571b thru 0x00cce8 (0x75ce bytes) 000982 1773686181.943789 write 0x037b01 thru 0x03ffff (0x84ff bytes) 000985 1773686181.951960 trunc from 0x040000 to 0x0355a1 (0xaa5e bytes) 000988 1773686181.965966 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x270:0x0]: Invalid argument (22) 000989 1773686182.040136 trunc from 0x0355a2 to 0x006222 (0x2f37f bytes) 000990 1773686182.048446 mapwrite 0x026fe6 thru 0x02e814 (0x782f bytes) 000991 1773686182.116440 mapread 0x00bb3f thru 0x00cd4c (0x120e bytes) 000996 1773686182.127005 read 0x0123ab thru 0x01f7aa (0xd400 bytes) 000997 1773686182.132306 mapwrite 0x003f80 thru 0x00b3c9 (0x744a bytes) 000998 1773686182.159793 read 0x00290b thru 0x00971a (0x6e10 bytes) 000999 1773686182.160567 mapread 0x00e404 thru 0x01aa67 (0xc664 bytes) 001000 1773686182.162830 read 0x010b55 thru 0x0201ba (0xf666 bytes) All operations completed A-OK! PASS 70a (52s) == sanity-flr test 71: check mirror extend parent fid ==== 14:36:24 (1773686184) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 14:36:26 (1773686186) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111188 s, 18.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105234 s, 10.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 14:36:31 (1773686191) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to Extending file size to 6933376 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8171488 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8543168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8220672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8336608 .. failed Extending file size to 9074496 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8358240 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8462624 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8592608 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9248064 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9118944 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9216256 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58137 58138 58140 58143 58144 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) 192.168.203.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:37:39 (1773686259) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5922208 .. Extending file size to 1081408 .. failed Extending file size to 6266048 .. Extending file size to 5874464 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6668640 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 6814496 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7234208 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7003104 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7004096 .. Extending file size to 7455360 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7734656 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8011520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8684224 .. Extending file size to 9047936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9006208 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60703 60704 60706 60708 60710 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) 192.168.203.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:38:46 (1773686326) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 28672 3740672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 49152 7489536 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 3968 2204544 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 28672 3740672 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 49152 7489536 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0572694 s, 36.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241607 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4e645992 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:38:57 (1773686337) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0431814 s, 48.6 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 14:39:01 (1773686341) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x58:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:39:04 (1773686344) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0885096 s, 23.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x358:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x359:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:39:09 (1773686349) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0876593 s, 23.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:39:13 (1773686353) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11924 s, 17.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:39:39 (1773686379) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407441 s, 25.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35b:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:40:27 (1773686427) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0861446 s, 24.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:41:11 (1773686471) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg304-server: Pool lustre.test_204f created Adding targets to pool oleg304-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg304-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0707205 s, 29.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg304-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg304-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg304-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:42:12 (1773686532) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:42:15 (1773686535) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:42:19 (1773686539) oleg304-server: Pool lustre.test_206 created Adding targets to pool oleg304-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg304-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg304-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg304-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg304-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:42:36 (1773686556) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342655 s, 30.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014056 s, 74.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaede6de5 Mirror 3: 0xaede6de5 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:42:41 (1773686561) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:42:42 (1773686562) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:42:44 (1773686564) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0571668 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x25f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x25f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x25f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1bc0000 Read this (4 bytes): BBBB 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:43:02 (1773686582) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0101094 s, 810 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:43:05 (1773686585) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225835 s, 46.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x79 lmm_fid: [0x200000405:0x79:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x79 lmm_fid: [0x200000405:0x79:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611257 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 14:43:06.000000000 -0400 Modify: 2026-03-16 14:43:06.000000000 -0400 Change: 2026-03-16 14:43:06.000000000 -0400 Birth: 2026-03-16 14:43:05.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:43:09 (1773686589) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:43:10 (1773686590) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.229551 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194507 s, 211 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 16 14:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 16 14:43 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:43:14 (1773686594) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:43:17 (1773686597) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1139 sec =========== 14:43:21 (1773686601) === sanity-flr: start cleanup 14:43:22 (1773686602) === === sanity-flr: finish cleanup 14:43:25 (1773686605) ===