-----============= acceptance-small: sanity-flr ============----- Mon Dec 22 09:16:35 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg250-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:16:52 (1766413012) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ca7c983c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ca7c983c800.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:17:24 (1766413044) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8191] == sanity-flr test 0a: lfs mirror create with -N option == 09:17:27 (1766413047) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:17:44 (1766413064) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:17:47 (1766413067) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:17:51 (1766413071) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:18:15 (1766413095) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:18:18 (1766413098) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:18:21 (1766413101) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ca7c983c800.stats=clear osc.lustre-OST0001-osc-ffff8ca7c983c800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:18:30 (1766413110) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:18:41 (1766413121) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:18:49 (1766413129) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:18:52 (1766413132) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:19:01 (1766413141) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:19:07 (1766413147) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:19:14 (1766413154) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:19:20 (1766413160) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:19:30 (1766413170) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:19:36 (1766413176) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.47446 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124239 s, 8.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:19:46 (1766413186) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0986282 s, 31.9 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:19:55 (1766413195) Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:21:03 (1766413263) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.521901 s, 20.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:21:43 (1766413303) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:22:53 (1766413373) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.95878 s, 17.7 MB/s -rw-r--r-- 1 root root 52428800 Dec 22 09:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.039s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.049s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.349s user 0m0.000s sys 0m0.197s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca7c983c800.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:24:14 (1766413454) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:24:18 (1766413458) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:24:21 (1766413461) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:24:24 (1766413464) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:24:32 (1766413472) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.464157 s, 27.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:24:44 (1766413484) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219574 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018334 s, 57.2 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:24:50 (1766413490) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:24:58 (1766413498) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000775922 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00235032 s, 436 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000948625 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000876037 s, 1.2 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 09:25:06 (1766413506) create test files with size 15744225 .. b0b4d621fb134a99c225b6f9c0f83659 - 08bb32d1b2526081e5a8bc5937b02ec3 - 808a3c76733eef33d11e7e0f100b1b35 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 09:25:27 (1766413527) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:25:40 (1766413540) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:25:45 (1766413545) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750085 s, 14.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209819 s, 50.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x40:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 09:25:53 (1766413553) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0792924 s, 26.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.257014 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146275 s, 14.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 09:26:06 (1766413566) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:26:09 (1766413569) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:26:12 (1766413572) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218264 s, 48.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 09:26:19 (1766413579) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259739 s, 12.1 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:26:27 (1766413587) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:26:34 (1766413594) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295771 s, 35.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:26:43 (1766413603) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ca7ca308800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36366 s, 28.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8ca7ca3ef000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:26:57 (1766413617) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:27:31 (1766413651) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:28:00 (1766413680) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:28:04 (1766413684) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.148532 s, 21.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.132741 s, 23.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1513e43e (stale) Mirror 2: 0x1513e43e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 8cde97c179e2e23ad08c627ab66e0ac0 - mirror 1 checksum: 031bb6587262511364d6abb036e79415 - mirror 2 checksum: 8cde97c179e2e23ad08c627ab66e0ac0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 22 09:28:07 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 22 09:28:07 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8cde97c179e2e23ad08c627ab66e0ac0 - mirror 1 checksum: 031bb6587262511364d6abb036e79415 - mirror 2 checksum: 031bb6587262511364d6abb036e79415 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 22 09:28:07 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 22 09:28:07 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:28:15 (1766413695) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024714 s, 42.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:28:22 (1766413702) Failing mds1 on oleg250-server Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 09:28:34 (1766413714) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server Failover mds1 to oleg250-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 09:28:55 (1766413735) targets are mounted 09:28:55 (1766413735) facet_failover done oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314783 s, 33.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:29:10 (1766413750) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0402964 s, 508 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0102507 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:29:20 (1766413760) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0675045 s, 303 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0158705 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:29:31 (1766413771) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0121842 s, 1.3 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:29:40 (1766413780) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0480589 s, 426 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:29:47 (1766413787) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00930166 s, 440 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0322903 s, 127 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:29:55 (1766413795) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:30:19 (1766413819) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:30:48 (1766413848) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:31:11 (1766413871) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:31:34 (1766413894) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 09:31:41 (1766413901) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55892 mirror split pid 55893 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55892 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55893 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 09:32:48 (1766413968) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3970 000001 1766413970.595318 write 0x01a719 thru 0x02026e (0x5b56 bytes) 000002 1766413970.606049 mapwrite 0x025bdd thru 0x026457 (0x87b bytes) 000003 1766413970.645567 trunc from 0x00f645 to 0x01acb0 (0xb66c bytes) 000004 1766413970.651296 read 0x00319a thru 0x00ebfe (0xba65 bytes) 000005 1766413970.661133 mapwrite 0x03c42e thru 0x03ffff (0x3bd2 bytes) 000007 1766413970.710819 MIRROR_EXTEND to 2 mirrors 000008 1766413971.304074 read 0x01b2e0 thru 0x01cd4f (0x1a70 bytes) 000009 1766413971.316025 trunc from 0x007118 to 0x008bd7 (0x1ac0 bytes) 000010 1766413971.343124 mapread 0x006bbd thru 0x00ce3d (0x6281 bytes) 000011 1766413971.387819 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000012 1766413971.549863 write 0x00de02 thru 0x01bf99 (0xe198 bytes) 000014 1766413971.567410 punch from 0x00f5db to 0x01429a (0x4cc0 bytes) 000015 1766413971.584340 write 0x01563a thru 0x019e90 (0x4857 bytes) 000016 1766413971.585301 punch from 0x003929 to 0x007f8e (0x4666 bytes) fallocating to largest ever: 0x1bf29 000017 1766413971.590930 fallocate 0x01aa56 thru 0x01bf28 (0x14d3 bytes) zero_range to largest ever: 0x25235 000018 1766413971.596036 trunc from 0x015308 to 0x025234 (0xff2d bytes) 000019 1766413971.628430 trunc from 0x040000 to 0x022d75 (0x1d28a bytes) 000020 1766413971.674093 trunc from 0x019110 to 0x01a72c (0x161d bytes) 000021 1766413971.687125 MIRROR_EXTEND to 2 mirrors 000022 1766413972.318525 punch from 0x021c0e to 0x022d75 (0x1168 bytes) 000023 1766413972.363624 read 0x002028 thru 0x009d40 (0x7d19 bytes) 000024 1766413972.373664 trunc from 0x007b19 to 0x00e7c3 (0x6cab bytes) 000025 1766413972.378972 mapread 0x00f61b thru 0x0102ad (0xc93 bytes) truncating to largest ever: 0x360d7 000026 1766413972.393452 trunc from 0x022d76 to 0x0360d6 (0x13361 bytes) 000027 1766413972.417135 read 0x008dc6 thru 0x011973 (0x8bae bytes) 000028 1766413972.429373 trunc from 0x0360d7 to 0x0353ab (0xd2b bytes) 000029 1766413972.449462 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000030 1766413972.645552 read 0x017b29 thru 0x019e06 (0x22de bytes) 000031 1766413972.652441 mapread 0x0252c0 thru 0x02947e (0x41bf bytes) 000032 1766413972.674504 trunc from 0x00bf71 to 0x00e6a8 (0x2738 bytes) 000033 1766413972.686709 read 0x01eb37 thru 0x022c50 (0x411a bytes) 000035 1766413972.714222 write 0x008da4 thru 0x00f63e (0x689b bytes) 000036 1766413972.720806 mapwrite 0x010020 thru 0x013d82 (0x3d63 bytes) 000037 1766413972.743840 mapread 0x02db11 thru 0x02e8b3 (0xda3 bytes) 000038 1766413972.744676 trunc from 0x0353ac to 0x01bdc2 (0x195e9 bytes) 000039 1766413972.775436 mapread 0x00370e thru 0x005513 (0x1e06 bytes) 000041 1766413972.779515 read 0x007f86 thru 0x00cb8b (0x4c06 bytes) truncating to largest ever: 0x3d67d 000042 1766413972.781088 trunc from 0x01bdc3 to 0x03d67c (0x218ba bytes) 000043 1766413972.801235 trunc from 0x00b2d2 to 0x013e7b (0x8baa bytes) 000044 1766413972.807139 mapwrite 0x01c1af thru 0x01e42b (0x227d bytes) 000045 1766413972.852016 mapwrite 0x01b8db thru 0x0257e8 (0x9f0e bytes) 000046 1766413972.881487 trunc from 0x03d67d to 0x0247cf (0x18ead bytes) 000047 1766413972.901950 mapread 0x002632 thru 0x002abc (0x48b bytes) 000048 1766413972.903102 trunc from 0x0247d0 to 0x01a2d1 (0xa4fe bytes) 000049 1766413972.918823 trunc from 0x006760 to 0x00e9a7 (0x8248 bytes) 000050 1766413972.926315 mapread 0x00681b thru 0x008088 (0x186e bytes) 000052 1766413972.955750 mapread 0x006dd8 thru 0x010783 (0x99ac bytes) 000053 1766413972.989496 write 0x01a6f8 thru 0x0261e9 (0xbaf2 bytes) 000057 1766413972.995711 write 0x02ec0d thru 0x034100 (0x54f4 bytes) 000058 1766413972.999052 write 0x002a80 thru 0x01164e (0xebcf bytes) 000060 1766413973.002288 fallocate 0x01c5d9 thru 0x0271f7 (0xac1f bytes) 000062 1766413973.007990 trunc from 0x020b07 to 0x02f695 (0xeb8f bytes) 000063 1766413973.033818 read 0x0242c4 thru 0x02fe84 (0xbbc1 bytes) 000065 1766413973.045933 fallocate 0x021209 thru 0x023c06 (0x29fe bytes) 000066 1766413973.059646 mapwrite 0x02c661 thru 0x03376e (0x710e bytes) 000067 1766413973.074785 trunc from 0x0033cc to 0x00d7e3 (0xa418 bytes) 000068 1766413973.079511 mapwrite 0x031bf2 thru 0x03e7d6 (0xcbe5 bytes) 000069 1766413973.116086 trunc from 0x03e7d7 to 0x00f2ae (0x2f528 bytes) 000070 1766413973.130450 write 0x013b5c thru 0x016a2d (0x2ed2 bytes) 000071 1766413973.131384 fallocate 0x00f4ae thru 0x010587 (0x10da bytes) 000072 1766413973.135757 trunc from 0x016a2e to 0x0131fd (0x3830 bytes) 000073 1766413973.155117 punch from 0x0052a4 to 0x005c89 (0x9e6 bytes) 000074 1766413973.163522 punch from 0x00dc46 to 0x01044b (0x2806 bytes) 000075 1766413973.171203 mapwrite 0x02893f thru 0x034e6a (0xc52c bytes) 000076 1766413973.307006 fallocate 0x025ac7 thru 0x02d466 (0x79a0 bytes) 000077 1766413973.313681 mapread 0x02580c thru 0x02e458 (0x8c4d bytes) 000078 1766413973.333595 punch from 0x0141d7 to 0x01bf29 (0x7d53 bytes) 000080 1766413973.338814 read 0x02c7d7 thru 0x034e6a (0x8694 bytes) 000081 1766413973.339368 trunc from 0x034e6b to 0x015c0c (0x1f25e bytes) 000082 1766413973.353015 trunc from 0x015c0d to 0x03ccef (0x270e3 bytes) 000083 1766413973.371972 write 0x02ed50 thru 0x03be60 (0xd111 bytes) 000084 1766413973.383443 read 0x02ee4a thru 0x03ccef (0xdea6 bytes) 000085 1766413973.384070 trunc from 0x002a1c to 0x010c6d (0xe252 bytes) 000086 1766413973.389205 mapread 0x0383be thru 0x0387e6 (0x429 bytes) 000087 1766413973.391627 fallocate 0x027356 thru 0x027994 (0x63f bytes) 000088 1766413973.397625 trunc from 0x03ccf0 to 0x0087a6 (0x34549 bytes) 000089 1766413973.416675 trunc from 0x004e85 to 0x0087a6 (0x3922 bytes) 000090 1766413973.421147 write 0x00ae15 thru 0x0142a0 (0x948c bytes) 000092 1766413973.423754 fallocate 0x016b43 thru 0x0198f1 (0x2daf bytes) 000093 1766413973.429053 read 0x000ddb thru 0x007828 (0x6a4e bytes) 000094 1766413973.438760 read 0x00dbd7 thru 0x0142a0 (0x66ca bytes) 000095 1766413973.439357 fallocate 0x01d1ad thru 0x0297b7 (0xc60b bytes) 000096 1766413973.445039 punch from 0x0121ac to 0x0142a0 (0x20f5 bytes) 000098 1766413973.462472 trunc from 0x006765 to 0x01364d (0xcee9 bytes) 000100 1766413973.470571 read 0x00521e thru 0x00b7dd (0x65c0 bytes) 000101 1766413973.482406 trunc from 0x0142a1 to 0x0301bb (0x1bf1b bytes) 000102 1766413973.509935 trunc from 0x020411 to 0x02a93b (0xa52b bytes) 000103 1766413973.531142 write 0x016b9e thru 0x020a56 (0x9eb9 bytes) 000104 1766413973.566917 mapwrite 0x01c52a thru 0x0288ca (0xc3a1 bytes) 000105 1766413973.613144 write 0x00e2a0 thru 0x00fdcb (0x1b2c bytes) 000106 1766413973.613990 read 0x005acc thru 0x00d947 (0x7e7c bytes) 000107 1766413973.615306 trunc from 0x000f2c to 0x008c29 (0x7cfe bytes) 000109 1766413973.619824 mapwrite 0x02d596 thru 0x037514 (0x9f7f bytes) 000110 1766413973.690558 write 0x008c5d thru 0x00eec7 (0x626b bytes) 000112 1766413973.693808 write 0x026030 thru 0x032af6 (0xcac7 bytes) 000113 1766413973.698480 mapread 0x031948 thru 0x037514 (0x5bcd bytes) 000114 1766413973.705564 fallocate 0x02a769 thru 0x02ceaa (0x2742 bytes) 000115 1766413973.721874 punch from 0x02006f to 0x0220c7 (0x2059 bytes) truncating to largest ever: 0x3f0bb 000116 1766413973.735973 trunc from 0x037515 to 0x03f0ba (0x7ba6 bytes) 000117 1766413973.765053 read 0x024c37 thru 0x032458 (0xd822 bytes) 000118 1766413973.765758 mapwrite 0x015b53 thru 0x018d3a (0x31e8 bytes) 000120 1766413973.797437 trunc from 0x03f0bb to 0x03e60e (0xaac bytes) 000121 1766413973.817318 trunc from 0x03e60f to 0x02628b (0x18383 bytes) 000122 1766413973.834388 mapread 0x00ba9a thru 0x00c4e6 (0xa4d bytes) 000123 1766413973.835231 trunc from 0x02628c to 0x0050a0 (0x211eb bytes) 000124 1766413973.855509 write 0x007100 thru 0x00a496 (0x3397 bytes) 000125 1766413973.856887 mapread 0x00a17a thru 0x00a496 (0x31d bytes) 000126 1766413973.858796 write 0x03ff5f thru 0x03ffff (0xa1 bytes) 000127 1766413973.860637 trunc from 0x00dde0 to 0x01c795 (0xe9b6 bytes) 000128 1766413973.867379 trunc from 0x001565 to 0x00c406 (0xaea2 bytes) 000129 1766413973.895048 trunc from 0x0076cc to 0x010a9b (0x93d0 bytes) 000130 1766413973.898681 trunc from 0x040000 to 0x01238a (0x2dc75 bytes) 000131 1766413973.910876 trunc from 0x003dba to 0x00e888 (0xaacf bytes) 000132 1766413973.916253 punch from 0x00df69 to 0x01238a (0x4422 bytes) 000133 1766413973.920952 trunc from 0x01238b to 0x029b9c (0x17812 bytes) 000136 1766413973.941228 punch from 0x001b48 to 0x004f7b (0x3434 bytes) 000137 1766413973.949808 mapread 0x0100a2 thru 0x018f03 (0x8e62 bytes) 000139 1766413973.960677 mapwrite 0x03ac81 thru 0x03ffff (0x537f bytes) 000141 1766413974.007738 read 0x000cf0 thru 0x006bb1 (0x5ec2 bytes) 000142 1766413974.008374 write 0x025024 thru 0x0306c9 (0xb6a6 bytes) 000143 1766413974.010715 trunc from 0x025cdd to 0x02eb5a (0x8e7e bytes) 000144 1766413974.026598 fallocate 0x0099c9 thru 0x01586e (0xbea6 bytes) 000145 1766413974.036305 trunc from 0x0244cf to 0x031a22 (0xd554 bytes) 000146 1766413974.046857 mapread 0x0148c8 thru 0x01cb5c (0x8295 bytes) 000147 1766413974.049349 trunc from 0x040000 to 0x01544b (0x2abb4 bytes) 000148 1766413974.068610 mapread 0x00f1c8 thru 0x0116c8 (0x2501 bytes) 000149 1766413974.069531 mapread 0x01011f thru 0x01105f (0xf41 bytes) 000150 1766413974.070282 trunc from 0x01544c to 0x00c83a (0x8c11 bytes) 000151 1766413974.085568 trunc from 0x0009b7 to 0x00c83a (0xbe84 bytes) 000152 1766413974.099049 mapread 0x005d7e thru 0x00c83a (0x6abd bytes) 000154 1766413974.111385 mapwrite 0x022bea thru 0x025335 (0x274c bytes) 000155 1766413974.169111 trunc from 0x00b429 to 0x00bb39 (0x711 bytes) 000156 1766413974.174198 trunc from 0x025336 to 0x015387 (0xffae bytes) 000157 1766413974.188485 fallocate 0x034bbe thru 0x03ce36 (0x8279 bytes) 000158 1766413974.193197 mapwrite 0x018407 thru 0x026218 (0xde12 bytes) 000159 1766413974.327819 trunc from 0x00488b to 0x00e2cb (0x9a41 bytes) 000160 1766413974.335534 write 0x036a54 thru 0x03ffff (0x95ac bytes) 000161 1766413974.336849 fallocate 0x00d3a5 thru 0x00ffc6 (0x2c22 bytes) 000162 1766413974.341818 punch from 0x00093b to 0x009f4d (0x9613 bytes) 000163 1766413974.351024 MIRROR_EXTEND to 2 mirrors 000164 1766413975.100132 punch from 0x03c23d to 0x03ffff (0x3dc3 bytes) 000165 1766413975.144291 mapwrite 0x0390ce thru 0x03ffff (0x6f32 bytes) 000166 1766413975.197052 trunc from 0x040000 to 0x03941f (0x6be0 bytes) 000167 1766413975.251026 trunc from 0x018b09 to 0x021d0a (0x9202 bytes) 000168 1766413975.257305 mapread 0x02af21 thru 0x03941f (0xe4ff bytes) 000169 1766413975.275920 trunc from 0x039420 to 0x02520f (0x14210 bytes) 000170 1766413975.295435 trunc from 0x025210 to 0x01539c (0xfe73 bytes) 000171 1766413975.317824 mapread 0x003013 thru 0x007d70 (0x4d5e bytes) 000172 1766413975.327338 trunc from 0x01539d to 0x03abf4 (0x25858 bytes) 000173 1766413975.351360 mapwrite 0x03fca9 thru 0x03ffff (0x357 bytes) 000174 1766413975.407972 trunc from 0x040000 to 0x0375de (0x8a21 bytes) 000175 1766413975.425760 trunc from 0x01aa34 to 0x01bdb3 (0x1380 bytes) 000176 1766413975.439051 MIRROR_RESYNC 2 mirrors 000178 1766413976.030962 trunc from 0x01cc91 to 0x01df26 (0x1296 bytes) 000179 1766413976.065711 mapread 0x00b4bc thru 0x011300 (0x5e45 bytes) 000180 1766413976.098742 write 0x03baba thru 0x03ffff (0x4546 bytes) 000181 1766413976.106818 MIRROR_RESYNC 2 mirrors 000182 1766413976.575798 MIRROR_RESYNC 2 mirrors 000183 1766413976.994509 trunc from 0x040000 to 0x0074a8 (0x38b57 bytes) 000184 1766413977.082793 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000185 1766413977.316644 punch from 0x000392 to 0x0074a8 (0x7117 bytes) 000186 1766413977.322753 mapread 0x00677a thru 0x0074a8 (0xd2f bytes) 000188 1766413977.339636 fallocate 0x02553d thru 0x02676e (0x1232 bytes) 000189 1766413977.349730 mapread 0x001d68 thru 0x0044df (0x2778 bytes) 000191 1766413977.364909 punch from 0x023142 to 0x02393d (0x7fc bytes) 000193 1766413977.371039 punch from 0x00f7c2 to 0x018789 (0x8fc8 bytes) 000195 1766413977.382533 mapread 0x00f31c thru 0x012227 (0x2f0c bytes) 000197 1766413977.398943 read 0x01f850 thru 0x023c31 (0x43e2 bytes) 000198 1766413977.415731 read 0x019881 thru 0x02398e (0xa10e bytes) 000200 1766413977.427727 write 0x02b386 thru 0x02df7e (0x2bf9 bytes) 000201 1766413977.436137 trunc from 0x02df7f to 0x02af83 (0x2ffb bytes) 000202 1766413977.453673 fallocate 0x032cf7 thru 0x03d738 (0xaa42 bytes) 000203 1766413977.465860 MIRROR_EXTEND to 2 mirrors 000204 1766413977.942091 write 0x00d76b thru 0x018296 (0xab2c bytes) 000205 1766413978.008114 trunc from 0x020b16 to 0x027dde (0x72c9 bytes) 000206 1766413978.014552 mapread 0x024e70 thru 0x027c2a (0x2dbb bytes) 000207 1766413978.028972 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000208 1766413978.213433 mapwrite 0x018e23 thru 0x01dfd5 (0x51b3 bytes) 000209 1766413978.267005 fallocate 0x031ae1 thru 0x037cb0 (0x61d0 bytes) 000210 1766413978.282141 mapread 0x01b295 thru 0x01f708 (0x4474 bytes) 000211 1766413978.284807 mapwrite 0x01e949 thru 0x024c6b (0x6323 bytes) 000212 1766413978.332701 trunc from 0x00dac3 to 0x0161eb (0x8729 bytes) 000213 1766413978.342743 trunc from 0x02af84 to 0x01f99f (0xb5e4 bytes) 000214 1766413978.398602 read 0x002e16 thru 0x007300 (0x44eb bytes) 000216 1766413978.414682 trunc from 0x01f9a0 to 0x02f59b (0xfbfc bytes) 000217 1766413978.438448 write 0x02597f thru 0x0261d0 (0x852 bytes) 000218 1766413978.453586 punch from 0x0064cc to 0x009ab5 (0x35ea bytes) 000219 1766413978.460113 mapread 0x02dbaa thru 0x02f59b (0x19f2 bytes) 000220 1766413978.476910 trunc from 0x02f59c to 0x010e1f (0x1e77c bytes) 000221 1766413978.500657 write 0x016084 thru 0x021af3 (0xba70 bytes) 000222 1766413978.502483 fallocate 0x03996e thru 0x03c06f (0x2702 bytes) 000223 1766413978.507807 punch from 0x002d0a to 0x002fdc (0x2d3 bytes) 000225 1766413978.512128 trunc from 0x03c070 to 0x0154aa (0x26bc5 bytes) 000226 1766413978.536879 mapwrite 0x01e917 thru 0x02435e (0x5a48 bytes) 000227 1766413978.593285 trunc from 0x0011fc to 0x00a611 (0x9416 bytes) 000228 1766413978.599125 read 0x0156ca thru 0x01cd79 (0x76b0 bytes) 000229 1766413978.607156 write 0x0230b7 thru 0x030841 (0xd78b bytes) 000230 1766413978.608852 mapread 0x027ffd thru 0x02a215 (0x2219 bytes) 000231 1766413978.610282 trunc from 0x01673c to 0x01d235 (0x6afa bytes) 000232 1766413978.614682 fallocate 0x02e302 thru 0x03abef (0xc8ee bytes) 000233 1766413978.619751 trunc from 0x0018c7 to 0x00b339 (0x9a73 bytes) 000234 1766413978.625770 trunc from 0x028333 to 0x02cd0e (0x49dc bytes) 000235 1766413978.652603 trunc from 0x030842 to 0x005147 (0x2b6fa bytes) 000237 1766413978.675062 mapread 0x002610 thru 0x005147 (0x2b38 bytes) 000238 1766413978.696486 mapwrite 0x00411e thru 0x00ab3d (0x6a20 bytes) 000239 1766413978.777565 trunc from 0x00ab3e to 0x031a4f (0x26f12 bytes) 000240 1766413978.812414 mapwrite 0x017822 thru 0x01f924 (0x8103 bytes) 000241 1766413978.921652 trunc from 0x031a50 to 0x005962 (0x2c0ed bytes) 000242 1766413978.946805 fallocate 0x00233e thru 0x006bb1 (0x4874 bytes) 000243 1766413978.955536 trunc from 0x006bb2 to 0x029afe (0x22f4d bytes) 000244 1766413978.984811 mapread 0x00371e thru 0x00efd1 (0xb8b4 bytes) 000245 1766413979.030924 fallocate 0x00955d thru 0x012c6d (0x9711 bytes) 000246 1766413979.035291 trunc from 0x00c2c3 to 0x00dccf (0x1a0d bytes) 000247 1766413979.046468 MIRROR_EXTEND to 2 mirrors 000248 1766413979.665574 mapwrite 0x03731d thru 0x03a7d7 (0x34bb bytes) 000249 1766413979.738678 write 0x027d1b thru 0x02a84e (0x2b34 bytes) 000250 1766413979.747423 write 0x01d62c thru 0x029f43 (0xc918 bytes) 000251 1766413979.757042 trunc from 0x03a7d8 to 0x03cb71 (0x239a bytes) 000252 1766413979.775429 write 0x03c4bd thru 0x03ffff (0x3b43 bytes) 000253 1766413979.781822 fallocate 0x012432 thru 0x0201e9 (0xddb8 bytes) 000254 1766413979.789679 punch from 0x00e6ba to 0x01e433 (0xfd7a bytes) 000256 1766413979.804868 trunc from 0x040000 to 0x03ee8a (0x1175 bytes) 000257 1766413979.820033 trunc from 0x020582 to 0x028789 (0x8208 bytes) 000258 1766413979.829723 punch from 0x034a93 to 0x03ee8a (0xa3f8 bytes) 000259 1766413979.839225 mapwrite 0x023e4a thru 0x029a8c (0x5c43 bytes) 000260 1766413979.883115 fallocate 0x017553 thru 0x027049 (0xfaf7 bytes) 000261 1766413979.898536 fallocate 0x0219fe thru 0x031834 (0xfe37 bytes) 000262 1766413979.908798 trunc from 0x03a7d8 to 0x03ee8a (0x46b3 bytes) 000264 1766413979.924794 mapwrite 0x015ad2 thru 0x01c8d3 (0x6e02 bytes) 000265 1766413979.961578 read 0x034090 thru 0x03ee8a (0xadfb bytes) 000266 1766413979.970833 write 0x03db2b thru 0x03ffff (0x24d5 bytes) 000267 1766413979.971795 trunc from 0x040000 to 0x016bb3 (0x2944c bytes) 000268 1766413979.986873 mapwrite 0x010890 thru 0x017932 (0x70a3 bytes) 000269 1766413980.026416 punch from 0x008940 to 0x00cf70 (0x4631 bytes) 000271 1766413980.031604 mapwrite 0x0352f0 thru 0x03ffff (0xad10 bytes) 000272 1766413980.077990 mapwrite 0x011905 thru 0x0204f4 (0xebf0 bytes) 000273 1766413980.115867 trunc from 0x021dc1 to 0x0261b0 (0x43f0 bytes) 000274 1766413980.133704 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x134d3341 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a379dfb Mirror 3: 0x8a379dfb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000275 1766413981.158018 fallocate 0x01caec thru 0x02ba06 (0xef1b bytes) 000276 1766413981.214096 write 0x00f5b5 thru 0x019307 (0x9d53 bytes) 000277 1766413981.228465 read 0x00cbe5 thru 0x01478e (0x7baa bytes) 000278 1766413981.233099 trunc from 0x028cc7 to 0x035537 (0xc871 bytes) 000279 1766413981.242680 mapread 0x00ca6b thru 0x0177fa (0xad90 bytes) 000280 1766413981.243646 trunc from 0x006d12 to 0x00a0cd (0x33bc bytes) 000281 1766413981.253603 write 0x0262d6 thru 0x027fd6 (0x1d01 bytes) 000282 1766413981.265326 trunc from 0x03d4a0 to 0x03ffff (0x2b60 bytes) 000283 1766413981.280346 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8a379dfb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8a379dfb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000284 1766413981.553049 fallocate 0x004900 thru 0x0064d7 (0x1bd8 bytes) 000286 1766413981.566542 mapread 0x033f8a thru 0x03e84b (0xa8c2 bytes) 000287 1766413981.617849 punch from 0x0283c5 to 0x031f2c (0x9b68 bytes) 000291 1766413981.639586 MIRROR_RESYNC 2 mirrors 000292 1766413982.580852 mapread 0x03f581 thru 0x03ffff (0xa7f bytes) 000293 1766413982.586030 trunc from 0x040000 to 0x0169e8 (0x29617 bytes) 000294 1766413982.657487 trunc from 0x0001bb to 0x009fff (0x9e45 bytes) 000295 1766413982.671286 mapread 0x00fe94 thru 0x0169e8 (0x6b55 bytes) 000296 1766413982.687570 read 0x004652 thru 0x01343a (0xede9 bytes) 000297 1766413982.710868 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000298 1766413982.902397 trunc from 0x0169e9 to 0x017ade (0x10f6 bytes) 000299 1766413982.947685 trunc from 0x017adf to 0x0294d7 (0x119f9 bytes) 000300 1766413982.968076 mapread 0x01530d thru 0x020da2 (0xba96 bytes) 000301 1766413983.031097 mapread 0x01e0eb thru 0x02304a (0x4f60 bytes) 000302 1766413983.032720 trunc from 0x005df6 to 0x00e779 (0x8984 bytes) 000303 1766413983.038430 fallocate 0x016c89 thru 0x017307 (0x67f bytes) 000304 1766413983.048862 trunc from 0x024f3c to 0x0294d7 (0x459c bytes) 000305 1766413983.055968 punch from 0x00b7d2 to 0x011cee (0x651d bytes) 000306 1766413983.061739 read 0x00717e thru 0x014927 (0xd7aa bytes) 000307 1766413983.075824 MIRROR_EXTEND to 2 mirrors 000308 1766413983.811913 fallocate 0x03755e thru 0x03d3e0 (0x5e83 bytes) 000309 1766413983.848480 trunc from 0x00995a to 0x011540 (0x7be7 bytes) 000310 1766413983.864365 trunc from 0x02bcdd to 0x03a046 (0xe36a bytes) 000311 1766413983.872278 fallocate 0x005c72 thru 0x005de7 (0x176 bytes) 000312 1766413983.877835 mapread 0x015034 thru 0x020e2b (0xbdf8 bytes) 000313 1766413983.918411 mapread 0x016c85 thru 0x020154 (0x94d0 bytes) 000315 1766413983.919713 mapread 0x000635 thru 0x000bb9 (0x585 bytes) 000316 1766413983.935752 trunc from 0x03d3e1 to 0x03c795 (0xc4b bytes) 000317 1766413983.982357 trunc from 0x02b309 to 0x039e41 (0xeb39 bytes) 000318 1766413983.990281 trunc from 0x00bb98 to 0x0123e6 (0x684f bytes) 000319 1766413983.996793 trunc from 0x020cde to 0x021fbd (0x12e0 bytes) 000320 1766413984.008515 punch from 0x01ceda to 0x02b9f0 (0xeb17 bytes) 000321 1766413984.018417 punch from 0x01f103 to 0x02e37e (0xf27c bytes) 000322 1766413984.035447 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c796): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c796): Mirror 2: 0x43c095c9 (stale) CRC-32 checksum value for chunk [0, 0x3c796): Mirror 1: 0x61d58e33 Mirror 3: 0x61d58e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c796: skipped fallocating to largest ever: 0x40000 000323 1766413984.802983 fallocate 0x03bcbb thru 0x03ffff (0x4345 bytes) 000324 1766413984.837624 trunc from 0x040000 to 0x01384c (0x2c7b3 bytes) 000325 1766413984.871772 trunc from 0x01045d to 0x012166 (0x1d0a bytes) 000326 1766413984.876915 mapread 0x00d2a0 thru 0x00e787 (0x14e8 bytes) 000327 1766413984.887997 mapread 0x00f949 thru 0x01384c (0x3f04 bytes) 000328 1766413984.896899 mapread 0x00f0a7 thru 0x01384c (0x47a6 bytes) 000330 1766413984.898222 mapread 0x0050bc thru 0x01018d (0xb0d2 bytes) 000331 1766413984.905506 trunc from 0x001994 to 0x00b0b2 (0x971f bytes) 000332 1766413984.911834 write 0x0332af thru 0x03574f (0x24a1 bytes) 000333 1766413984.913672 write 0x022ee8 thru 0x02435a (0x1473 bytes) 000334 1766413984.922773 trunc from 0x01be0b to 0x02941e (0xd614 bytes) 000335 1766413984.933539 punch from 0x02f6b3 to 0x032794 (0x30e2 bytes) 000336 1766413984.940036 trunc from 0x012ba2 to 0x013cb6 (0x1115 bytes) 000337 1766413984.946001 write 0x00703c thru 0x007bce (0xb93 bytes) 000338 1766413984.961093 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35750): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35750): Mirror 2: 0x3eaafbef (stale) Mirror 3: 0x7c7b1756 (stale) CRC-32 checksum value for chunk [0, 0x35750): Mirror 1: 0x9733bbe8 Mirror 4: 0x9733bbe8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35750: skipped 000339 1766413985.770963 trunc from 0x0270ab to 0x0301b4 (0x910a bytes) 000340 1766413985.814127 punch from 0x01c4e9 to 0x0200c2 (0x3bda bytes) 000341 1766413985.822856 fallocate 0x0235a0 thru 0x02c20e (0x8c6f bytes) 000342 1766413985.827279 punch from 0x0268ea to 0x02813c (0x1853 bytes) 000343 1766413985.844294 trunc from 0x035750 to 0x00c3bc (0x29393 bytes) 000345 1766413985.870496 mapread 0x001687 thru 0x007306 (0x5c80 bytes) 000346 1766413985.892370 punch from 0x00acb4 to 0x00c3bc (0x1709 bytes) 000347 1766413985.899049 trunc from 0x0027d6 to 0x00c3bc (0x9be7 bytes) 000348 1766413985.904912 mapread 0x0019eb thru 0x00c3bc (0xa9d2 bytes) 000349 1766413985.927472 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xc3bd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3bd): Mirror 2: 0xc02ba3a1 (stale) Mirror 3: 0xc02ba3a1 (stale) Mirror 4: 0x60525ef2 (stale) CRC-32 checksum value for chunk [0, 0xc3bd): Mirror 1: 0xc02ba3a1 Mirror 5: 0xc02ba3a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3bd: skipped 000350 1766413986.607068 fallocate 0x0384c0 thru 0x03c74c (0x428d bytes) 000351 1766413986.639393 mapread 0x009cdc thru 0x00c3bc (0x26e1 bytes) 000352 1766413986.658011 punch from 0x006029 to 0x00c3bc (0x6394 bytes) 000353 1766413986.672849 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc3bd): Mirror 3: 0xc02ba3a1 (stale) Mirror 4: 0x60525ef2 (stale) Mirror 5: 0xc02ba3a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3bd): Mirror 3: 0xc02ba3a1 (stale) Mirror 4: 0x60525ef2 (stale) Mirror 5: 0xc02ba3a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3bd: skipped 000354 1766413986.987945 mapwrite 0x03b9d4 thru 0x03ffff (0x462c bytes) 000355 1766413987.045360 mapwrite 0x01f221 thru 0x022502 (0x32e2 bytes) 000356 1766413987.067551 write 0x03427e thru 0x03cb3d (0x88c0 bytes) 000357 1766413987.077061 write 0x026cdc thru 0x02dcbd (0x6fe2 bytes) 000358 1766413987.078405 mapwrite 0x014f71 thru 0x01f53b (0xa5cb bytes) 000359 1766413987.107551 punch from 0x00c243 to 0x012769 (0x6527 bytes) 000360 1766413987.113786 write 0x030b8d thru 0x03916c (0x85e0 bytes) 000361 1766413987.115001 trunc from 0x00ea62 to 0x014981 (0x5f20 bytes) 000362 1766413987.124741 write 0x01ee97 thru 0x023098 (0x4202 bytes) 000363 1766413987.130797 punch from 0x037c80 to 0x03ec37 (0x6fb8 bytes) 000364 1766413987.165546 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6f3436d4 (stale) Mirror 4: 0x23ee4774 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96d61e3b Mirror 6: 0x96d61e3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000365 1766413988.068337 trunc from 0x040000 to 0x03f0a0 (0xf5f bytes) 000366 1766413988.129677 punch from 0x02fce1 to 0x032659 (0x2979 bytes) 000367 1766413988.134458 trunc from 0x03f0a1 to 0x0365c9 (0x8ad7 bytes) 000368 1766413988.151055 trunc from 0x00a9a9 to 0x013d1d (0x9375 bytes) 000369 1766413988.156268 punch from 0x03213f to 0x033523 (0x13e5 bytes) 000370 1766413988.161891 read 0x01d6b1 thru 0x020615 (0x2f65 bytes) 000371 1766413988.167844 mapread 0x00c02d thru 0x014d06 (0x8cda bytes) 000372 1766413988.186000 read 0x00ed39 thru 0x018b0c (0x9dd4 bytes) 000373 1766413988.194840 mapread 0x019e6b thru 0x0249f4 (0xab8a bytes) 000374 1766413988.195866 read 0x00d425 thru 0x019319 (0xbef5 bytes) 000375 1766413988.205762 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x365ca): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x365ca): Mirror 3: 0xd5f68898 (stale) Mirror 4: 0xeadf0bca (stale) Mirror 5: 0x7d5191c7 (stale) Mirror 6: 0x423391 (stale) CRC-32 checksum value for chunk [0, 0x365ca): Mirror 1: 0xaf12416f Mirror 7: 0xaf12416f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x365ca: skipped 000376 1766413989.077772 mapwrite 0x027c61 thru 0x03604d (0xe3ed bytes) 000377 1766413989.213411 fallocate 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000378 1766413989.219662 trunc from 0x040000 to 0x00124d (0x3edb2 bytes) 000379 1766413989.286780 trunc from 0x000627 to 0x00124d (0xc27 bytes) 000380 1766413989.298385 trunc from 0x0002e0 to 0x00124d (0xf6e bytes) 000381 1766413989.308129 read 0x0003a3 thru 0x00124d (0xeab bytes) 000382 1766413989.316014 write 0x01c448 thru 0x0286ff (0xc2b8 bytes) 000383 1766413989.321397 write 0x03f70c thru 0x03ffff (0x8f4 bytes) 000384 1766413989.322783 write 0x0187f8 thru 0x01ca29 (0x4232 bytes) 000385 1766413989.339722 trunc from 0x040000 to 0x021f3c (0x1e0c3 bytes) 000386 1766413989.373945 fallocate 0x03c2c4 thru 0x03ffff (0x3d3c bytes) 000387 1766413989.382845 read 0x008405 thru 0x00f19b (0x6d97 bytes) 000388 1766413989.394372 read 0x03aa9f thru 0x03e2d2 (0x3834 bytes) 000389 1766413989.395006 mapwrite 0x026559 thru 0x031edb (0xb983 bytes) 000390 1766413989.421052 read 0x00124d thru 0x002c7e (0x1a32 bytes) 000391 1766413989.421702 trunc from 0x040000 to 0x02c42f (0x13bd0 bytes) 000392 1766413989.446396 read 0x01767a thru 0x01f936 (0x82bd bytes) 000393 1766413989.447782 mapwrite 0x0283f1 thru 0x037aa2 (0xf6b2 bytes) 000394 1766413989.492750 mapread 0x00de23 thru 0x019234 (0xb412 bytes) 000395 1766413989.494300 punch from 0x00ed0a to 0x01dde8 (0xf0df bytes) 000396 1766413989.499511 trunc from 0x037aa3 to 0x013ca1 (0x23e01 bytes) 000397 1766413989.518588 MIRROR_RESYNC 6 mirrors 000398 1766413990.279660 mapread 0x0082f1 thru 0x013ca1 (0xb9b1 bytes) 000399 1766413990.311111 mapwrite 0x0182ff thru 0x01a848 (0x254a bytes) 000400 1766413990.480848 fallocate 0x01023d thru 0x017f2b (0x7cef bytes) 000401 1766413990.494812 MIRROR_RESYNC 6 mirrors 000402 1766413991.705310 read 0x001db5 thru 0x006b5a (0x4da6 bytes) 000403 1766413991.732061 MIRROR_RESYNC 6 mirrors 000404 1766413992.129960 mapwrite 0x02de92 thru 0x03c1a6 (0xe315 bytes) 000405 1766413992.225368 write 0x0097b9 thru 0x00c717 (0x2f5f bytes) 000406 1766413992.238738 mapread 0x0218ff thru 0x021c10 (0x312 bytes) 000407 1766413992.253446 mapread 0x00886b thru 0x012633 (0x9dc9 bytes) 000408 1766413992.261992 punch from 0x035263 to 0x03be95 (0x6c33 bytes) 000409 1766413992.270368 punch from 0x00fd8f to 0x012e9a (0x310c bytes) 000410 1766413992.276730 mapread 0x01fd2e thru 0x02daeb (0xddbe bytes) 000411 1766413992.278041 fallocate 0x0397e5 thru 0x03ffff (0x681b bytes) 000412 1766413992.286681 write 0x023bfc thru 0x02990a (0x5d0f bytes) 000413 1766413992.289894 mapwrite 0x015a53 thru 0x024f96 (0xf544 bytes) 000415 1766413992.312039 punch from 0x01fa70 to 0x02096f (0xf00 bytes) 000416 1766413992.318019 trunc from 0x0081a7 to 0x00f3d0 (0x722a bytes) 000417 1766413992.329747 punch from 0x0128b6 to 0x017d18 (0x5463 bytes) 000418 1766413992.341883 punch from 0x03aaa6 to 0x03c1a6 (0x1701 bytes) 000419 1766413992.349779 punch from 0x01c465 to 0x023864 (0x7400 bytes) 000420 1766413992.360913 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1a7): Mirror 3: 0x321a553d (stale) Mirror 5: 0x321a553d (stale) Mirror 6: 0x321a553d (stale) Mirror 7: 0x321a553d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1a7): Mirror 3: 0x321a553d (stale) Mirror 5: 0x321a553d (stale) Mirror 6: 0x321a553d (stale) Mirror 7: 0x321a553d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1a7: skipped 000421 1766413992.732361 write 0x02ebe7 thru 0x03b403 (0xc81d bytes) 000422 1766413992.765464 read 0x015594 thru 0x0203e7 (0xae54 bytes) 000423 1766413992.785242 write 0x01842b thru 0x01c780 (0x4356 bytes) 000424 1766413992.788447 mapread 0x00523c thru 0x008869 (0x362e bytes) 000426 1766413992.815255 read 0x00e1df thru 0x01bc8b (0xdaad bytes) 000427 1766413992.833543 trunc from 0x03c1a7 to 0x0136c3 (0x28ae3 bytes) 000428 1766413992.858015 mapread 0x010274 thru 0x0136c3 (0x3450 bytes) 000429 1766413992.859983 read 0x010131 thru 0x010138 (0x8 bytes) 000430 1766413992.861497 write 0x012e8e thru 0x01ea68 (0xbbdb bytes) 000431 1766413992.862937 fallocate 0x018f55 thru 0x025d98 (0xce44 bytes) 000432 1766413992.873687 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25d99): Mirror 3: 0xab1b9f44 (stale) Mirror 5: 0xab1b9f44 (stale) Mirror 6: 0xab1b9f44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d99): Mirror 3: 0xab1b9f44 (stale) Mirror 5: 0xab1b9f44 (stale) Mirror 6: 0xab1b9f44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d99: skipped 000434 1766413993.259525 mapread 0x001ebb thru 0x00c987 (0xaacd bytes) 000435 1766413993.285919 read 0x02593b thru 0x025d98 (0x45e bytes) 000436 1766413993.292373 fallocate 0x03b407 thru 0x03ffff (0x4bf9 bytes) 000437 1766413993.297964 punch from 0x007033 to 0x0097c5 (0x2793 bytes) 000438 1766413993.303766 fallocate 0x024b56 thru 0x033a84 (0xef2f bytes) 000439 1766413993.319724 write 0x032aaf thru 0x03fa0a (0xcf5c bytes) 000440 1766413993.338100 mapread 0x03be4c thru 0x03ffff (0x41b4 bytes) 000441 1766413993.338894 trunc from 0x03a35a to 0x03cc58 (0x28ff bytes) 000442 1766413993.354922 read 0x02e032 thru 0x03a78a (0xc759 bytes) 000443 1766413993.365798 trunc from 0x040000 to 0x01d96f (0x22690 bytes) 000444 1766413993.382497 punch from 0x00061e to 0x0039af (0x3392 bytes) 000445 1766413993.386666 trunc from 0x01d8c9 to 0x01d96f (0xa7 bytes) 000446 1766413993.393245 write 0x0073d5 thru 0x014635 (0xd261 bytes) 000447 1766413993.395541 punch from 0x00b55a to 0x0108fd (0x53a4 bytes) 000448 1766413993.409991 mapread 0x017c35 thru 0x01d96f (0x5d3b bytes) 000449 1766413993.416749 punch from 0x00e23a to 0x01cdc0 (0xeb87 bytes) 000450 1766413993.422277 write 0x01c6bb thru 0x0246d3 (0x8019 bytes) 000451 1766413993.427926 trunc from 0x0142a8 to 0x0158b8 (0x1611 bytes) 000452 1766413993.432853 punch from 0x022f9c to 0x0246d3 (0x1738 bytes) 000453 1766413993.449703 read 0x0226b1 thru 0x0246d3 (0x2023 bytes) 000454 1766413993.471455 mapread 0x010fd3 thru 0x01d9e8 (0xca16 bytes) 000455 1766413993.473740 mapwrite 0x03ff7e thru 0x03ffff (0x82 bytes) 000456 1766413993.544553 punch from 0x01a6ce to 0x01bb28 (0x145b bytes) 000457 1766413993.555873 trunc from 0x003293 to 0x007701 (0x446f bytes) 000458 1766413993.564197 write 0x008e75 thru 0x012335 (0x94c1 bytes) 000459 1766413993.565432 trunc from 0x027f12 to 0x0373b6 (0xf4a5 bytes) 000460 1766413993.579798 mapread 0x004544 thru 0x00bb42 (0x75ff bytes) 000461 1766413993.619602 trunc from 0x040000 to 0x02661e (0x199e1 bytes) 000462 1766413993.642845 trunc from 0x019f7c to 0x02661e (0xc6a3 bytes) 000464 1766413993.656303 MIRROR_RESYNC 4 mirrors 000466 1766413994.607325 mapwrite 0x02472c thru 0x029ee9 (0x57be bytes) 000467 1766413994.725981 mapwrite 0x03b854 thru 0x03ff2b (0x46d8 bytes) 000468 1766413994.796584 read 0x035af8 thru 0x038346 (0x284f bytes) 000469 1766413994.797340 mapread 0x018ef1 thru 0x01df89 (0x5099 bytes) 000470 1766413994.818777 mapwrite 0x03f739 thru 0x03ffff (0x8c7 bytes) 000471 1766413994.858917 punch from 0x0373b3 to 0x03ffff (0x8c4d bytes) 000472 1766413994.882082 mapread 0x016c6c thru 0x0241dd (0xd572 bytes) 000473 1766413994.887533 mapread 0x00cf22 thru 0x01bdfd (0xeedc bytes) 000474 1766413994.904981 mapwrite 0x01d724 thru 0x02ac39 (0xd516 bytes) 000475 1766413994.960087 mapwrite 0x0245c3 thru 0x02b394 (0x6dd2 bytes) 000476 1766413994.980591 mapread 0x03bf42 thru 0x03ffff (0x40be bytes) 000477 1766413994.990671 punch from 0x00f0b2 to 0x015903 (0x6852 bytes) 000478 1766413994.998879 mapread 0x03da1a thru 0x03ffff (0x25e6 bytes) 000479 1766413995.000370 mapread 0x024338 thru 0x031216 (0xcedf bytes) 000480 1766413995.009280 MIRROR_RESYNC 4 mirrors 000481 1766413996.176150 trunc from 0x040000 to 0x03fe39 (0x1c6 bytes) 000482 1766413996.217878 write 0x00b150 thru 0x0129cd (0x787e bytes) 000483 1766413996.233442 mapread 0x02517b thru 0x025905 (0x78b bytes) 000484 1766413996.246959 write 0x022e5a thru 0x02e398 (0xb53f bytes) 000485 1766413996.253681 trunc from 0x009097 to 0x012ee8 (0x9e52 bytes) 000486 1766413996.273658 read 0x016f78 thru 0x01bb8a (0x4c13 bytes) 000487 1766413996.288815 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe3a): Mirror 3: 0x917dda63 (stale) Mirror 5: 0x917dda63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe3a): Mirror 3: 0x917dda63 (stale) Mirror 5: 0x917dda63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe3a: skipped 000488 1766413996.775920 mapread 0x03b1e2 thru 0x03d170 (0x1f8f bytes) 000489 1766413996.834854 trunc from 0x03fe3a to 0x00ea45 (0x313f4 bytes) 000490 1766413996.880283 fallocate 0x008736 thru 0x00e0df (0x59aa bytes) 000491 1766413996.888297 mapread 0x00e2a5 thru 0x00ea45 (0x7a1 bytes) 000492 1766413996.904120 read 0x003a8b thru 0x00cdf6 (0x936c bytes) 000493 1766413996.913765 write 0x03707a thru 0x03d91e (0x68a5 bytes) 000494 1766413996.915734 trunc from 0x03d91f to 0x027783 (0x1619b bytes) 000495 1766413996.942014 mapwrite 0x00d917 thru 0x018066 (0xa750 bytes) 000496 1766413996.992369 trunc from 0x00ffcd to 0x010423 (0x457 bytes) 000497 1766413996.999894 trunc from 0x027784 to 0x006525 (0x2125e bytes) 000498 1766413997.022476 fallocate 0x009df7 thru 0x00b2f2 (0x14fc bytes) 000499 1766413997.033658 fallocate 0x02011e thru 0x02e909 (0xe7ec bytes) 000500 1766413997.039364 mapread 0x007c5b thru 0x008e05 (0x11ab bytes) 000501 1766413997.051815 read 0x00c727 thru 0x017db1 (0xb68b bytes) 000502 1766413997.065429 read 0x022691 thru 0x02d7ef (0xb15f bytes) 000503 1766413997.066156 read 0x0171f5 thru 0x01d0a6 (0x5eb2 bytes) 000504 1766413997.066839 read 0x013fee thru 0x01cf43 (0x8f56 bytes) 000507 1766413997.068318 trunc from 0x022720 to 0x0235c2 (0xea3 bytes) 000508 1766413997.073963 trunc from 0x02e90a to 0x02d645 (0x12c4 bytes) 000509 1766413997.094047 trunc from 0x015585 to 0x01de0c (0x8888 bytes) 000510 1766413997.099430 trunc from 0x02d646 to 0x01c3e3 (0x11262 bytes) 000511 1766413997.116607 punch from 0x00c43e to 0x00d7ea (0x13ad bytes) 000512 1766413997.124003 trunc from 0x01c3e4 to 0x03fd25 (0x23942 bytes) 000513 1766413997.151286 fallocate 0x033f96 thru 0x03d33d (0x93a8 bytes) 000514 1766413997.158055 trunc from 0x03fd26 to 0x02fc69 (0x100bc bytes) 000515 1766413997.176513 read 0x0176e5 thru 0x02682d (0xf149 bytes) 000516 1766413997.202641 mapwrite 0x02364e thru 0x02a766 (0x7119 bytes) 000517 1766413997.240953 punch from 0x00fafd to 0x012388 (0x288c bytes) 000518 1766413997.258627 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fc6a): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc6a): Mirror 3: 0x2cbb4812 (stale) Mirror 5: 0x2cbb4812 (stale) CRC-32 checksum value for chunk [0, 0x2fc6a): Mirror 1: 0x27975e03 Mirror 6: 0x27975e03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc6a: skipped 000519 1766413998.211580 mapread 0x0192b6 thru 0x019be4 (0x92f bytes) 000520 1766413998.245786 trunc from 0x02fc6a to 0x00d622 (0x22647 bytes) 000521 1766413998.358422 trunc from 0x009d6d to 0x00d622 (0x38b6 bytes) 000522 1766413998.368926 write 0x015324 thru 0x01e18e (0x8e6b bytes) 000523 1766413998.370665 write 0x0088ec thru 0x00e723 (0x5e38 bytes) 000524 1766413998.390456 mapwrite 0x00dde4 thru 0x012dd0 (0x4fed bytes) 000525 1766413998.452402 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e18f): Mirror 5: 0x9f180ed7 (stale) Mirror 6: 0x741ee107 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e18f): Mirror 5: 0x9f180ed7 (stale) Mirror 6: 0x741ee107 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e18f: skipped 000526 1766413998.910073 trunc from 0x00b5ff to 0x011704 (0x6106 bytes) 000528 1766413998.924264 trunc from 0x015639 to 0x01e18e (0x8b56 bytes) 000529 1766413998.937419 MIRROR_RESYNC 3 mirrors 000530 1766414000.393840 read 0x007902 thru 0x00ba5d (0x415c bytes) 000531 1766414000.408785 mapwrite 0x03a399 thru 0x03ffff (0x5c67 bytes) 000532 1766414000.533497 fallocate 0x032679 thru 0x03e9a4 (0xc32c bytes) 000533 1766414000.550544 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfbbf11d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfbbf11d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000534 1766414000.984775 mapwrite 0x0138f3 thru 0x01e918 (0xb026 bytes) 000535 1766414001.049648 mapwrite 0x02b632 thru 0x0301f7 (0x4bc6 bytes) 000536 1766414001.073511 trunc from 0x040000 to 0x000fd2 (0x3f02d bytes) 000537 1766414001.094347 trunc from 0x000fbe to 0x000fd2 (0x15 bytes) 000539 1766414001.103098 trunc from 0x000fd3 to 0x03fc21 (0x3ec4f bytes) 000540 1766414001.123124 trunc from 0x022f30 to 0x02b903 (0x89d4 bytes) 000541 1766414001.129087 fallocate 0x014b63 thru 0x01f579 (0xaa17 bytes) 000542 1766414001.134297 fallocate 0x016792 thru 0x0229e7 (0xc256 bytes) 000543 1766414001.143377 trunc from 0x03fc22 to 0x034954 (0xb2cd bytes) 000545 1766414001.170292 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000546 1766414001.323857 trunc from 0x034955 to 0x03edc5 (0xa471 bytes) 000547 1766414001.354560 read 0x00b247 thru 0x016a6b (0xb825 bytes) 000548 1766414001.369152 trunc from 0x03edc6 to 0x01964e (0x25777 bytes) 000549 1766414001.393308 trunc from 0x00a9b6 to 0x00b647 (0xc92 bytes) 000550 1766414001.400535 fallocate 0x03eb2c thru 0x03ffff (0x14d4 bytes) 000551 1766414001.405602 trunc from 0x009f80 to 0x00e360 (0x43e1 bytes) 000552 1766414001.410616 read 0x014ccf thru 0x018efb (0x422d bytes) 000553 1766414001.420292 punch from 0x00ed70 to 0x01545d (0x66ee bytes) 000554 1766414001.423978 trunc from 0x01964f to 0x00ee02 (0xa84c bytes) 000555 1766414001.439511 mapwrite 0x0325a8 thru 0x03ffff (0xda58 bytes) 000557 1766414001.527597 trunc from 0x040000 to 0x03ec09 (0x13f6 bytes) 000558 1766414001.552107 punch from 0x022df0 to 0x0324ab (0xf6bc bytes) 000559 1766414001.567657 trunc from 0x0284bd to 0x02ea31 (0x6575 bytes) 000560 1766414001.579888 punch from 0x02cbd1 to 0x0351c3 (0x85f3 bytes) 000561 1766414001.597140 write 0x03fecf thru 0x03ffff (0x131 bytes) 000563 1766414001.598591 trunc from 0x00561a to 0x009114 (0x3afb bytes) 000564 1766414001.609544 read 0x0209e5 thru 0x02887e (0x7e9a bytes) 000565 1766414001.626977 trunc from 0x040000 to 0x00674a (0x398b5 bytes) 000566 1766414001.652713 read 0x005af9 thru 0x00674a (0xc52 bytes) 000567 1766414001.653355 mapwrite 0x00d6d1 thru 0x00f4de (0x1e0e bytes) 000568 1766414001.718291 mapwrite 0x030a95 thru 0x03d783 (0xccef bytes) 000569 1766414001.779949 trunc from 0x00d4a1 to 0x00fc3a (0x279a bytes) 000570 1766414001.790664 write 0x01d2c0 thru 0x024f3f (0x7c80 bytes) 000571 1766414001.793440 mapread 0x03a580 thru 0x03d783 (0x3204 bytes) 000574 1766414001.798162 trunc from 0x035902 to 0x03a6fe (0x4dfd bytes) 000575 1766414001.810549 fallocate 0x003970 thru 0x00d334 (0x99c5 bytes) 000576 1766414001.817020 trunc from 0x03d784 to 0x03cd66 (0xa1d bytes) 000577 1766414001.846090 fallocate 0x006888 thru 0x014e5a (0xe5d3 bytes) 000578 1766414001.852572 mapwrite 0x000699 thru 0x0031ba (0x2b22 bytes) 000579 1766414001.884528 mapread 0x003070 thru 0x003a70 (0xa01 bytes) 000580 1766414001.885385 mapread 0x02372d thru 0x032cd2 (0xf5a6 bytes) 000583 1766414001.899569 MIRROR_EXTEND to 2 mirrors 000584 1766414002.628763 mapread 0x03b80c thru 0x03cd66 (0x155b bytes) 000585 1766414002.667098 read 0x009f78 thru 0x0151f6 (0xb27f bytes) 000586 1766414002.681066 punch from 0x0285c1 to 0x03548b (0xcecb bytes) 000587 1766414002.723819 punch from 0x023a8c to 0x0260a2 (0x2617 bytes) 000588 1766414002.729564 read 0x02195c thru 0x02d080 (0xb725 bytes) 000589 1766414002.739260 mapwrite 0x013553 thru 0x0194b8 (0x5f66 bytes) 000590 1766414002.756339 mapread 0x037ba2 thru 0x038c61 (0x10c0 bytes) 000592 1766414002.757587 write 0x002276 thru 0x006036 (0x3dc1 bytes) 000593 1766414002.769009 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000594 1766414002.901418 fallocate 0x0181a9 thru 0x01ff21 (0x7d79 bytes) 000595 1766414002.907342 mapwrite 0x01e3b1 thru 0x0282ff (0x9f4f bytes) 000597 1766414002.985092 MIRROR_EXTEND to 2 mirrors 000598 1766414003.725046 punch from 0x03c617 to 0x03cd66 (0x750 bytes) 000600 1766414003.772847 trunc from 0x0364cb to 0x03a996 (0x44cc bytes) 000601 1766414003.780913 punch from 0x01d6a8 to 0x02284e (0x51a7 bytes) 000602 1766414003.785378 trunc from 0x021c18 to 0x02b410 (0x97f9 bytes) 000603 1766414003.791055 read 0x027ee9 thru 0x02c7ee (0x4906 bytes) 000604 1766414003.800572 read 0x024ef4 thru 0x02bc05 (0x6d12 bytes) 000605 1766414003.814577 fallocate 0x018c0e thru 0x01a997 (0x1d8a bytes) 000606 1766414003.819318 read 0x02ce6b thru 0x0379bf (0xab55 bytes) 000608 1766414003.823700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000609 1766414004.001945 mapwrite 0x03789c thru 0x038101 (0x866 bytes) 000611 1766414004.027031 trunc from 0x024e54 to 0x026b8e (0x1d3b bytes) 000612 1766414004.032162 fallocate 0x02b9c4 thru 0x02d6ea (0x1d27 bytes) 000613 1766414004.037767 punch from 0x027bbd to 0x02e6c2 (0x6b06 bytes) 000614 1766414004.041607 trunc from 0x03cd67 to 0x0012be (0x3baa8 bytes) 000615 1766414004.068080 read 0x0002b9 thru 0x0012be (0x1006 bytes) 000616 1766414004.074662 mapwrite 0x029711 thru 0x029d2e (0x61e bytes) 000617 1766414004.110682 trunc from 0x02684b to 0x029d2e (0x34e4 bytes) 000618 1766414004.116800 mapwrite 0x006805 thru 0x00ce86 (0x6682 bytes) 000619 1766414004.138376 mapwrite 0x010789 thru 0x0192d0 (0x8b48 bytes) 000620 1766414004.176628 punch from 0x00d1ec to 0x01bc14 (0xea29 bytes) 000621 1766414004.185530 mapwrite 0x00c2f1 thru 0x010f29 (0x4c39 bytes) 000623 1766414004.243003 mapwrite 0x028083 thru 0x0357f2 (0xd770 bytes) 000624 1766414004.385810 fallocate 0x021a6f thru 0x02e849 (0xcddb bytes) 000626 1766414004.393453 read 0x0134e7 thru 0x017c3e (0x4758 bytes) 000627 1766414004.414124 fallocate 0x000cf4 thru 0x007b99 (0x6ea6 bytes) 000629 1766414004.433612 MIRROR_EXTEND to 2 mirrors 000630 1766414005.007444 read 0x017291 thru 0x01afed (0x3d5d bytes) 000631 1766414005.024036 mapread 0x0341e4 thru 0x0357f2 (0x160f bytes) 000632 1766414005.069352 trunc from 0x0357f3 to 0x02e42f (0x73c3 bytes) 000633 1766414005.159404 fallocate 0x03c432 thru 0x03ffff (0x3bce bytes) 000634 1766414005.170939 trunc from 0x0029d2 to 0x00dec4 (0xb4f3 bytes) 000635 1766414005.179117 mapwrite 0x03d1f8 thru 0x03ffff (0x2e08 bytes) 000636 1766414005.237153 fallocate 0x034a2a thru 0x03b4ca (0x6aa1 bytes) 000637 1766414005.252953 mapwrite 0x01990e thru 0x019975 (0x68 bytes) 000638 1766414005.292052 fallocate 0x03fe5d thru 0x03ffff (0x1a3 bytes) 000639 1766414005.296997 punch from 0x02573e to 0x028d8e (0x3651 bytes) 000640 1766414005.302412 mapread 0x02f519 thru 0x032e5f (0x3947 bytes) 000641 1766414005.312528 read 0x015587 thru 0x01e3e4 (0x8e5e bytes) 000642 1766414005.313144 trunc from 0x03296c to 0x03476f (0x1e04 bytes) 000643 1766414005.321441 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000644 1766414005.463787 mapwrite 0x02adca thru 0x034d1d (0x9f54 bytes) 000645 1766414005.507338 trunc from 0x01adbe to 0x026f14 (0xc157 bytes) 000646 1766414005.526198 mapread 0x0216b4 thru 0x02f0da (0xda27 bytes) 000651 1766414005.542432 punch from 0x0393c0 to 0x03ffff (0x6c40 bytes) 000652 1766414005.552836 write 0x019870 thru 0x02350c (0x9c9d bytes) 000653 1766414005.573273 mapread 0x009bc9 thru 0x00f67a (0x5ab2 bytes) 000654 1766414005.586330 punch from 0x0242e7 to 0x027fc3 (0x3cdd bytes) 000655 1766414005.596235 trunc from 0x040000 to 0x01f448 (0x20bb7 bytes) 000656 1766414005.620842 read 0x00e7ff thru 0x014293 (0x5a95 bytes) 000657 1766414005.621493 trunc from 0x003d5d to 0x011869 (0xdb0d bytes) 000658 1766414005.626446 trunc from 0x01c1eb to 0x01f448 (0x325e bytes) 000659 1766414005.646616 punch from 0x01ec0f to 0x01f448 (0x83a bytes) 000660 1766414005.653237 punch from 0x019249 to 0x01abd8 (0x1990 bytes) 000662 1766414005.658703 mapread 0x002f9f thru 0x010ce1 (0xdd43 bytes) 000664 1766414005.667833 trunc from 0x01c095 to 0x01df67 (0x1ed3 bytes) 000665 1766414005.672328 trunc from 0x01a808 to 0x01f448 (0x4c41 bytes) 000666 1766414005.679401 mapwrite 0x01c4a4 thru 0x01ee3a (0x2997 bytes) 000668 1766414005.707479 fallocate 0x0217f8 thru 0x02bca2 (0xa4ab bytes) 000669 1766414005.712074 mapwrite 0x02483d thru 0x02cdce (0x8592 bytes) 000670 1766414005.753970 mapwrite 0x0152f9 thru 0x015cfb (0xa03 bytes) 000671 1766414005.765589 fallocate 0x00b35a thru 0x011950 (0x65f7 bytes) 000672 1766414005.774883 read 0x016f10 thru 0x026af7 (0xfbe8 bytes) 000673 1766414005.775787 trunc from 0x02cdcf to 0x022586 (0xa848 bytes) 000675 1766414005.796887 mapread 0x00f4a8 thru 0x01666a (0x71c3 bytes) 000676 1766414005.809831 mapread 0x015335 thru 0x022586 (0xd252 bytes) 000677 1766414005.810750 trunc from 0x022587 to 0x037c01 (0x1567b bytes) 000678 1766414005.847941 mapread 0x007900 thru 0x012261 (0xa962 bytes) 000681 1766414005.851538 read 0x01b269 thru 0x01f026 (0x3dbe bytes) 000683 1766414005.858407 punch from 0x0200f9 to 0x020d3c (0xc44 bytes) 000684 1766414005.870709 mapread 0x01d05b thru 0x01e401 (0x13a7 bytes) 000685 1766414005.871543 trunc from 0x037c02 to 0x031b42 (0x60bf bytes) 000686 1766414005.899657 trunc from 0x031b43 to 0x0269fd (0xb145 bytes) 000687 1766414005.932216 trunc from 0x0269fe to 0x0211ac (0x5851 bytes) 000688 1766414005.953775 punch from 0x013503 to 0x0199dc (0x64da bytes) 000689 1766414005.973926 punch from 0x014a7a to 0x01d734 (0x8cbb bytes) 000690 1766414005.989520 write 0x02ba64 thru 0x032d0b (0x72a8 bytes) 000691 1766414005.993629 mapread 0x026bfa thru 0x029818 (0x2c1f bytes) 000692 1766414006.013681 mapwrite 0x0299ef thru 0x02daf7 (0x4109 bytes) 000693 1766414006.061959 trunc from 0x001d70 to 0x00b9cf (0x9c60 bytes) 000694 1766414006.070988 write 0x03c3bf thru 0x03ffff (0x3c41 bytes) 000696 1766414006.072558 write 0x007579 thru 0x007f39 (0x9c1 bytes) 000697 1766414006.088536 punch from 0x01f2b6 to 0x020ad3 (0x181e bytes) 000698 1766414006.094712 write 0x02e781 thru 0x034ec5 (0x6745 bytes) 000699 1766414006.099664 trunc from 0x01bbea to 0x023c0a (0x8021 bytes) 000700 1766414006.104598 punch from 0x037189 to 0x03ffff (0x8e77 bytes) 000701 1766414006.151588 punch from 0x03983a to 0x03cb9c (0x3363 bytes) 000703 1766414006.162703 read 0x023fca thru 0x0313a4 (0xd3db bytes) 000704 1766414006.179975 mapread 0x0287e0 thru 0x02ecbf (0x64e0 bytes) 000706 1766414006.181549 read 0x018873 thru 0x02853e (0xfccc bytes) 000707 1766414006.182100 trunc from 0x00fc5c to 0x019987 (0x9d2c bytes) 000708 1766414006.188350 trunc from 0x040000 to 0x00a937 (0x356c8 bytes) 000709 1766414006.209464 trunc from 0x00a938 to 0x01e267 (0x13930 bytes) 000710 1766414006.228969 fallocate 0x02786d thru 0x028325 (0xab9 bytes) 000711 1766414006.237492 mapread 0x0147d5 thru 0x01b64a (0x6e76 bytes) 000712 1766414006.251316 trunc from 0x01e268 to 0x0325f9 (0x14392 bytes) 000713 1766414006.271263 mapwrite 0x01c24e thru 0x01d9fa (0x17ad bytes) 000714 1766414006.295868 mapread 0x008048 thru 0x00875e (0x717 bytes) 000716 1766414006.297407 mapwrite 0x01e3fc thru 0x0238a5 (0x54aa bytes) 000717 1766414006.323901 trunc from 0x0325fa to 0x010688 (0x21f71 bytes) 000718 1766414006.340023 trunc from 0x010689 to 0x015976 (0x52ee bytes) 000719 1766414006.365903 trunc from 0x015977 to 0x027bac (0x12236 bytes) 000720 1766414006.393728 fallocate 0x01eba5 thru 0x02e531 (0xf98d bytes) 000721 1766414006.398012 read 0x0189a0 thru 0x0189c0 (0x21 bytes) 000722 1766414006.407562 trunc from 0x02e532 to 0x0233df (0xb152 bytes) 000723 1766414006.419414 trunc from 0x0106b4 to 0x016e00 (0x674d bytes) 000724 1766414006.426919 write 0x03424b thru 0x038db9 (0x4b6f bytes) 000725 1766414006.431604 mapread 0x00df30 thru 0x00e560 (0x631 bytes) 000726 1766414006.432801 fallocate 0x00fa81 thru 0x01406a (0x45ea bytes) 000727 1766414006.439876 fallocate 0x0385e0 thru 0x03ffff (0x7a20 bytes) 000728 1766414006.448551 fallocate 0x02d5e4 thru 0x02e900 (0x131d bytes) 000729 1766414006.456364 fallocate 0x0036af thru 0x00d363 (0x9cb5 bytes) 000731 1766414006.466104 trunc from 0x040000 to 0x03c665 (0x399a bytes) 000732 1766414006.486875 mapwrite 0x01dcb1 thru 0x02463b (0x698b bytes) 000733 1766414006.529771 mapwrite 0x02fba6 thru 0x03d220 (0xd67b bytes) 000735 1766414006.612837 read 0x012259 thru 0x019505 (0x72ad bytes) 000737 1766414006.638602 mapread 0x000120 thru 0x00506b (0x4f4c bytes) 000738 1766414006.642813 mapwrite 0x017833 thru 0x021330 (0x9afe bytes) 000739 1766414006.669491 trunc from 0x03d221 to 0x016072 (0x271ae bytes) 000740 1766414006.692682 fallocate 0x017afd thru 0x0251fa (0xd6fe bytes) 000742 1766414006.699678 write 0x02bbf2 thru 0x036c59 (0xb068 bytes) 000743 1766414006.701214 fallocate 0x022951 thru 0x0232a5 (0x955 bytes) 000744 1766414006.706858 trunc from 0x036c5a to 0x00a2c3 (0x2c996 bytes) 000746 1766414006.738342 write 0x00bbaa thru 0x00d881 (0x1cd8 bytes) 000747 1766414006.739376 trunc from 0x00d882 to 0x01da2e (0x101ad bytes) 000748 1766414006.759967 mapwrite 0x01ade1 thru 0x02454a (0x976a bytes) 000750 1766414006.844514 punch from 0x01add2 to 0x02454a (0x9779 bytes) 000751 1766414006.856994 fallocate 0x01d50b thru 0x020b2e (0x3624 bytes) 000752 1766414006.861928 punch from 0x0148b1 to 0x020f63 (0xc6b3 bytes) 000755 1766414006.868385 fallocate 0x02bce5 thru 0x039d84 (0xe0a0 bytes) 000756 1766414006.876094 punch from 0x0206ee to 0x0254e9 (0x4dfc bytes) 000757 1766414006.880003 punch from 0x02142d to 0x023f5c (0x2b30 bytes) 000758 1766414006.884585 mapread 0x00bdb5 thru 0x01742b (0xb677 bytes) 000759 1766414006.938513 fallocate 0x03c21f thru 0x03ffff (0x3de1 bytes) 000760 1766414006.942594 trunc from 0x039d85 to 0x001780 (0x38604 bytes) 000761 1766414006.961894 read 0x000d16 thru 0x001780 (0xa6b bytes) 000762 1766414006.962568 fallocate 0x0028a0 thru 0x00c1ca (0x992b bytes) 000763 1766414006.968570 write 0x0238bf thru 0x030592 (0xccd4 bytes) 000764 1766414006.971386 punch from 0x030059 to 0x030592 (0x53a bytes) 000765 1766414006.984043 mapwrite 0x033b6b thru 0x03e8a2 (0xad38 bytes) 000767 1766414007.021289 read 0x015003 thru 0x0182fa (0x32f8 bytes) 000768 1766414007.022941 trunc from 0x03e8a3 to 0x02a68b (0x14217 bytes) 000769 1766414007.043795 mapread 0x01b18d thru 0x021a7d (0x68f1 bytes) 000771 1766414007.045227 mapread 0x0236de thru 0x02a68b (0x6fae bytes) 000772 1766414007.045990 punch from 0x0121ef to 0x01db83 (0xb995 bytes) 000773 1766414007.051162 trunc from 0x01a5e4 to 0x028ccc (0xe6e9 bytes) 000774 1766414007.057502 mapwrite 0x0299ac thru 0x038098 (0xe6ed bytes) 000777 1766414007.106757 mapread 0x00c1a0 thru 0x017e87 (0xbce8 bytes) 000778 1766414007.108083 mapwrite 0x009038 thru 0x0142ec (0xb2b5 bytes) 000779 1766414007.122568 read 0x0213ba thru 0x021600 (0x247 bytes) 000780 1766414007.123148 mapread 0x02707f thru 0x02dbd8 (0x6b5a bytes) 000781 1766414007.123957 write 0x02c869 thru 0x034163 (0x78fb bytes) 000782 1766414007.124809 read 0x0031cc thru 0x010f83 (0xddb8 bytes) 000783 1766414007.125392 mapwrite 0x035fbe thru 0x038770 (0x27b3 bytes) 000784 1766414007.154160 trunc from 0x0287a0 to 0x02894c (0x1ad bytes) 000785 1766414007.165426 MIRROR_EXTEND to 2 mirrors 000786 1766414007.740847 MIRROR_RESYNC 2 mirrors 000787 1766414008.024681 mapwrite 0x020df4 thru 0x0244a5 (0x36b2 bytes) 000788 1766414008.155222 mapwrite 0x00a443 thru 0x015086 (0xac44 bytes) 000789 1766414008.202818 punch from 0x008383 to 0x00b186 (0x2e04 bytes) 000790 1766414008.208127 fallocate 0x0239bb thru 0x03026b (0xc8b1 bytes) 000791 1766414008.212665 fallocate 0x03bac2 thru 0x03ffff (0x453e bytes) 000792 1766414008.224692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000793 1766414008.431912 read 0x022eec thru 0x030792 (0xd8a7 bytes) 000794 1766414008.461558 fallocate 0x033162 thru 0x03c276 (0x9115 bytes) 000795 1766414008.467246 trunc from 0x009d0c to 0x00c269 (0x255e bytes) 000796 1766414008.481677 trunc from 0x03c277 to 0x0088d4 (0x339a2 bytes) 000797 1766414008.543451 mapwrite 0x0367b0 thru 0x03ffff (0x9850 bytes) 000798 1766414008.615874 MIRROR_EXTEND to 2 mirrors 000800 1766414009.128853 trunc from 0x040000 to 0x030b2e (0xf4d1 bytes) 000803 1766414009.205517 trunc from 0x01b026 to 0x0247ff (0x97da bytes) 000804 1766414009.210803 read 0x01dfde thru 0x0228e4 (0x4907 bytes) 000805 1766414009.219320 write 0x00eb44 thru 0x01d35a (0xe817 bytes) 000806 1766414009.229389 fallocate 0x00ad19 thru 0x00d01d (0x2305 bytes) 000807 1766414009.234650 write 0x01fdd0 thru 0x02d028 (0xd259 bytes) 000808 1766414009.242819 mapwrite 0x032dea thru 0x034d65 (0x1f7c bytes) 000809 1766414009.303734 mapwrite 0x02ecf0 thru 0x030d06 (0x2017 bytes) 000810 1766414009.317818 mapwrite 0x01c3b3 thru 0x025897 (0x94e5 bytes) 000811 1766414009.336112 mapread 0x02c312 thru 0x034d65 (0x8a54 bytes) 000812 1766414009.336979 fallocate 0x036e80 thru 0x037499 (0x61a bytes) 000813 1766414009.345881 read 0x037246 thru 0x037499 (0x254 bytes) 000815 1766414009.353004 trunc from 0x00f9f5 to 0x01006e (0x67a bytes) 000817 1766414009.358843 trunc from 0x02814e to 0x0292aa (0x115d bytes) 000818 1766414009.364352 read 0x026c1f thru 0x033552 (0xc934 bytes) 000819 1766414009.374733 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3749a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3749a): Mirror 2: 0xc2ea0eb (stale) CRC-32 checksum value for chunk [0, 0x3749a): Mirror 1: 0x8084a204 Mirror 3: 0x8084a204 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3749a: skipped 000820 1766414010.048902 mapwrite 0x02a429 thru 0x034446 (0xa01e bytes) 000821 1766414010.171121 fallocate 0x018a44 thru 0x018aa9 (0x66 bytes) 000822 1766414010.178880 mapread 0x006ff8 thru 0x00ed03 (0x7d0c bytes) 000823 1766414010.190011 write 0x012bdc thru 0x017596 (0x49bb bytes) 000824 1766414010.210545 read 0x010fbc thru 0x01b481 (0xa4c6 bytes) 000825 1766414010.218096 punch from 0x024006 to 0x02f72e (0xb729 bytes) 000826 1766414010.224643 punch from 0x009686 to 0x01887b (0xf1f6 bytes) 000828 1766414010.256972 mapread 0x002bcd thru 0x002e5c (0x290 bytes) 000829 1766414010.258776 write 0x03d832 thru 0x03ffff (0x27ce bytes) 000830 1766414010.260071 write 0x03591b thru 0x03911c (0x3802 bytes) 000831 1766414010.270246 trunc from 0x040000 to 0x025b38 (0x1a4c7 bytes) 000832 1766414010.335304 fallocate 0x033d5d thru 0x03b194 (0x7438 bytes) 000833 1766414010.342335 fallocate 0x03792b thru 0x03ffff (0x86d5 bytes) 000834 1766414010.348413 mapwrite 0x03e1b1 thru 0x03f45c (0x12ac bytes) 000835 1766414010.408227 punch from 0x01c203 to 0x0283ff (0xc1fd bytes) 000836 1766414010.414992 fallocate 0x0283ef thru 0x02f17c (0x6d8e bytes) 000837 1766414010.422900 punch from 0x01ce54 to 0x025712 (0x88bf bytes) 000839 1766414010.436763 fallocate 0x0135d6 thru 0x020999 (0xd3c4 bytes) 000840 1766414010.442993 read 0x00ca86 thru 0x01bb5e (0xf0d9 bytes) 000841 1766414010.457044 read 0x029e0d thru 0x02a962 (0xb56 bytes) 000842 1766414010.457733 read 0x0032c0 thru 0x00fde8 (0xcb29 bytes) 000843 1766414010.458302 mapwrite 0x02c8cd thru 0x03823a (0xb96e bytes) 000844 1766414010.483070 write 0x01a52b thru 0x029dd4 (0xf8aa bytes) 000845 1766414010.485229 fallocate 0x00cecc thru 0x018132 (0xb267 bytes) 000846 1766414010.492708 write 0x011ce3 thru 0x01dd69 (0xc087 bytes) 000847 1766414010.495020 mapwrite 0x021735 thru 0x02fa7c (0xe348 bytes) 000848 1766414010.537950 MIRROR_RESYNC 3 mirrors 000850 1766414011.615031 mapwrite 0x0371d0 thru 0x03c187 (0x4fb8 bytes) 000851 1766414011.764790 mapread 0x000cd7 thru 0x001d98 (0x10c2 bytes) 000852 1766414011.785004 mapread 0x02b9af thru 0x0376f0 (0xbd42 bytes) 000853 1766414011.809121 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f45d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f45d): Mirror 2: 0x5b79f013 (stale) Mirror 3: 0x5b79f013 (stale) CRC-32 checksum value for chunk [0, 0x3f45d): Mirror 1: 0x382db765 Mirror 4: 0x382db765 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f45d: skipped 000854 1766414012.841852 fallocate 0x0353b5 thru 0x035d0f (0x95b bytes) 000855 1766414012.890456 trunc from 0x03f45d to 0x01fc09 (0x1f853 bytes) 000856 1766414012.929469 mapread 0x013520 thru 0x01fc09 (0xc6ea bytes) 000857 1766414012.966951 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fc0a): Mirror 2: 0xb6c26b0c (stale) Mirror 4: 0xb6c26b0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc0a): Mirror 2: 0xb6c26b0c (stale) Mirror 4: 0xb6c26b0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc0a: skipped 000858 1766414013.329603 fallocate 0x031e83 thru 0x03c6d2 (0xa850 bytes) 000859 1766414013.336950 read 0x00de3b thru 0x019a6b (0xbc31 bytes) 000860 1766414013.347400 trunc from 0x03c6d3 to 0x038430 (0x42a2 bytes) 000861 1766414013.377039 mapwrite 0x02defe thru 0x038f93 (0xb096 bytes) 000862 1766414013.480696 write 0x026c9c thru 0x035894 (0xebf9 bytes) 000863 1766414013.505657 mapwrite 0x011f8d thru 0x01f6cb (0xd73f bytes) 000864 1766414013.547398 mapread 0x02c62f thru 0x03726f (0xac41 bytes) 000865 1766414013.548380 trunc from 0x005aa1 to 0x0100aa (0xa60a bytes) 000866 1766414013.555225 trunc from 0x038f94 to 0x002ffa (0x35f99 bytes) 000867 1766414013.584015 fallocate 0x0372fa thru 0x03ffff (0x8d06 bytes) 000868 1766414013.591077 mapwrite 0x0007f9 thru 0x009639 (0x8e41 bytes) 000869 1766414013.650845 mapwrite 0x022212 thru 0x02440b (0x21fa bytes) 000870 1766414013.714041 write 0x02d94a thru 0x039209 (0xb8c0 bytes) 000871 1766414013.721386 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3920a): Mirror 2: 0xddc45c94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3920a): Mirror 2: 0xddc45c94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3920a: skipped 000872 1766414014.071512 mapwrite 0x02ad3d thru 0x0396ad (0xe971 bytes) 000873 1766414014.140509 trunc from 0x0165d2 to 0x01e794 (0x81c3 bytes) 000874 1766414014.147103 trunc from 0x02cefe to 0x037c4a (0xad4d bytes) 000875 1766414014.152252 write 0x0133f1 thru 0x02139d (0xdfad bytes) 000876 1766414014.206976 punch from 0x030435 to 0x034660 (0x422c bytes) 000877 1766414014.215495 write 0x02e455 thru 0x03785a (0x9406 bytes) 000878 1766414014.260770 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000879 1766414014.446491 punch from 0x00a9f2 to 0x01339f (0x89ae bytes) 000880 1766414014.455974 trunc from 0x008788 to 0x017f44 (0xf7bd bytes) 000881 1766414014.462054 trunc from 0x02d1e4 to 0x035d19 (0x8b36 bytes) 000882 1766414014.466812 mapwrite 0x00437f thru 0x011150 (0xcdd2 bytes) 000883 1766414014.536863 read 0x01b730 thru 0x01c399 (0xc6a bytes) 000884 1766414014.555138 trunc from 0x0396ae to 0x02d305 (0xc3a8 bytes) 000885 1766414014.583691 fallocate 0x029bf9 thru 0x02e96f (0x4d77 bytes) 000887 1766414014.589556 fallocate 0x018937 thru 0x01d5ca (0x4c94 bytes) 000888 1766414014.600859 MIRROR_EXTEND to 2 mirrors 000889 1766414015.404919 trunc from 0x02d306 to 0x01b97d (0x11988 bytes) 000890 1766414015.447305 punch from 0x00ec91 to 0x01b97d (0xcced bytes) 000891 1766414015.451579 write 0x0359d6 thru 0x03ffff (0xa62a bytes) 000892 1766414015.453741 mapwrite 0x03471b thru 0x03a678 (0x5f5e bytes) 000893 1766414015.472942 punch from 0x039941 to 0x03c31e (0x29de bytes) 000894 1766414015.481104 punch from 0x02959f to 0x029e56 (0x8b8 bytes) 000895 1766414015.485982 write 0x0378a2 thru 0x03ffff (0x875e bytes) 000896 1766414015.494136 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdcf0787f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b647b39 Mirror 3: 0x8b647b39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000897 1766414016.343255 trunc from 0x03eb37 to 0x03ffff (0x14c9 bytes) 000898 1766414016.411501 punch from 0x019a4a to 0x02104a (0x7601 bytes) 000899 1766414016.420322 mapwrite 0x000906 thru 0x005a84 (0x517f bytes) 000900 1766414016.463347 mapwrite 0x017821 thru 0x026004 (0xe7e4 bytes) 000901 1766414016.501344 MIRROR_RESYNC 3 mirrors 000902 1766414017.420487 trunc from 0x040000 to 0x004665 (0x3b99a bytes) 000903 1766414017.454936 mapwrite 0x015f6e thru 0x01f045 (0x90d8 bytes) 000904 1766414017.577598 MIRROR_RESYNC 3 mirrors 000905 1766414018.149703 punch from 0x014485 to 0x01f045 (0xabc1 bytes) 000906 1766414018.187515 read 0x00da4f thru 0x0110ee (0x36a0 bytes) 000907 1766414018.200777 trunc from 0x01f046 to 0x00ece5 (0x10360 bytes) 000909 1766414018.269044 mapread 0x008527 thru 0x00ece5 (0x67bf bytes) 000910 1766414018.295763 read 0x0097a8 thru 0x00ece5 (0x553e bytes) 000911 1766414018.296364 fallocate 0x020b4a thru 0x028fff (0x84b6 bytes) 000913 1766414018.303961 mapread 0x003d4c thru 0x00cf95 (0x924a bytes) 000915 1766414018.305767 punch from 0x00daad to 0x00db6e (0xc2 bytes) 000916 1766414018.313731 mapwrite 0x003e14 thru 0x011bc8 (0xddb5 bytes) 000917 1766414018.412346 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11bc9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bc9): Mirror 2: 0xd4912236 (stale) Mirror 3: 0xd4912236 (stale) CRC-32 checksum value for chunk [0, 0x11bc9): Mirror 1: 0x5d474012 Mirror 4: 0x5d474012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bc9: skipped 000918 1766414019.426665 read 0x00540a thru 0x010f73 (0xbb6a bytes) 000919 1766414019.445700 write 0x00057c thru 0x00d1a8 (0xcc2d bytes) 000920 1766414019.523825 mapread 0x00c01e thru 0x00ecec (0x2ccf bytes) 000921 1766414019.531047 mapwrite 0x025e7c thru 0x027cde (0x1e63 bytes) 000922 1766414019.666667 fallocate 0x0004d8 thru 0x00b2c6 (0xadef bytes) 000923 1766414019.678633 read 0x01f9ca thru 0x027cde (0x8315 bytes) 000924 1766414019.679316 mapread 0x021a40 thru 0x027cde (0x629f bytes) 000926 1766414019.680531 trunc from 0x027cdf to 0x03bf04 (0x14226 bytes) 000927 1766414019.718037 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf05): Mirror 2: 0xec82771a (stale) Mirror 3: 0xec82771a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf05): Mirror 2: 0xec82771a (stale) Mirror 3: 0xec82771a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf05: skipped 000928 1766414020.078556 mapwrite 0x031915 thru 0x03e778 (0xce64 bytes) 000929 1766414020.165603 mapwrite 0x038b02 thru 0x03ffff (0x74fe bytes) 000930 1766414020.221566 trunc from 0x02af24 to 0x039f7b (0xf058 bytes) 000931 1766414020.227241 mapwrite 0x028820 thru 0x02d1f4 (0x49d5 bytes) 000932 1766414020.251805 read 0x03ad3d thru 0x03ddeb (0x30af bytes) 000933 1766414020.252540 fallocate 0x03fd34 thru 0x03ffff (0x2cc bytes) 000934 1766414020.257481 mapread 0x02cf40 thru 0x038512 (0xb5d3 bytes) 000935 1766414020.273943 write 0x015efc thru 0x01b799 (0x589e bytes) 000937 1766414020.301210 read 0x0326e9 thru 0x03be28 (0x9740 bytes) 000938 1766414020.302940 trunc from 0x008129 to 0x00e0f9 (0x5fd1 bytes) 000939 1766414020.309876 trunc from 0x00e928 to 0x01091e (0x1ff7 bytes) 000941 1766414020.319592 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf81b4826 (stale) Mirror 3: 0xf81b4826 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfbbb69c Mirror 4: 0xcfbbb69c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000942 1766414021.533142 mapread 0x02fd51 thru 0x034375 (0x4625 bytes) 000943 1766414021.568612 trunc from 0x040000 to 0x01aeef (0x25110 bytes) 000944 1766414021.664132 trunc from 0x01aef0 to 0x0100d5 (0xae1a bytes) 000945 1766414021.699982 write 0x00669b thru 0x00d794 (0x70fa bytes) 000946 1766414021.720330 punch from 0x0032d9 to 0x00e69f (0xb3c7 bytes) 000947 1766414021.748693 punch from 0x00ab3d to 0x00ba39 (0xefd bytes) 000948 1766414021.758748 mapwrite 0x038f37 thru 0x03ae56 (0x1f20 bytes) 000949 1766414021.807097 trunc from 0x03ae57 to 0x0025d1 (0x38885 bytes) 000951 1766414021.838615 mapread 0x00219f thru 0x0025d1 (0x433 bytes) 000952 1766414021.841582 mapwrite 0x001f64 thru 0x00321b (0x12b8 bytes) 000953 1766414021.923073 read 0x002f50 thru 0x00321b (0x2cc bytes) 000954 1766414021.923700 mapwrite 0x006867 thru 0x012c4f (0xc3e9 bytes) 000956 1766414021.995800 write 0x013709 thru 0x01580f (0x2107 bytes) 000957 1766414021.997643 write 0x002231 thru 0x00660a (0x43da bytes) 000959 1766414021.999706 trunc from 0x015810 to 0x03465b (0x1ee4c bytes) 000960 1766414022.026998 trunc from 0x03465c to 0x00dd2a (0x26931 bytes) 000961 1766414022.047568 trunc from 0x00dd2b to 0x03845d (0x2a733 bytes) 000962 1766414022.069037 punch from 0x007b9d to 0x00da2b (0x5e8f bytes) 000963 1766414022.075947 read 0x016136 thru 0x01d50c (0x73d7 bytes) 000964 1766414022.087107 trunc from 0x01b1bd to 0x0240c9 (0x8f0d bytes) 000965 1766414022.102921 mapread 0x00db20 thru 0x019d70 (0xc251 bytes) 000966 1766414022.103839 write 0x029799 thru 0x02d08a (0x38f2 bytes) 000967 1766414022.115818 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3845e): Mirror 2: 0x42708988 (stale) Mirror 4: 0x4b93b2d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3845e): Mirror 2: 0x42708988 (stale) Mirror 4: 0x4b93b2d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3845e: skipped 000968 1766414022.491747 trunc from 0x03845e to 0x011315 (0x27148 bytes) 000969 1766414022.518160 fallocate 0x018c05 thru 0x01ab7e (0x1f7a bytes) 000970 1766414022.525341 punch from 0x00d153 to 0x010e6f (0x3d1d bytes) 000971 1766414022.529343 trunc from 0x00e732 to 0x011315 (0x2be4 bytes) 000972 1766414022.533712 fallocate 0x02241e thru 0x02f7f6 (0xd3d9 bytes) 000973 1766414022.539509 write 0x02e6ec thru 0x03dda2 (0xf6b7 bytes) 000974 1766414022.547142 punch from 0x00f614 to 0x01e7fd (0xf1ea bytes) 000975 1766414022.558728 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dda3): Mirror 2: 0x600af5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dda3): Mirror 2: 0x600af5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dda3: skipped 000976 1766414022.934592 fallocate 0x004fe5 thru 0x00d391 (0x83ad bytes) 000977 1766414022.941371 punch from 0x03bb1c to 0x03dda2 (0x2287 bytes) 000978 1766414022.949100 fallocate 0x02614f thru 0x026e6b (0xd1d bytes) 000979 1766414022.953520 fallocate 0x009ec9 thru 0x00ed42 (0x4e7a bytes) 000980 1766414022.958558 punch from 0x01af7c to 0x0239ec (0x8a71 bytes) 000981 1766414022.963725 mapread 0x03843e thru 0x03cabf (0x4682 bytes) 000982 1766414022.980814 mapwrite 0x00d20f thru 0x019c0e (0xca00 bytes) 000983 1766414023.031478 trunc from 0x03dda3 to 0x00e3d5 (0x2f9cd bytes) 000984 1766414023.050123 mapread 0x00bc14 thru 0x00e3d5 (0x27c2 bytes) 000985 1766414023.051246 trunc from 0x00e3d6 to 0x01fe5d (0x11a88 bytes) 000986 1766414023.062309 mapread 0x00298d thru 0x00f4de (0xcb52 bytes) 000987 1766414023.073584 write 0x01c7e5 thru 0x02a254 (0xda70 bytes) 000988 1766414023.095492 trunc from 0x02a255 to 0x02a1f7 (0x5d bytes) 000989 1766414023.122988 punch from 0x00ff08 to 0x016011 (0x610a bytes) 000990 1766414023.129730 mapwrite 0x02bf21 thru 0x03a897 (0xe977 bytes) 000991 1766414023.212973 trunc from 0x0188b2 to 0x026248 (0xd997 bytes) 000992 1766414023.220459 read 0x00c3e5 thru 0x01281f (0x643b bytes) 000993 1766414023.226051 trunc from 0x01760e to 0x01c40c (0x4dff bytes) 000994 1766414023.230279 punch from 0x01a723 to 0x01b5d9 (0xeb7 bytes) 000995 1766414023.233728 fallocate 0x026737 thru 0x028290 (0x1b5a bytes) 000996 1766414023.238494 mapwrite 0x036944 thru 0x03c6d6 (0x5d93 bytes) 000997 1766414023.273327 read 0x009d95 thru 0x00c2c9 (0x2535 bytes) 000998 1766414023.273864 write 0x00af4c thru 0x01968b (0xe740 bytes) 000999 1766414023.284473 mapwrite 0x01e0d4 thru 0x0225d2 (0x44ff bytes) 001000 1766414023.329154 fallocate 0x03c38a thru 0x03ffff (0x3c76 bytes) All operations completed A-OK! PASS 70a (59s) == sanity-flr test 71: check mirror extend parent fid ==== 09:33:47 (1766414027) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000405:0x1ca:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 09:33:56 (1766414036) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248561 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730042 s, 14.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:34:03 (1766414043) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2989984 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2207360 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7127712 .. Extending file size to 3479968 .. Extending file size to 7629344 .. Extending file size to 7766176 .. Extending file size to 8169184 .. Extending file size to 8190080 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8773120 .. Extending file size to 8450496 .. Extending file size to 9047264 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8890432 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9169376 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9235584 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59673 59675 59677 59678 59680 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:35:21 (1766414121) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2052000 .. Extending file size to 7717696 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4169376 .. Extending file size to 6542048 .. Extending file size to 7683168 .. Extending file size to 8177760 .. Extending file size to 8388608 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9027968 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8794016 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9088704 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9105248 .. Waiting 61794 61795 61798 61799 61801 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:36:37 (1766414197) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2332 1268972 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18760 3588260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24632 3582388 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43392 7170648 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2332 1268972 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18760 3588260 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24632 3582388 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43392 7170648 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0969357 s, 21.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176761 s, 11.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb9b7a75 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:36:53 (1766414213) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0656717 s, 31.9 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 09:36:58 (1766414218) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x6b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbd:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:37:04 (1766414224) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0768153 s, 27.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:37:11 (1766414231) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0826651 s, 25.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:37:18 (1766414238) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162499 s, 12.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x232:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x232:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x232:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x232:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x233:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:37:49 (1766414269) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782884 s, 13.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x233:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x235:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x234:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x233:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x235:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x234:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x234:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:38:36 (1766414316) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0831037 s, 25.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:39:21 (1766414361) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg250-server: Pool lustre.test_204f created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143634 s, 14.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg250-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:40:33 (1766414433) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:40:40 (1766414440) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:40:47 (1766414447) oleg250-server: Pool lustre.test_206 created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg250-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:41:13 (1766414473) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078013 s, 13.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317103 s, 33.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8fe570b7 Mirror 3: 0x8fe570b7 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:41:23 (1766414483) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:41:26 (1766414486) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:41:29 (1766414489) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0723254 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2310000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2310000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2310000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2520000 Read this (4 bytes): BBBB 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:42:03 (1766414523) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0195036 s, 420 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:42:11 (1766414531) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622681 s, 16.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8c lmm_fid: [0x200000405:0x8c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8c lmm_fid: [0x200000405:0x8c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611276 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 09:42:14.000000000 -0500 Modify: 2025-12-22 09:42:14.000000000 -0500 Change: 2025-12-22 09:42:15.000000000 -0500 Birth: 2025-12-22 09:42:14.000000000 -0500 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:42:22 (1766414542) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306666 s, 34.2 MB/s oleg250-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg250-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg250-server: O/280000401/d14/ce: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:42:46 (1766414566) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.181227 s, 226 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0452166 s, 90.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 22 09:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x91:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 22 09:42 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:42:53 (1766414573) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:42:59 (1766414579) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1590 sec =========== 09:43:07 (1766414587) === sanity-flr: start cleanup 09:43:08 (1766414588) === === sanity-flr: finish cleanup 09:43:15 (1766414595) ===