-----============= acceptance-small: sanity-flr ============----- Fri Feb 27 01:58:33 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg323-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:58:52 (1772175532) === oleg323-client.virtnet: executing check_config_client /mnt/lustre oleg323-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg323-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c41852dc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c41852dc000.idle_timeout=debug disable quota as required oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:59:29 (1772175569) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8217] == sanity-flr test 0a: lfs mirror create with -N option == 01:59:31 (1772175571) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:59:45 (1772175585) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:59:49 (1772175589) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:59:52 (1772175592) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:00:15 (1772175615) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:00:19 (1772175619) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:00:22 (1772175622) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c41852dc000.stats=clear osc.lustre-OST0001-osc-ffff8c41852dc000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:00:32 (1772175632) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:00:41 (1772175641) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:00:49 (1772175649) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:00:51 (1772175651) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:00:59 (1772175659) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:01:06 (1772175666) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:01:13 (1772175673) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:01:20 (1772175680) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:01:29 (1772175689) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:01:35 (1772175695) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.287641 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121649 s, 8.6 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:01:42 (1772175702) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.18109 s, 29.0 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:01:49 (1772175709) Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:02:55 (1772175775) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0857228 s, 24.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:03:29 (1772175809) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (64s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:04:33 (1772175873) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.35368 s, 22.3 MB/s -rw-r--r-- 1 root root 52428800 Feb 27 02:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.879s user 0m0.000s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.921s user 0m0.000s sys 0m0.189s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.132s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c41852dc000.ost_server_uuid in FULL state after 0 sec PASS 33b (70s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:05:44 (1772175944) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:05:47 (1772175947) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:05:50 (1772175950) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 02:05:52 (1772175952) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 02:05:59 (1772175959) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.568087 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:06:09 (1772175969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755361 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255964 s, 41.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:06:14 (1772175974) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:06:20 (1772175980) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121258 s, 844 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00185568 s, 552 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122145 s, 838 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147516 s, 694 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 02:06:28 (1772175988) create test files with size 15754488 .. 9a87e23d289595df3afd6ff7c621ae05 - 635a48644649033852ee5de01b87308f - 9feda17bb9ccb2de9ced16d96f58ec1e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 02:06:47 (1772176007) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:06:58 (1772176018) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:07:03 (1772176023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680281 s, 15.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234927 s, 44.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 02:07:10 (1772176030) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122277 s, 17.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.298242 s, 14.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128133 s, 16.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 02:07:22 (1772176042) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:07:25 (1772176045) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:07:27 (1772176047) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043906 s, 23.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 02:07:33 (1772176053) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.167597 s, 18.8 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:07:40 (1772176060) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x43:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:07:47 (1772176067) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.279923 s, 37.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:07:56 (1772176076) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c4190658000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.282999 s, 37.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c41939f6000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:08:08 (1772176088) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:08:32 (1772176112) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (24s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:08:56 (1772176136) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 02:08:58 (1772176138) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.156395 s, 20.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.103439 s, 30.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xeaeca7b1 (stale) Mirror 2: 0xeaeca7b1 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 92ff2530c97a1bbae1f74f9e8e9400b1 - mirror 1 checksum: b2d44e7cd2989ebe7c5ff38e2b9da388 - mirror 2 checksum: 92ff2530c97a1bbae1f74f9e8e9400b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 27 02:09:01 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 27 02:09:01 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 92ff2530c97a1bbae1f74f9e8e9400b1 - mirror 1 checksum: b2d44e7cd2989ebe7c5ff38e2b9da388 - mirror 2 checksum: b2d44e7cd2989ebe7c5ff38e2b9da388 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 27 02:09:01 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 27 02:09:01 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (7s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:09:05 (1772176145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288053 s, 36.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:09:10 (1772176150) Failing mds1 on oleg323-server Stopping /mnt/lustre-mds1 (opts:) on oleg323-server 02:09:19 (1772176159) shut down facet: mds1 facet_host: oleg323-server facet_failover_host: oleg323-server Failover mds1 to oleg323-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 02:09:39 (1772176179) targets are mounted 02:09:39 (1772176179) facet_failover done oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013443 s, 78.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (39s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:09:49 (1772176189) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0378344 s, 541 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106515 s, 1.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:09:57 (1772176197) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0510544 s, 401 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0103276 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:10:03 (1772176203) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00567641 s, 2.9 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:10:09 (1772176209) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0304677 s, 672 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:10:15 (1772176215) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108957 s, 376 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0288449 s, 142 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:10:20 (1772176220) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (16s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:10:36 (1772176236) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:11:00 (1772176260) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:11:20 (1772176280) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:11:41 (1772176301) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 02:11:46 (1772176306) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55922 mirror split pid 55923 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55922 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55923 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 02:12:52 (1772176372) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6373 skipping zero length punch hole skipping zero size read fallocating to largest ever: 0x3e1ba 000003 1772176373.745629 fallocate 0x037c8f thru 0x03e1b9 (0x652b bytes) 000004 1772176373.752033 fallocate 0x01295e thru 0x01d15e (0xa801 bytes) skipping zero length zero range skipping zero size read 000007 1772176373.760654 trunc from 00000000 to 0x022c9f (0x22ca0 bytes) 000008 1772176373.778171 read 0x00fce2 thru 0x01982e (0x9b4d bytes) 000009 1772176373.792261 trunc from 0x022ca0 to 0x032d93 (0x100f4 bytes) 000010 1772176373.813374 trunc from 0x00917b to 0x00e5ad (0x5433 bytes) 000011 1772176373.818433 mapread 0x02d9d4 thru 0x02f04f (0x167c bytes) 000012 1772176373.847835 read 0x01e0c0 thru 0x02c235 (0xe176 bytes) 000013 1772176373.867653 mapwrite 0x03ae02 thru 0x03ffff (0x51fe bytes) 000014 1772176373.914549 mapread 0x03346a thru 0x03e132 (0xacc9 bytes) 000015 1772176373.921154 trunc from 0x02771d to 0x03236d (0xac51 bytes) 000016 1772176373.926019 trunc from 0x015a37 to 0x019005 (0x35cf bytes) 000018 1772176373.934848 fallocate 0x019e46 thru 0x01e75d (0x4918 bytes) 000020 1772176373.950923 mapread 0x035b77 thru 0x03ffff (0xa489 bytes) 000021 1772176373.954828 trunc from 0x032125 to 0x038162 (0x603e bytes) 000022 1772176373.963326 trunc from 0x02cd00 to 0x03b35f (0xe660 bytes) 000025 1772176373.991911 trunc from 0x0295d7 to 0x034fa5 (0xb9cf bytes) 000026 1772176374.002500 mapread 0x025f6c thru 0x0268da (0x96f bytes) fallocating to largest ever: 0x40000 000027 1772176374.005025 fallocate 0x03e2c1 thru 0x03ffff (0x1d3f bytes) 000029 1772176374.013960 trunc from 0x040000 to 0x017a6b (0x28594 bytes) 000030 1772176374.039265 write 0x001247 thru 0x00f06a (0xde24 bytes) 000033 1772176374.042304 mapread 0x00a20b thru 0x017a6b (0xd861 bytes) 000034 1772176374.061428 mapwrite 0x0150ca thru 0x01d7b5 (0x86ec bytes) 000036 1772176374.144977 MIRROR_EXTEND to 2 mirrors 000037 1772176374.722118 trunc from 0x01baa0 to 0x01d7b5 (0x1d16 bytes) 000038 1772176374.751216 mapread 0x011ef3 thru 0x01681b (0x4929 bytes) 000039 1772176374.780423 mapread 0x00570d thru 0x00c6d7 (0x6fcb bytes) 000040 1772176374.794980 trunc from 0x0110ba to 0x01d7b5 (0xc6fc bytes) 000041 1772176374.798943 trunc from 0x01d7b6 to 0x003820 (0x19f95 bytes) 000042 1772176374.850732 write 0x039808 thru 0x03ffff (0x67f8 bytes) 000043 1772176374.853867 punch from 0x009ac8 to 0x010308 (0x6841 bytes) 000044 1772176374.858284 punch from 0x00a685 to 0x018c2c (0xe5a8 bytes) 000045 1772176374.862682 fallocate 0x001945 thru 0x00c260 (0xa91c bytes) 000046 1772176374.868646 fallocate 0x000b2c thru 0x008e4d (0x8322 bytes) 000047 1772176374.876662 punch from 0x032cd4 to 0x03bbf6 (0x8f23 bytes) 000048 1772176374.893413 trunc from 0x033276 to 0x035ebb (0x2c46 bytes) 000049 1772176374.899147 write 0x037f2c thru 0x03ffff (0x80d4 bytes) 000051 1772176374.913364 mapread 0x038fa8 thru 0x039e66 (0xebf bytes) 000052 1772176374.915301 mapwrite 0x02424e thru 0x0258b6 (0x1669 bytes) 000053 1772176374.942237 write 0x01b4a4 thru 0x01d6cb (0x2228 bytes) 000054 1772176374.962467 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe88967e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19969cf9 Mirror 3: 0x19969cf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000055 1772176375.904356 mapwrite 0x02555c thru 0x025f7d (0xa22 bytes) 000056 1772176375.989997 read 0x03fd21 thru 0x03ffff (0x2df bytes) 000057 1772176375.995748 fallocate 0x004ec9 thru 0x013b1a (0xec52 bytes) 000058 1772176376.011688 MIRROR_RESYNC 3 mirrors 000059 1772176376.899355 write 0x03c709 thru 0x03ffff (0x38f7 bytes) 000061 1772176376.949354 read 0x015bf4 thru 0x01d58e (0x799b bytes) 000062 1772176376.961130 mapwrite 0x02e669 thru 0x0304b3 (0x1e4b bytes) 000063 1772176376.992978 read 0x037375 thru 0x03c1b5 (0x4e41 bytes) 000065 1772176377.012898 MIRROR_RESYNC 3 mirrors 000066 1772176377.804697 write 0x00edcf thru 0x01b926 (0xcb58 bytes) 000067 1772176377.848941 trunc from 0x040000 to 0x01f2ed (0x20d12 bytes) 000068 1772176377.864160 fallocate 0x03e2d3 thru 0x03ffff (0x1d2d bytes) 000069 1772176377.879214 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f2ee): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2ee): Mirror 2: 0x5960de75 (stale) Mirror 3: 0x5960de75 (stale) CRC-32 checksum value for chunk [0, 0x1f2ee): Mirror 1: 0xa9feaed4 Mirror 4: 0xa9feaed4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2ee: skipped 000070 1772176378.685850 trunc from 0x01f2ee to 0x02af3b (0xbc4e bytes) 000071 1772176378.735184 write 0x018eec thru 0x01cbad (0x3cc2 bytes) 000072 1772176378.744309 mapread 0x000d4d thru 0x0100a7 (0xf35b bytes) 000073 1772176378.765106 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2af3c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af3c): Mirror 2: 0xe653c5f4 (stale) Mirror 3: 0xe653c5f4 (stale) Mirror 4: 0x59c1640a (stale) CRC-32 checksum value for chunk [0, 0x2af3c): Mirror 1: 0xa5075be3 Mirror 5: 0xa5075be3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af3c: skipped 000074 1772176379.468842 trunc from 0x02af3c to 0x033e45 (0x8f0a bytes) 000075 1772176379.497898 mapwrite 0x034a4c thru 0x0380a6 (0x365b bytes) 000076 1772176379.533445 mapread 0x01dd3b thru 0x02a224 (0xc4ea bytes) 000077 1772176379.548572 fallocate 0x004f63 thru 0x00a465 (0x5503 bytes) 000078 1772176379.555962 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380a7): Mirror 3: 0x3660f1fd (stale) Mirror 4: 0x903ac0c5 (stale) Mirror 5: 0x3155144c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380a7): Mirror 3: 0x3660f1fd (stale) Mirror 4: 0x903ac0c5 (stale) Mirror 5: 0x3155144c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380a7: skipped 000079 1772176379.887772 punch from 0x01fdc4 to 0x025186 (0x53c3 bytes) 000080 1772176379.893147 punch from 0x020fda to 0x024453 (0x347a bytes) 000081 1772176379.897994 read 0x0353e4 thru 0x0380a6 (0x2cc3 bytes) 000082 1772176379.915893 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x380a7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380a7): Mirror 3: 0x3660f1fd (stale) Mirror 4: 0x903ac0c5 (stale) Mirror 5: 0x3155144c (stale) CRC-32 checksum value for chunk [0, 0x380a7): Mirror 1: 0xe356d8bc Mirror 6: 0xe356d8bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380a7: skipped 000083 1772176380.637057 trunc from 0x0380a7 to 0x02ba01 (0xc6a5 bytes) 000084 1772176380.674708 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ba02): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba02): Mirror 3: 0x6ccb522d (stale) Mirror 4: 0x58636ece (stale) Mirror 5: 0xc74045f0 (stale) Mirror 6: 0xc74045f0 (stale) CRC-32 checksum value for chunk [0, 0x2ba02): Mirror 1: 0xc74045f0 Mirror 7: 0xc74045f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba02: skipped 000086 1772176381.362600 punch from 0x00727b to 0x011150 (0x9ed6 bytes) 000087 1772176381.394375 mapread 0x01ab9e thru 0x0224f0 (0x7953 bytes) 000089 1772176381.410343 mapread 0x006ecb thru 0x0115e6 (0xa71c bytes) 000090 1772176381.427609 mapread 0x00ec59 thru 0x01474e (0x5af6 bytes) 000091 1772176381.428769 write 0x02b6c3 thru 0x03116e (0x5aac bytes) 000092 1772176381.446596 mapwrite 0x02074a thru 0x027c88 (0x753f bytes) 000093 1772176381.474414 write 0x029ae8 thru 0x032784 (0x8c9d bytes) 000094 1772176381.482867 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32785): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32785): Mirror 3: 0x85c3f4f5 (stale) Mirror 4: 0xeca552d2 (stale) Mirror 5: 0x47c609f3 (stale) Mirror 6: 0x47c609f3 (stale) Mirror 7: 0x47c609f3 (stale) CRC-32 checksum value for chunk [0, 0x32785): Mirror 1: 0xbd09dbc9 Mirror 8: 0xbd09dbc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32785: skipped 000095 1772176382.106827 fallocate 0x00819d thru 0x014de8 (0xcc4c bytes) 000097 1772176382.130298 read 0x011768 thru 0x0183a3 (0x6c3c bytes) 000098 1772176382.136054 fallocate 0x012ddd thru 0x0211e5 (0xe409 bytes) 000099 1772176382.138753 mapwrite 0x02aa00 thru 0x02c82b (0x1e2c bytes) 000100 1772176382.158561 punch from 0x005dd9 to 0x007503 (0x172b bytes) 000101 1772176382.162020 mapwrite 0x005b40 thru 0x010ec3 (0xb384 bytes) 000102 1772176382.181582 trunc from 0x0291cd to 0x031882 (0x86b6 bytes) 000103 1772176382.186192 trunc from 0x02595f to 0x0306ff (0xada1 bytes) 000104 1772176382.193286 MIRROR_RESYNC 7 mirrors 000105 1772176383.561316 mapwrite 0x021236 thru 0x02de1f (0xcbea bytes) 000107 1772176383.683838 write 0x032663 thru 0x03f7fb (0xd199 bytes) 000108 1772176383.687057 mapread 0x03682f thru 0x03f7fb (0x8fcd bytes) 000109 1772176383.688563 trunc from 0x032054 to 0x03f7fb (0xd7a8 bytes) 000110 1772176383.705682 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f7fc): Mirror 3: 0x4ef7efe5 (stale) Mirror 4: 0x4ef7efe5 (stale) Mirror 6: 0x4ef7efe5 (stale) Mirror 7: 0x4ef7efe5 (stale) Mirror 8: 0x4ef7efe5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7fc): Mirror 3: 0x4ef7efe5 (stale) Mirror 4: 0x4ef7efe5 (stale) Mirror 6: 0x4ef7efe5 (stale) Mirror 7: 0x4ef7efe5 (stale) Mirror 8: 0x4ef7efe5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7fc: skipped 000111 1772176384.045451 read 0x03f630 thru 0x03f7fb (0x1cc bytes) 000112 1772176384.052399 trunc from 0x03f7fc to 0x004cd9 (0x3ab22 bytes) 000113 1772176384.075579 mapwrite 0x003fe2 thru 0x00475f (0x77e bytes) 000114 1772176384.099055 write 0x011013 thru 0x01198a (0x978 bytes) 000116 1772176384.100661 read 0x009927 thru 0x01198a (0x8064 bytes) 000117 1772176384.116865 MIRROR_RESYNC 6 mirrors 000119 1772176385.058884 read 0x006a69 thru 0x00be33 (0x53cb bytes) 000120 1772176385.070497 MIRROR_RESYNC 6 mirrors 000123 1772176385.310456 mapwrite 0x02ce10 thru 0x03770c (0xa8fd bytes) 000124 1772176385.391164 read 0x01c146 thru 0x01fc9a (0x3b55 bytes) 000126 1772176385.398243 mapwrite 0x028470 thru 0x02eb15 (0x66a6 bytes) 000127 1772176385.425317 mapwrite 0x014a90 thru 0x02338a (0xe8fb bytes) 000128 1772176385.467259 mapwrite 0x029999 thru 0x032fb5 (0x961d bytes) 000129 1772176385.490802 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3770d): Mirror 3: 0x4c74874 (stale) Mirror 6: 0x4c74874 (stale) Mirror 7: 0x4c74874 (stale) Mirror 8: 0x4c74874 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3770d): Mirror 3: 0x4c74874 (stale) Mirror 6: 0x4c74874 (stale) Mirror 7: 0x4c74874 (stale) Mirror 8: 0x4c74874 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3770d: skipped 000130 1772176385.812299 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3770d): Mirror 6: 0x4c74874 (stale) Mirror 7: 0x4c74874 (stale) Mirror 8: 0x4c74874 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3770d): Mirror 6: 0x4c74874 (stale) Mirror 7: 0x4c74874 (stale) Mirror 8: 0x4c74874 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3770d: skipped 000131 1772176386.056183 trunc from 0x01aff2 to 0x021ca6 (0x6cb5 bytes) 000133 1772176386.060530 trunc from 0x03770d to 0x0326fc (0x5010 bytes) 000134 1772176386.077875 trunc from 0x018f3b to 0x01e2a2 (0x5368 bytes) 000135 1772176386.081932 write 0x00c700 thru 0x00cacf (0x3d0 bytes) 000136 1772176386.086168 fallocate 0x0279e7 thru 0x02a901 (0x2f1b bytes) 000137 1772176386.090181 trunc from 0x021aad to 0x02fd39 (0xe28d bytes) 000138 1772176386.095423 trunc from 0x00bcc1 to 0x01122c (0x556c bytes) 000139 1772176386.102471 trunc from 0x0326fd to 0x0117eb (0x20f11 bytes) 000142 1772176386.117287 trunc from 0x0084df to 0x010676 (0x8198 bytes) 000143 1772176386.124907 read 0x00213e thru 0x002b32 (0x9f5 bytes) 000144 1772176386.129476 mapwrite 0x01caa2 thru 0x02959c (0xcafb bytes) 000145 1772176386.178076 punch from 0x028c68 to 0x02959c (0x935 bytes) 000146 1772176386.185717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2959d): Mirror 7: 0xc5588c49 (stale) Mirror 8: 0xc5588c49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2959d): Mirror 7: 0xc5588c49 (stale) Mirror 8: 0xc5588c49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2959d: skipped 000147 1772176386.416955 trunc from 0x012438 to 0x01cd97 (0xa960 bytes) 000148 1772176386.420396 mapread 0x0253e5 thru 0x0274fd (0x2119 bytes) 000149 1772176386.430896 trunc from 0x001e63 to 0x008c45 (0x6de3 bytes) 000150 1772176386.434630 trunc from 0x006c57 to 0x0076ff (0xaa9 bytes) 000152 1772176386.437740 trunc from 0x02959d to 0x02cf10 (0x3974 bytes) 000153 1772176386.461602 write 0x032580 thru 0x03451a (0x1f9b bytes) 000154 1772176386.462725 mapread 0x009c20 thru 0x0193e3 (0xf7c4 bytes) 000155 1772176386.480991 mapread 0x012883 thru 0x01fb6d (0xd2eb bytes) 000156 1772176386.490011 mapwrite 0x01aab6 thru 0x026bce (0xc119 bytes) 000157 1772176386.513254 mapwrite 0x03ed3b thru 0x03ffff (0x12c5 bytes) 000158 1772176386.545705 write 0x030b62 thru 0x031f1c (0x13bb bytes) 000159 1772176386.551791 write 0x00ef9c thru 0x01ca9d (0xdb02 bytes) 000160 1772176386.552917 trunc from 0x03f978 to 0x03ffff (0x688 bytes) 000161 1772176386.556577 mapread 0x039e51 thru 0x03ffff (0x61af bytes) 000162 1772176386.563078 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xbd29cbc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xbd29cbc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000163 1772176386.777105 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000164 1772176386.903372 mapread 0x00fb38 thru 0x017889 (0x7d52 bytes) 000165 1772176386.923577 fallocate 0x002081 thru 0x00d799 (0xb719 bytes) 000166 1772176386.928259 read 0x0046e5 thru 0x00966a (0x4f86 bytes) 000167 1772176386.941673 punch from 0x016fc3 to 0x025b37 (0xeb75 bytes) 000168 1772176386.947253 trunc from 0x03845f to 0x03ffff (0x7ba1 bytes) 000169 1772176386.952057 read 0x01df92 thru 0x025a18 (0x7a87 bytes) 000170 1772176386.965395 write 0x006f71 thru 0x00d541 (0x65d1 bytes) 000171 1772176386.970420 mapread 0x015bfc thru 0x01dae1 (0x7ee6 bytes) 000173 1772176386.973045 mapwrite 0x03e67e thru 0x03fa8f (0x1412 bytes) 000175 1772176386.994081 trunc from 0x000045 to 0x00a52e (0xa4ea bytes) 000176 1772176386.998874 read 0x0139c4 thru 0x022a6d (0xf0aa bytes) 000177 1772176386.999697 fallocate 0x0120ce thru 0x0164d0 (0x4403 bytes) 000179 1772176387.006667 trunc from 0x018ecc to 0x020cb5 (0x7dea bytes) 000180 1772176387.012697 trunc from 0x040000 to 0x02220f (0x1ddf0 bytes) 000181 1772176387.032155 write 0x0010dd thru 0x004ba6 (0x3aca bytes) 000182 1772176387.050867 MIRROR_EXTEND to 2 mirrors 000183 1772176387.626833 trunc from 0x006a96 to 0x009d19 (0x3284 bytes) 000184 1772176387.681122 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000185 1772176387.808391 trunc from 0x022210 to 0x02c6b6 (0xa4a7 bytes) 000186 1772176387.826467 fallocate 0x014d3b thru 0x015135 (0x3fb bytes) 000187 1772176387.832803 mapwrite 0x002f4e thru 0x00d860 (0xa913 bytes) 000188 1772176387.873861 trunc from 0x02bff2 to 0x02c6b6 (0x6c5 bytes) 000189 1772176387.878831 fallocate 0x02eacf thru 0x030ad9 (0x200b bytes) 000190 1772176387.882915 read 0x00b08c thru 0x019b24 (0xea99 bytes) 000192 1772176387.897390 mapwrite 0x00df24 thru 0x0143d5 (0x64b2 bytes) 000193 1772176387.917575 trunc from 0x0056b3 to 0x014e9d (0xf7eb bytes) 000194 1772176387.923405 fallocate 0x016066 thru 0x01fb5d (0x9af8 bytes) 000195 1772176387.929430 punch from 0x01824b to 0x01de63 (0x5c19 bytes) 000196 1772176387.935505 write 0x03ba17 thru 0x03ffff (0x45e9 bytes) 000197 1772176387.937488 read 0x01de76 thru 0x01f538 (0x16c3 bytes) 000198 1772176387.951711 punch from 0x02371d to 0x02b83e (0x8122 bytes) 000199 1772176387.961464 fallocate 0x0106f9 thru 0x01477d (0x4085 bytes) 000200 1772176387.970745 trunc from 0x03c155 to 0x03ffff (0x3eab bytes) 000201 1772176387.990430 mapwrite 0x000c9c thru 0x0103b3 (0xf718 bytes) 000202 1772176388.025700 fallocate 0x03484a thru 0x03e5a0 (0x9d57 bytes) 000203 1772176388.035315 mapwrite 0x007847 thru 0x0112e7 (0x9aa1 bytes) 000204 1772176388.064614 punch from 0x00e66f to 0x01c94f (0xe2e1 bytes) 000205 1772176388.073692 trunc from 0x040000 to 0x03fa5a (0x5a5 bytes) 000206 1772176388.091696 read 0x01b885 thru 0x01cf45 (0x16c1 bytes) 000207 1772176388.101651 mapread 0x004a4a thru 0x00948f (0x4a46 bytes) 000209 1772176388.103206 mapread 0x03d0d5 thru 0x03fa5a (0x2986 bytes) 000210 1772176388.104087 trunc from 0x03fa5b to 0x02eeaa (0x10bb0 bytes) 000211 1772176388.126600 punch from 0x0284ef to 0x02eeaa (0x69bc bytes) 000212 1772176388.134627 trunc from 0x02752e to 0x02eeaa (0x797d bytes) 000213 1772176388.139376 write 0x02c8f9 thru 0x03c386 (0xfa8e bytes) 000214 1772176388.145761 punch from 0x00ee3c to 0x01936e (0xa533 bytes) 000216 1772176388.153585 mapread 0x015755 thru 0x015b61 (0x40d bytes) 000217 1772176388.159149 punch from 0x03a65f to 0x03c386 (0x1d28 bytes) 000218 1772176388.178987 write 0x009896 thru 0x0148c1 (0xb02c bytes) 000219 1772176388.194835 MIRROR_EXTEND to 2 mirrors 000220 1772176388.725053 trunc from 0x015db7 to 0x01d139 (0x7383 bytes) 000221 1772176388.761421 trunc from 0x03c387 to 0x02efa5 (0xd3e1 bytes) 000223 1772176388.804179 mapread 0x02910f thru 0x02d5ce (0x44c0 bytes) 000224 1772176388.834264 read 0x0140c6 thru 0x02043b (0xc376 bytes) 000225 1772176388.856502 MIRROR_RESYNC 2 mirrors 000226 1772176389.280793 write 0x00b25b thru 0x0161cc (0xaf72 bytes) 000227 1772176389.335671 mapwrite 0x0127c9 thru 0x019499 (0x6cd1 bytes) 000228 1772176389.371316 fallocate 0x0384be thru 0x03ffff (0x7b42 bytes) 000229 1772176389.376250 read 0x00388e thru 0x00f561 (0xbcd4 bytes) 000230 1772176389.384490 trunc from 0x040000 to 0x01ba65 (0x2459a bytes) 000232 1772176389.408196 trunc from 0x01b758 to 0x01ba65 (0x30e bytes) 000233 1772176389.413802 mapread 0x016587 thru 0x018ef2 (0x296c bytes) 000234 1772176389.415938 trunc from 0x00d65d to 0x0124a6 (0x4e4a bytes) 000235 1772176389.421221 trunc from 0x01ba66 to 0x01ec9b (0x3236 bytes) 000236 1772176389.438874 write 0x0216d0 thru 0x023210 (0x1b41 bytes) 000238 1772176389.445766 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000239 1772176389.582651 trunc from 0x023211 to 0x014e67 (0xe3a9 bytes) 000240 1772176389.605571 trunc from 0x014e68 to 0x0328c3 (0x1da5c bytes) 000241 1772176389.625868 mapwrite 0x003f7c thru 0x006806 (0x288b bytes) 000242 1772176389.666881 write 0x0131e9 thru 0x01e815 (0xb62d bytes) 000243 1772176389.677690 trunc from 0x0328c4 to 0x03d3ce (0xab0b bytes) 000244 1772176389.702439 trunc from 0x017b36 to 0x01e9d0 (0x6e9b bytes) 000245 1772176389.716219 mapwrite 0x00c78b thru 0x00fff3 (0x3869 bytes) 000246 1772176389.730410 trunc from 0x02ae35 to 0x037dbc (0xcf88 bytes) 000247 1772176389.735748 punch from 0x030fb5 to 0x03d1f0 (0xc23c bytes) 000248 1772176389.741148 trunc from 0x03d3cf to 0x011a2f (0x2b99f bytes) 000249 1772176389.757446 punch from 0x0033d9 to 0x01000b (0xcc33 bytes) 000251 1772176389.770448 MIRROR_EXTEND to 2 mirrors 000252 1772176390.439236 trunc from 0x00bc67 to 0x011a2f (0x5dc9 bytes) 000253 1772176390.470800 punch from 0x010cdb to 0x011a2f (0xd55 bytes) 000254 1772176390.479008 mapread 0x00a759 thru 0x0113ce (0x6c76 bytes) 000256 1772176390.498490 trunc from 0x011a30 to 0x019fa4 (0x8575 bytes) 000257 1772176390.523222 MIRROR_RESYNC 2 mirrors 000258 1772176391.020286 write 0x03e07f thru 0x03ffff (0x1f81 bytes) 000259 1772176391.057307 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3a260855 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd65b9f61 Mirror 3: 0xd65b9f61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000260 1772176391.694305 punch from 0x014dc7 to 0x014ff0 (0x22a bytes) 000261 1772176391.731438 trunc from 0x040000 to 0x02ba49 (0x145b6 bytes) 000262 1772176391.755685 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba4a): Mirror 3: 0xd1577c81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba4a): Mirror 3: 0xd1577c81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba4a: skipped 000263 1772176391.943967 write 0x02799a thru 0x02b81c (0x3e83 bytes) 000264 1772176391.952466 mapwrite 0x00c8d5 thru 0x00dab9 (0x11e5 bytes) 000265 1772176391.972324 read 0x0105f0 thru 0x01bb54 (0xb565 bytes) 000267 1772176391.978279 trunc from 0x02ba4a to 0x0012df (0x2a76a bytes) 000268 1772176391.990857 punch from 0x000941 to 0x0012df (0x99f bytes) 000270 1772176392.002422 write 0x015b24 thru 0x01ab35 (0x5012 bytes) 000271 1772176392.004026 trunc from 0x00fef0 to 0x017b2b (0x7c3c bytes) 000272 1772176392.018055 punch from 0x014e77 to 0x01ab35 (0x5cbf bytes) 000273 1772176392.022699 trunc from 0x01ab36 to 0x004ec4 (0x15c71 bytes) 000274 1772176392.031939 read 0x0018e2 thru 0x004ec4 (0x35e3 bytes) 000275 1772176392.037137 fallocate 0x01e995 thru 0x026d81 (0x83ed bytes) 000276 1772176392.043830 fallocate 0x030a54 thru 0x03ee98 (0xe445 bytes) 000277 1772176392.050631 fallocate 0x015127 thru 0x01abc9 (0x5aa3 bytes) 000278 1772176392.055640 trunc from 0x0229d0 to 0x023722 (0xd53 bytes) 000279 1772176392.060294 punch from 0x006588 to 0x00b2b1 (0x4d2a bytes) 000280 1772176392.065462 fallocate 0x01d582 thru 0x026606 (0x9085 bytes) 000281 1772176392.071320 trunc from 0x03ee99 to 0x021347 (0x1db51 bytes) 000282 1772176392.085385 write 0x02786e thru 0x02a152 (0x28e5 bytes) 000283 1772176392.086354 fallocate 0x019c14 thru 0x021ae6 (0x7ed3 bytes) 000284 1772176392.092723 trunc from 0x00841d to 0x015d34 (0xd918 bytes) 000285 1772176392.100737 mapread 0x0165ed thru 0x02520b (0xec1f bytes) 000287 1772176392.114558 read 0x0231dc thru 0x02a152 (0x6f77 bytes) 000288 1772176392.116075 write 0x003db8 thru 0x009cc6 (0x5f0f bytes) 000289 1772176392.117872 trunc from 0x02a153 to 0x035d0e (0xbbbc bytes) 000290 1772176392.133165 trunc from 0x035d0f to 0x020284 (0x15a8a bytes) 000291 1772176392.153338 trunc from 0x01e135 to 0x020284 (0x2150 bytes) 000292 1772176392.157413 trunc from 0x020285 to 0x002591 (0x1dcf3 bytes) 000293 1772176392.179415 fallocate 0x01a6a6 thru 0x01eb88 (0x44e3 bytes) 000294 1772176392.193827 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000295 1772176392.318646 mapread 0x00640f thru 0x00e0bc (0x7cae bytes) 000297 1772176392.341465 mapwrite 0x02cdac thru 0x02df51 (0x11a6 bytes) 000298 1772176392.381447 trunc from 0x00a725 to 0x011cfb (0x75d7 bytes) 000300 1772176392.388124 read 0x014a0e thru 0x0244da (0xfacd bytes) 000302 1772176392.404286 write 0x0263de thru 0x0331cf (0xcdf2 bytes) 000303 1772176392.407405 trunc from 0x031787 to 0x0331cf (0x1a49 bytes) 000304 1772176392.420028 read 0x000378 thru 0x008694 (0x831d bytes) 000305 1772176392.426854 read 0x008875 thru 0x013fa4 (0xb730 bytes) 000307 1772176392.428129 write 0x0287a2 thru 0x034451 (0xbcb0 bytes) 000308 1772176392.430768 punch from 0x01eb11 to 0x023d27 (0x5217 bytes) 000309 1772176392.435267 write 0x03c312 thru 0x03ffff (0x3cee bytes) 000310 1772176392.436573 trunc from 0x02cd89 to 0x02e6d4 (0x194c bytes) 000311 1772176392.456576 MIRROR_EXTEND to 2 mirrors 000312 1772176392.978421 write 0x03621e thru 0x03769e (0x1481 bytes) 000313 1772176393.030480 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xee545118 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb264edc1 Mirror 3: 0xb264edc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000314 1772176393.752883 trunc from 0x040000 to 0x00e212 (0x31ded bytes) 000315 1772176393.781620 trunc from 0x00e213 to 0x014080 (0x5e6e bytes) 000316 1772176393.795217 mapwrite 0x03738e thru 0x03a568 (0x31db bytes) 000317 1772176393.829133 read 0x0036db thru 0x00bc9a (0x85c0 bytes) 000318 1772176393.840407 trunc from 0x03a569 to 0x0038c7 (0x36ca1 bytes) 000319 1772176393.851351 read 0x000b74 thru 0x0016e7 (0xb74 bytes) 000320 1772176393.855931 fallocate 0x008eab thru 0x00f79e (0x68f4 bytes) 000321 1772176393.859587 write 0x01b2da thru 0x024ffe (0x9d25 bytes) 000322 1772176393.861036 punch from 0x01041d to 0x01bdd4 (0xb9b8 bytes) 000323 1772176393.873775 trunc from 0x00be52 to 0x00f0e5 (0x3294 bytes) 000324 1772176393.876397 punch from 0x0066ba to 0x011286 (0xabcd bytes) 000325 1772176393.881790 mapwrite 0x01f792 thru 0x0245fd (0x4e6c bytes) 000326 1772176393.901845 mapwrite 0x00afc9 thru 0x0177ad (0xc7e5 bytes) 000327 1772176393.948213 read 0x01ce08 thru 0x0228da (0x5ad3 bytes) 000328 1772176393.949136 trunc from 0x024fff to 0x0220cd (0x2f31 bytes) 000329 1772176393.965498 punch from 0x00e51f to 0x011c69 (0x374b bytes) 000330 1772176393.970722 read 0x004ced thru 0x009119 (0x442d bytes) 000331 1772176393.971558 mapread 0x01e15b thru 0x0220cd (0x3f73 bytes) 000332 1772176393.972687 punch from 0x019f10 to 0x0220cd (0x81be bytes) 000333 1772176393.979248 fallocate 0x012773 thru 0x01ef2d (0xc7bb bytes) 000334 1772176393.984623 read 0x012cce thru 0x015b7a (0x2ead bytes) 000337 1772176393.990270 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x220ce): Mirror 3: 0xed1349d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x220ce): Mirror 3: 0xed1349d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x220ce: skipped 000338 1772176394.229302 trunc from 0x01db53 to 0x0220cd (0x457b bytes) 000339 1772176394.234578 read 0x01f3f4 thru 0x0220cd (0x2cda bytes) 000340 1772176394.241466 fallocate 0x025c61 thru 0x032714 (0xcab4 bytes) 000341 1772176394.246273 mapwrite 0x012d73 thru 0x01c12f (0x93bd bytes) 000342 1772176394.281293 read 0x01d54d thru 0x022ffd (0x5ab1 bytes) 000343 1772176394.297202 read 0x02751c thru 0x032714 (0xb1f9 bytes) 000344 1772176394.305418 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32715): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32715): Mirror 3: 0x99b42c1 (stale) CRC-32 checksum value for chunk [0, 0x32715): Mirror 1: 0x2d97afdc Mirror 4: 0x2d97afdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32715: skipped 000345 1772176395.016990 trunc from 0x032715 to 0x02ba76 (0x6c9e bytes) 000346 1772176395.104278 trunc from 0x02ba77 to 0x017d51 (0x13d25 bytes) 000348 1772176395.131014 write 0x037d5b thru 0x03a6f3 (0x2999 bytes) 000349 1772176395.132267 mapwrite 0x000261 thru 0x00581e (0x55be bytes) 000350 1772176395.194437 punch from 0x017cee to 0x02177a (0x9a8d bytes) 000351 1772176395.201818 mapwrite 0x020492 thru 0x02baf8 (0xb667 bytes) 000352 1772176395.241373 MIRROR_RESYNC 3 mirrors 000353 1772176396.281282 trunc from 0x034ef4 to 0x03a6f3 (0x5800 bytes) 000354 1772176396.315230 mapwrite 0x0388b2 thru 0x03ffff (0x774e bytes) 000356 1772176396.371799 fallocate 0x027668 thru 0x0361c8 (0xeb61 bytes) 000357 1772176396.376069 trunc from 0x040000 to 0x019eee (0x26111 bytes) 000358 1772176396.402827 fallocate 0x03d572 thru 0x03ffff (0x2a8e bytes) 000359 1772176396.409481 read 0x00f592 thru 0x01e7b5 (0xf224 bytes) 000360 1772176396.423896 write 0x01a1f5 thru 0x0291b1 (0xefbd bytes) 000361 1772176396.432255 trunc from 0x03fa4f to 0x03ffff (0x5b1 bytes) 000362 1772176396.449505 MIRROR_RESYNC 3 mirrors 000363 1772176397.344841 mapwrite 0x0253dc thru 0x033c76 (0xe89b bytes) 000364 1772176397.449692 write 0x024653 thru 0x02b5c6 (0x6f74 bytes) 000365 1772176397.452314 write 0x020fd1 thru 0x02790d (0x693d bytes) 000366 1772176397.460361 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54fad712 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54fad712 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000367 1772176397.736728 trunc from 0x034f31 to 0x03ced4 (0x7fa4 bytes) 000368 1772176397.741776 read 0x033589 thru 0x03ffff (0xca77 bytes) 000369 1772176397.754039 punch from 0x013389 to 0x0135c1 (0x239 bytes) 000370 1772176397.759047 read 0x002974 thru 0x010d3f (0xe3cc bytes) 000371 1772176397.766198 trunc from 0x004e37 to 0x00670c (0x18d6 bytes) 000372 1772176397.770549 mapwrite 0x019cc2 thru 0x01c9e3 (0x2d22 bytes) 000375 1772176397.781316 read 0x02a112 thru 0x033a34 (0x9923 bytes) 000376 1772176397.782435 fallocate 0x008cdf thru 0x00ce6d (0x418f bytes) 000377 1772176397.786151 punch from 0x032a86 to 0x03e788 (0xbd03 bytes) 000378 1772176397.790503 write 0x000636 thru 0x00de6b (0xd836 bytes) 000379 1772176397.795582 trunc from 0x008e1a to 0x0181fc (0xf3e3 bytes) 000380 1772176397.824496 mapwrite 0x027463 thru 0x0315b3 (0xa151 bytes) 000381 1772176397.850948 punch from 0x00d06a to 0x01b1d8 (0xe16f bytes) 000382 1772176397.857727 write 0x037519 thru 0x03d8d0 (0x63b8 bytes) 000383 1772176397.874355 read 0x009f68 thru 0x00bdce (0x1e67 bytes) 000384 1772176397.875170 trunc from 0x022ae7 to 0x02d63c (0xab56 bytes) 000385 1772176397.881426 write 0x017840 thru 0x01acbb (0x347c bytes) 000386 1772176397.882465 mapread 0x000a2e thru 0x001f91 (0x1564 bytes) 000387 1772176397.888041 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000388 1772176398.013334 mapread 0x03d1f4 thru 0x03ffff (0x2e0c bytes) 000389 1772176398.027446 mapread 0x0096b4 thru 0x015f06 (0xc853 bytes) 000390 1772176398.046500 mapread 0x03138f thru 0x036a08 (0x567a bytes) 000391 1772176398.058893 trunc from 0x040000 to 0x018a3e (0x275c1 bytes) 000392 1772176398.077446 trunc from 0x00db47 to 0x00e00b (0x4c5 bytes) 000393 1772176398.082253 trunc from 0x013105 to 0x018a3e (0x593a bytes) 000394 1772176398.092600 mapwrite 0x0041de thru 0x00c9da (0x87fd bytes) 000395 1772176398.111576 trunc from 0x018a3f to 0x01005b (0x89e3 bytes) 000396 1772176398.139481 write 0x03d12c thru 0x03ffff (0x2ed4 bytes) 000397 1772176398.140908 fallocate 0x013b56 thru 0x01d734 (0x9bdf bytes) 000398 1772176398.156003 trunc from 0x040000 to 0x023b45 (0x1c4ba bytes) 000399 1772176398.175208 trunc from 0x0013d3 to 0x00f270 (0xde9e bytes) 000401 1772176398.181382 mapwrite 0x02b5bb thru 0x037c88 (0xc6ce bytes) 000402 1772176398.231120 trunc from 0x01dd1b to 0x02882b (0xab11 bytes) 000403 1772176398.235831 punch from 0x009632 to 0x00f3cc (0x5d9b bytes) 000404 1772176398.239650 read 0x006dc9 thru 0x015f87 (0xf1bf bytes) 000405 1772176398.247045 write 0x0032cc thru 0x008659 (0x538e bytes) 000406 1772176398.255748 mapwrite 0x033a11 thru 0x03ffff (0xc5ef bytes) 000408 1772176398.299660 write 0x031946 thru 0x037457 (0x5b12 bytes) 000409 1772176398.302457 write 0x010c03 thru 0x020a51 (0xfe4f bytes) 000410 1772176398.310034 fallocate 0x021126 thru 0x02bbba (0xaa95 bytes) 000411 1772176398.316739 read 0x02412b thru 0x02635b (0x2231 bytes) 000412 1772176398.323337 punch from 0x02b857 to 0x03a35d (0xeb07 bytes) 000413 1772176398.344657 mapread 0x011ce2 thru 0x0152d2 (0x35f1 bytes) 000414 1772176398.347998 write 0x039004 thru 0x03c66f (0x366c bytes) 000415 1772176398.353958 write 0x024f87 thru 0x032282 (0xd2fc bytes) 000416 1772176398.370011 fallocate 0x011aa1 thru 0x01b494 (0x99f4 bytes) 000417 1772176398.376010 trunc from 0x023583 to 0x0329e0 (0xf45e bytes) 000418 1772176398.396396 trunc from 0x03da7c to 0x03ffff (0x2584 bytes) 000419 1772176398.401705 mapwrite 0x0197c6 thru 0x02493b (0xb176 bytes) 000420 1772176398.431297 fallocate 0x01b722 thru 0x01c2a2 (0xb81 bytes) 000421 1772176398.435722 read 0x01fe59 thru 0x0278a5 (0x7a4d bytes) 000422 1772176398.444077 read 0x034309 thru 0x03acfb (0x69f3 bytes) 000423 1772176398.449555 MIRROR_EXTEND to 2 mirrors 000424 1772176399.037873 fallocate 0x00a8d1 thru 0x00fd73 (0x54a3 bytes) 000425 1772176399.075394 punch from 0x0273cb to 0x02a90e (0x3544 bytes) 000426 1772176399.084034 trunc from 0x040000 to 0x01028f (0x2fd70 bytes) 000427 1772176399.124451 mapwrite 0x010cd1 thru 0x01bf37 (0xb267 bytes) 000428 1772176399.177774 trunc from 0x01bf38 to 0x036ac4 (0x1ab8d bytes) 000430 1772176399.192620 write 0x003019 thru 0x0116f2 (0xe6da bytes) 000431 1772176399.204361 trunc from 0x036ac5 to 0x020425 (0x1669f bytes) 000432 1772176399.222025 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000434 1772176399.382000 write 0x0320a6 thru 0x03ffff (0xdf5a bytes) 000435 1772176399.389302 mapwrite 0x007533 thru 0x00835f (0xe2d bytes) 000436 1772176399.431305 read 0x02ff2a thru 0x03e2a7 (0xe37e bytes) 000437 1772176399.437780 punch from 0x03039e to 0x03e1fb (0xde5e bytes) 000438 1772176399.443827 mapwrite 0x03e1ab thru 0x03ffff (0x1e55 bytes) 000439 1772176399.465903 trunc from 0x040000 to 0x01972f (0x268d0 bytes) 000441 1772176399.485854 punch from 0x009f14 to 0x00b681 (0x176e bytes) 000442 1772176399.490117 fallocate 0x01aea9 thru 0x01aff6 (0x14e bytes) 000443 1772176399.495168 read 0x0043b9 thru 0x008b66 (0x47ae bytes) 000445 1772176399.506158 mapread 0x00fb79 thru 0x01a259 (0xa6e1 bytes) 000446 1772176399.507600 mapwrite 0x00eb86 thru 0x00ec54 (0xcf bytes) 000447 1772176399.518465 write 0x008697 thru 0x00f88c (0x71f6 bytes) 000449 1772176399.522181 trunc from 0x00963e to 0x015b4e (0xc511 bytes) 000450 1772176399.539189 punch from 0x0118d4 to 0x01aff6 (0x9723 bytes) 000451 1772176399.544354 mapwrite 0x0217ca thru 0x02f8ee (0xe125 bytes) 000452 1772176399.594004 write 0x036b97 thru 0x03ffff (0x9469 bytes) 000453 1772176399.596845 write 0x01f0e2 thru 0x01f9c9 (0x8e8 bytes) 000454 1772176399.600235 trunc from 0x040000 to 0x000065 (0x3ff9a bytes) 000455 1772176399.617153 read 0x000037 thru 0x000065 (0x2f bytes) 000456 1772176399.618262 read 0x000056 thru 0x000065 (0x10 bytes) 000457 1772176399.628029 MIRROR_EXTEND to 2 mirrors 000459 1772176400.043045 fallocate 0x02973f thru 0x037902 (0xe1c4 bytes) 000461 1772176400.087274 mapwrite 0x0277d8 thru 0x02ac1b (0x3444 bytes) 000462 1772176400.127961 mapread 0x01a800 thru 0x0230fd (0x88fe bytes) 000463 1772176400.135218 write 0x037fbf thru 0x03d188 (0x51ca bytes) 000467 1772176400.149697 fallocate 0x036108 thru 0x03d731 (0x762a bytes) 000468 1772176400.154617 punch from 0x01109d to 0x01e2ea (0xd24e bytes) 000470 1772176400.163199 punch from 0x02b0e9 to 0x039f4f (0xee67 bytes) 000471 1772176400.176719 mapread 0x03881a thru 0x03a598 (0x1d7f bytes) 000472 1772176400.185314 read 0x011821 thru 0x01bfd2 (0xa7b2 bytes) 000474 1772176400.200218 mapread 0x021869 thru 0x02a056 (0x87ee bytes) 000475 1772176400.208892 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000476 1772176400.324446 fallocate 0x02afae thru 0x03029e (0x52f1 bytes) 000478 1772176400.330028 mapread 0x0276a8 thru 0x03065f (0x8fb8 bytes) 000481 1772176400.349069 mapread 0x034a5a thru 0x039908 (0x4eaf bytes) 000482 1772176400.359357 trunc from 0x03d732 to 0x02faea (0xdc47 bytes) 000483 1772176400.414753 write 0x00bf65 thru 0x0123b8 (0x6454 bytes) 000484 1772176400.427008 read 0x01a8a5 thru 0x01b826 (0xf82 bytes) 000485 1772176400.435029 mapwrite 0x038867 thru 0x03ffff (0x7799 bytes) 000486 1772176400.479273 fallocate 0x01db77 thru 0x027f41 (0xa3cb bytes) 000487 1772176400.483499 write 0x00811a thru 0x011475 (0x935c bytes) 000488 1772176400.485703 punch from 0x017235 to 0x0200e5 (0x8eb1 bytes) 000489 1772176400.490398 mapread 0x01d707 thru 0x025cd6 (0x85d0 bytes) 000490 1772176400.499169 read 0x00e3d2 thru 0x010369 (0x1f98 bytes) 000491 1772176400.499825 mapread 0x03adbf thru 0x03ffff (0x5241 bytes) 000492 1772176400.501747 read 0x01b6fe thru 0x028781 (0xd084 bytes) 000494 1772176400.503273 fallocate 0x00f952 thru 0x01b2ba (0xb969 bytes) 000495 1772176400.508774 mapread 0x03ea76 thru 0x03ffff (0x158a bytes) 000496 1772176400.509852 fallocate 0x03e5ec thru 0x03ffff (0x1a14 bytes) 000497 1772176400.516982 trunc from 0x040000 to 0x007229 (0x38dd6 bytes) 000498 1772176400.538040 punch from 0x002f9a to 0x007229 (0x4290 bytes) 000499 1772176400.543632 mapwrite 0x013efa thru 0x017924 (0x3a2b bytes) 000500 1772176400.599376 write 0x008a90 thru 0x015ac1 (0xd032 bytes) 000501 1772176400.605351 punch from 0x00b5e0 to 0x010c0d (0x562e bytes) 000502 1772176400.618083 trunc from 0x017925 to 0x004da2 (0x12b82 bytes) 000503 1772176400.633007 trunc from 0x001e44 to 0x004cf8 (0x2eb5 bytes) 000504 1772176400.637359 write 0x01204e thru 0x016d76 (0x4d29 bytes) 000505 1772176400.639059 fallocate 0x000822 thru 0x00848b (0x7c6a bytes) 000507 1772176400.643851 trunc from 0x016d77 to 0x0264b0 (0xf73a bytes) 000508 1772176400.658805 read 0x0223a3 thru 0x0264b0 (0x410e bytes) 000509 1772176400.670927 mapread 0x0184f7 thru 0x01e04a (0x5b54 bytes) 000510 1772176400.672731 mapread 0x01a037 thru 0x0200d9 (0x60a3 bytes) 000512 1772176400.675333 trunc from 0x0264b1 to 0x01c036 (0xa47a bytes) 000513 1772176400.696823 punch from 0x00ac20 to 0x014ee8 (0xa2c9 bytes) 000514 1772176400.718724 MIRROR_EXTEND to 2 mirrors 000515 1772176401.149930 mapwrite 0x01e705 thru 0x01e97a (0x276 bytes) 000516 1772176401.231493 punch from 0x0088d0 to 0x00db56 (0x5287 bytes) 000518 1772176401.240086 punch from 0x014f6a to 0x01e327 (0x93be bytes) 000519 1772176401.248706 mapread 0x010577 thru 0x017ed0 (0x795a bytes) 000520 1772176401.275219 fallocate 0x02b34d thru 0x02b99b (0x64f bytes) 000521 1772176401.286024 MIRROR_RESYNC 2 mirrors 000522 1772176401.688683 read 0x004653 thru 0x00cfe2 (0x8990 bytes) 000523 1772176401.695822 write 0x01e4be thru 0x02bf11 (0xda54 bytes) 000524 1772176401.736369 trunc from 0x02bf12 to 0x03a73d (0xe82c bytes) 000525 1772176401.766850 read 0x036bb7 thru 0x03a73d (0x3b87 bytes) 000526 1772176401.775270 write 0x001c54 thru 0x009a1e (0x7dcb bytes) 000527 1772176401.776373 punch from 0x0345f4 to 0x03a73d (0x614a bytes) 000528 1772176401.784789 fallocate 0x005e75 thru 0x013cc1 (0xde4d bytes) 000529 1772176401.791468 mapwrite 0x01edbf thru 0x02177e (0x29c0 bytes) 000531 1772176401.808425 punch from 0x02de14 to 0x02f137 (0x1324 bytes) 000532 1772176401.814961 mapwrite 0x005be3 thru 0x00abfc (0x501a bytes) 000533 1772176401.833182 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000535 1772176401.981121 punch from 0x0132cc to 0x014e77 (0x1bac bytes) 000537 1772176401.986254 fallocate 0x024fc5 thru 0x031f2d (0xcf69 bytes) 000539 1772176401.991300 trunc from 0x03a73e to 0x00dc2c (0x2cb11 bytes) 000540 1772176402.005645 trunc from 0x00dc2d to 0x013dcc (0x61a0 bytes) 000541 1772176402.017328 trunc from 0x013dcd to 0x00cd56 (0x7076 bytes) 000542 1772176402.031800 fallocate 0x03dac5 thru 0x03ffff (0x253b bytes) 000543 1772176402.037003 fallocate 0x00481a thru 0x0066d5 (0x1ebc bytes) 000544 1772176402.042136 mapread 0x005d61 thru 0x009e4c (0x40ec bytes) 000545 1772176402.058629 punch from 0x00b0cd to 0x00bb86 (0xaba bytes) 000546 1772176402.063185 trunc from 0x002012 to 0x005d1f (0x3d0e bytes) 000547 1772176402.067970 trunc from 0x00cd57 to 0x031bbb (0x24e65 bytes) 000549 1772176402.082901 mapwrite 0x01fa38 thru 0x02286c (0x2e35 bytes) 000550 1772176402.108740 trunc from 0x031bbc to 0x03121c (0x99f bytes) 000552 1772176402.125116 mapwrite 0x026001 thru 0x033199 (0xd199 bytes) 000553 1772176402.184964 trunc from 0x03319a to 0x03ebac (0xba13 bytes) 000554 1772176402.196121 read 0x03e9b3 thru 0x03ebac (0x1fa bytes) 000555 1772176402.199335 mapwrite 0x0042e2 thru 0x0108d6 (0xc5f5 bytes) 000557 1772176402.225005 trunc from 0x03ebad to 0x03d958 (0x1254 bytes) 000558 1772176402.234806 read 0x003db3 thru 0x008574 (0x47c2 bytes) 000559 1772176402.235388 trunc from 0x03d959 to 0x01ca1e (0x20f3a bytes) 000560 1772176402.248764 trunc from 0x015d26 to 0x017d36 (0x2011 bytes) 000561 1772176402.252784 punch from 0x00581a to 0x014062 (0xe849 bytes) 000562 1772176402.256378 mapread 0x00a630 thru 0x00da06 (0x33d7 bytes) 000563 1772176402.262343 mapwrite 0x00dede thru 0x0120b0 (0x41d3 bytes) 000564 1772176402.276394 trunc from 0x01b7f0 to 0x01ca1e (0x122f bytes) 000565 1772176402.279797 trunc from 0x01ca1f to 0x019d54 (0x2cca bytes) 000567 1772176402.294583 mapwrite 0x0126ce thru 0x0155c8 (0x2efb bytes) 000568 1772176402.309437 MIRROR_EXTEND to 2 mirrors 000569 1772176402.822167 trunc from 0x00ccd3 to 0x019a2c (0xcd5a bytes) 000570 1772176402.848456 mapwrite 0x02a4ad thru 0x0392c8 (0xee1c bytes) 000571 1772176402.928486 write 0x03f7f9 thru 0x03ffff (0x807 bytes) 000572 1772176402.930446 trunc from 0x026851 to 0x02e011 (0x77c1 bytes) 000573 1772176402.936384 mapread 0x033654 thru 0x03afca (0x7977 bytes) 000574 1772176402.944641 trunc from 0x040000 to 0x031f29 (0xe0d6 bytes) 000576 1772176402.964220 MIRROR_RESYNC 2 mirrors 000577 1772176403.512756 trunc from 0x030c2f to 0x031f29 (0x12fb bytes) 000578 1772176403.546337 mapwrite 0x00029a thru 0x00a446 (0xa1ad bytes) 000579 1772176403.590456 trunc from 0x007e7e to 0x00b6cf (0x3852 bytes) 000580 1772176403.595285 write 0x003d0e thru 0x00ce98 (0x918b bytes) 000581 1772176403.600500 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31f2a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f2a): Mirror 2: 0x7c08da04 (stale) CRC-32 checksum value for chunk [0, 0x31f2a): Mirror 1: 0x67e3ba8f Mirror 3: 0x67e3ba8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f2a: skipped 000582 1772176404.192874 mapread 0x031610 thru 0x031f29 (0x91a bytes) 000583 1772176404.198947 trunc from 0x031f2a to 0x039fa7 (0x807e bytes) 000584 1772176404.262719 trunc from 0x039fa8 to 0x00ecc7 (0x2b2e0 bytes) 000586 1772176404.281668 mapread 0x00e4bb thru 0x00ecc7 (0x80d bytes) 000587 1772176404.286872 fallocate 0x0092b8 thru 0x010d80 (0x7ac9 bytes) 000588 1772176404.294764 trunc from 0x010d81 to 0x01a2b4 (0x9534 bytes) 000589 1772176404.308377 mapwrite 0x017e55 thru 0x01e5fd (0x67a9 bytes) 000590 1772176404.370712 mapwrite 0x01b29e thru 0x0234e0 (0x8243 bytes) 000591 1772176404.406070 read 0x02033a thru 0x0234e0 (0x31a7 bytes) 000592 1772176404.406784 trunc from 0x0234e1 to 0x038fe3 (0x15b03 bytes) 000593 1772176404.426856 trunc from 0x038fe4 to 0x005a29 (0x335ba bytes) 000594 1772176404.439218 write 0x00dc4e thru 0x01cffe (0xf3b1 bytes) 000595 1772176404.440819 fallocate 0x003378 thru 0x012d2f (0xf9b8 bytes) 000596 1772176404.446659 write 0x024723 thru 0x02796f (0x324d bytes) 000597 1772176404.447610 trunc from 0x027970 to 0x037715 (0xfda6 bytes) 000598 1772176404.471977 write 0x017826 thru 0x017c90 (0x46b bytes) 000599 1772176404.474045 trunc from 0x00313a to 0x00e2ba (0xb181 bytes) 000600 1772176404.491388 trunc from 0x027289 to 0x031a15 (0xa78d bytes) 000601 1772176404.497453 fallocate 0x01d834 thru 0x01e3ba (0xb87 bytes) 000602 1772176404.502417 fallocate 0x0258ec thru 0x033b46 (0xe25b bytes) 000603 1772176404.506697 read 0x029634 thru 0x02a930 (0x12fd bytes) 000604 1772176404.518672 mapwrite 0x037cde thru 0x03ffff (0x8322 bytes) 000605 1772176404.562634 trunc from 0x020d56 to 0x02ae69 (0xa114 bytes) 000606 1772176404.568874 trunc from 0x0201e9 to 0x02a034 (0x9e4c bytes) 000607 1772176404.578360 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8fb0e454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8fb0e454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1772176404.861774 write 0x03a7af thru 0x03ffff (0x5851 bytes) 000610 1772176404.874369 fallocate 0x00d6a5 thru 0x00f237 (0x1b93 bytes) 000611 1772176404.882887 write 0x02f012 thru 0x037da3 (0x8d92 bytes) 000612 1772176404.896731 mapread 0x02f057 thru 0x03a160 (0xb10a bytes) 000613 1772176404.908073 fallocate 0x012d93 thru 0x017f63 (0x51d1 bytes) 000614 1772176404.920045 trunc from 0x02f3e3 to 0x03103e (0x1c5c bytes) 000615 1772176404.942828 mapwrite 0x02a015 thru 0x0343bc (0xa3a8 bytes) 000616 1772176404.993059 trunc from 0x040000 to 0x013b8a (0x2c475 bytes) 000617 1772176405.008679 read 0x00b1da thru 0x013b8a (0x89b1 bytes) 000618 1772176405.023076 mapwrite 0x031d35 thru 0x037c32 (0x5efe bytes) 000619 1772176405.077158 trunc from 0x037c33 to 0x014ad7 (0x2315b bytes) 000620 1772176405.094910 mapread 0x0125a9 thru 0x014ad7 (0x252f bytes) 000621 1772176405.100993 fallocate 0x01c203 thru 0x01c908 (0x706 bytes) 000622 1772176405.106388 read 0x00d4e0 thru 0x00ea8e (0x15af bytes) 000623 1772176405.107757 fallocate 0x0397f2 thru 0x03ffff (0x680e bytes) 000624 1772176405.111742 write 0x01994c thru 0x01f40b (0x5ac0 bytes) 000625 1772176405.130660 read 0x01d050 thru 0x02847f (0xb430 bytes) 000626 1772176405.132822 mapwrite 0x01668e thru 0x023495 (0xce08 bytes) 000627 1772176405.153971 fallocate 0x03f073 thru 0x03ffff (0xf8d bytes) 000628 1772176405.159632 mapwrite 0x000bdd thru 0x00afce (0xa3f2 bytes) 000629 1772176405.181295 punch from 0x03a8de to 0x03c9d8 (0x20fb bytes) 000630 1772176405.188154 trunc from 0x040000 to 0x02f614 (0x109eb bytes) 000631 1772176405.202752 write 0x01707c thru 0x01f1f4 (0x8179 bytes) 000632 1772176405.205035 fallocate 0x03c96a thru 0x03ffff (0x3696 bytes) 000633 1772176405.210020 mapwrite 0x00fb28 thru 0x01deb4 (0xe38d bytes) 000634 1772176405.231495 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000635 1772176405.343904 fallocate 0x0097f3 thru 0x01173f (0x7f4d bytes) 000636 1772176405.352988 punch from 0x01cc3a to 0x028b43 (0xbf0a bytes) 000637 1772176405.357607 trunc from 0x02f615 to 0x00df9a (0x2167a bytes) 000640 1772176405.380347 punch from 0x009c03 to 0x00bcaa (0x20a8 bytes) 000641 1772176405.385489 write 0x019fa7 thru 0x0233e7 (0x9441 bytes) 000642 1772176405.387418 fallocate 0x02aeef thru 0x03ab36 (0xfc48 bytes) 000644 1772176405.392725 mapread 0x02cb83 thru 0x03a8c1 (0xdd3f bytes) 000645 1772176405.438656 read 0x038d37 thru 0x03ab36 (0x1e00 bytes) 000646 1772176405.439354 trunc from 0x03ab37 to 0x03a98e (0x1a8 bytes) 000648 1772176405.456710 mapread 0x035dff thru 0x035f37 (0x139 bytes) 000649 1772176405.457642 trunc from 0x03a98f to 0x008dc0 (0x31bce bytes) 000651 1772176405.480869 trunc from 0x008dc1 to 0x007ae1 (0x12df bytes) 000653 1772176405.499268 read 0x0063b3 thru 0x007ae1 (0x172f bytes) 000655 1772176405.516367 punch from 0x004c5a to 0x007ae1 (0x2e88 bytes) 000656 1772176405.526016 punch from 0x001e4e to 0x007ae1 (0x5c94 bytes) 000657 1772176405.536999 mapwrite 0x0273ad thru 0x02879d (0x13f1 bytes) 000658 1772176405.592830 fallocate 0x010e35 thru 0x01998d (0x8b59 bytes) 000660 1772176405.597186 fallocate 0x01a3a6 thru 0x01cb22 (0x277d bytes) 000661 1772176405.603063 read 0x016a9b thru 0x025453 (0xe9b9 bytes) 000662 1772176405.603714 mapwrite 0x007bdd thru 0x012b2b (0xaf4f bytes) 000663 1772176405.644377 trunc from 0x01732e to 0x01cf5c (0x5c2f bytes) 000664 1772176405.649437 read 0x0225e5 thru 0x02879d (0x61b9 bytes) 000665 1772176405.654202 MIRROR_EXTEND to 2 mirrors 000666 1772176406.224454 write 0x02bfd7 thru 0x0391c4 (0xd1ee bytes) 000667 1772176406.252359 trunc from 0x01334e to 0x022a10 (0xf6c3 bytes) 000668 1772176406.255790 punch from 0x027119 to 0x02c75d (0x5645 bytes) 000669 1772176406.267787 mapread 0x015bf9 thru 0x01f244 (0x964c bytes) 000670 1772176406.285636 mapread 0x010982 thru 0x01b9b5 (0xb034 bytes) 000671 1772176406.286637 mapread 0x01bde6 thru 0x02b3ea (0xf605 bytes) 000672 1772176406.301433 trunc from 0x015b5a to 0x017259 (0x1700 bytes) 000673 1772176406.305402 punch from 0x0149e2 to 0x01c7d9 (0x7df8 bytes) 000674 1772176406.309287 punch from 0x02e58f to 0x036a45 (0x84b7 bytes) 000675 1772176406.313960 punch from 0x00206d to 0x00fcf9 (0xdc8d bytes) 000676 1772176406.324571 fallocate 0x0178d0 thru 0x01bbcc (0x42fd bytes) 000678 1772176406.336325 read 0x035498 thru 0x0391c4 (0x3d2d bytes) 000681 1772176406.344396 write 0x03b281 thru 0x03ffff (0x4d7f bytes) 000682 1772176406.346009 write 0x024a3b thru 0x02dccb (0x9291 bytes) 000683 1772176406.348700 mapwrite 0x01df4a thru 0x022b81 (0x4c38 bytes) 000684 1772176406.366352 write 0x01f4ad thru 0x0222e7 (0x2e3b bytes) 000685 1772176406.367488 write 0x022f05 thru 0x0275b5 (0x46b1 bytes) 000686 1772176406.369815 mapread 0x01cac5 thru 0x024245 (0x7781 bytes) 000688 1772176406.385344 fallocate 0x01e38a thru 0x02a26d (0xbee4 bytes) 000689 1772176406.390656 mapread 0x02f278 thru 0x038501 (0x928a bytes) 000690 1772176406.406302 read 0x027e43 thru 0x034c13 (0xcdd1 bytes) 000691 1772176406.409238 mapwrite 0x015779 thru 0x01a980 (0x5208 bytes) 000692 1772176406.427969 punch from 0x01c55e to 0x01f9a1 (0x3444 bytes) 000693 1772176406.432879 write 0x01745f thru 0x0219cd (0xa56f bytes) 000694 1772176406.443067 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000695 1772176406.572140 trunc from 0x040000 to 0x03f495 (0xb6a bytes) 000696 1772176406.617358 read 0x03bf23 thru 0x03ef36 (0x3014 bytes) 000697 1772176406.630859 MIRROR_EXTEND to 2 mirrors 000698 1772176407.284418 mapwrite 0x012ffb thru 0x021c40 (0xec46 bytes) 000699 1772176407.365297 write 0x022949 thru 0x0271ec (0x48a4 bytes) 000701 1772176407.375282 trunc from 0x01d7ae to 0x01ec00 (0x1453 bytes) 000702 1772176407.379496 mapread 0x021626 thru 0x0312b5 (0xfc90 bytes) 000703 1772176407.392151 punch from 0x0025fd to 0x00cbc2 (0xa5c6 bytes) 000704 1772176407.395327 mapwrite 0x010b32 thru 0x020528 (0xf9f7 bytes) 000705 1772176407.420992 trunc from 0x03f496 to 0x01299c (0x2caf9 bytes) 000706 1772176407.443598 fallocate 0x031672 thru 0x039fa9 (0x8938 bytes) 000708 1772176407.449373 trunc from 0x039faa to 0x030992 (0x9617 bytes) 000710 1772176407.462423 trunc from 0x001a6a to 0x002863 (0xdfa bytes) 000711 1772176407.466260 punch from 0x020e7f to 0x023b0b (0x2c8d bytes) 000712 1772176407.469364 read 0x018736 thru 0x0207a2 (0x806d bytes) 000713 1772176407.477702 mapwrite 0x00088e thru 0x0062de (0x5a51 bytes) 000714 1772176407.524913 punch from 0x00d58d to 0x013795 (0x6209 bytes) 000716 1772176407.539274 write 0x01c8ce thru 0x027b33 (0xb266 bytes) 000717 1772176407.550654 trunc from 0x030993 to 0x009ad4 (0x26ebe bytes) 000718 1772176407.576106 read 0x00148d thru 0x00975d (0x82d1 bytes) 000719 1772176407.577134 write 0x00c509 thru 0x0193c1 (0xceb9 bytes) 000720 1772176407.579292 read 0x0009a9 thru 0x00ca31 (0xc089 bytes) 000721 1772176407.595321 fallocate 0x00e47c thru 0x0111e1 (0x2d66 bytes) 000722 1772176407.600178 mapread 0x003ecb thru 0x00e90e (0xaa44 bytes) 000724 1772176407.608908 trunc from 0x006a53 to 0x00d326 (0x68d4 bytes) 000726 1772176407.632942 write 0x036a05 thru 0x03e05e (0x765a bytes) 000727 1772176407.636112 mapread 0x02d36a thru 0x03a7ac (0xd443 bytes) 000728 1772176407.650132 mapread 0x00d8f5 thru 0x013692 (0x5d9e bytes) 000729 1772176407.657688 trunc from 0x03e05f to 0x01191a (0x2c744 bytes) 000730 1772176407.674417 trunc from 0x00f5e5 to 0x01191a (0x2336 bytes) 000731 1772176407.684641 MIRROR_RESYNC 2 mirrors 000732 1772176408.214125 mapwrite 0x01fb11 thru 0x020601 (0xaf1 bytes) 000733 1772176408.310453 mapread 0x019ec8 thru 0x020601 (0x673a bytes) 000734 1772176408.323376 mapread 0x01a1e5 thru 0x01b2d7 (0x10f3 bytes) 000735 1772176408.326447 fallocate 0x008054 thru 0x017e19 (0xfdc6 bytes) 000736 1772176408.333576 read 0x006bfe thru 0x015a34 (0xee37 bytes) 000737 1772176408.353870 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20602): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20602): Mirror 2: 0x454dbee3 (stale) CRC-32 checksum value for chunk [0, 0x20602): Mirror 1: 0x724445ab Mirror 3: 0x724445ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20602: skipped 000738 1772176408.995697 read 0x00c782 thru 0x00ee3f (0x26be bytes) 000739 1772176409.003844 trunc from 0x00cd88 to 0x01c870 (0xfae9 bytes) 000741 1772176409.050934 trunc from 0x0151aa to 0x01e4a7 (0x92fe bytes) 000742 1772176409.057657 punch from 0x00b7ba to 0x00bc12 (0x459 bytes) 000743 1772176409.066097 fallocate 0x00ccbc thru 0x01923c (0xc581 bytes) 000744 1772176409.070344 trunc from 0x01fbf6 to 0x020601 (0xa0c bytes) 000745 1772176409.077901 mapread 0x012b42 thru 0x01cd53 (0xa212 bytes) 000746 1772176409.094768 trunc from 0x020602 to 0x00fff3 (0x1060e bytes) 000747 1772176409.126355 mapread 0x00fc4d thru 0x00fff3 (0x3a7 bytes) 000748 1772176409.127392 punch from 0x002c33 to 0x00a36c (0x773a bytes) 000749 1772176409.132306 trunc from 0x00cdf0 to 0x00fff3 (0x3204 bytes) 000750 1772176409.136405 mapwrite 0x019e7a thru 0x01e85e (0x49e5 bytes) 000751 1772176409.184775 read 0x0070c5 thru 0x0082c4 (0x1200 bytes) 000752 1772176409.193711 mapwrite 0x030574 thru 0x03e68b (0xe118 bytes) 000753 1772176409.245279 mapwrite 0x022943 thru 0x026467 (0x3b25 bytes) 000754 1772176409.264681 mapwrite 0x00e63f thru 0x015cd4 (0x7696 bytes) 000755 1772176409.306556 mapread 0x01bd27 thru 0x021253 (0x552d bytes) 000756 1772176409.307765 read 0x038668 thru 0x03a351 (0x1cea bytes) 000757 1772176409.313339 mapread 0x01358a thru 0x0220f0 (0xeb67 bytes) 000758 1772176409.320333 read 0x030200 thru 0x034ebd (0x4cbe bytes) 000759 1772176409.324034 write 0x00fe3f thru 0x0137a3 (0x3965 bytes) 000760 1772176409.325384 read 0x026c67 thru 0x02eddd (0x8177 bytes) 000761 1772176409.332872 trunc from 0x0206ea to 0x026cfe (0x6615 bytes) 000762 1772176409.342040 read 0x038003 thru 0x03c0d6 (0x40d4 bytes) 000763 1772176409.344578 trunc from 0x03e68c to 0x019f02 (0x24789 bytes) 000764 1772176409.380766 mapwrite 0x02c983 thru 0x035c77 (0x92f5 bytes) 000765 1772176409.449659 trunc from 0x031109 to 0x035c77 (0x4b6f bytes) 000766 1772176409.471237 punch from 0x03251c to 0x033c14 (0x16f9 bytes) 000767 1772176409.484256 mapwrite 0x02a780 thru 0x02ead5 (0x4356 bytes) 000769 1772176409.506233 fallocate 0x0277f1 thru 0x03498d (0xd19d bytes) 000770 1772176409.517493 mapread 0x000364 thru 0x00ce1a (0xcab7 bytes) 000771 1772176409.518793 mapwrite 0x012e9c thru 0x021159 (0xe2be bytes) 000772 1772176409.560326 fallocate 0x031c15 thru 0x036c3c (0x5028 bytes) 000773 1772176409.565184 trunc from 0x035c78 to 0x014f12 (0x20d65 bytes) 000774 1772176409.581107 mapwrite 0x013d56 thru 0x0210f7 (0xd3a2 bytes) 000775 1772176409.742548 trunc from 0x0210f8 to 0x02dcbc (0xcbc5 bytes) 000776 1772176409.759398 mapread 0x003b3f thru 0x008f30 (0x53f2 bytes) 000777 1772176409.762597 punch from 0x006aa9 to 0x00a1e2 (0x373a bytes) 000778 1772176409.779630 MIRROR_RESYNC 3 mirrors 000779 1772176410.471282 mapread 0x01c47e thru 0x02a3de (0xdf61 bytes) 000780 1772176410.494146 read 0x029ab9 thru 0x02dcbc (0x4204 bytes) 000781 1772176410.494647 mapread 0x0234f5 thru 0x02dcbc (0xa7c8 bytes) 000782 1772176410.495311 write 0x002d14 thru 0x00d0de (0xa3cb bytes) 000783 1772176410.541549 mapwrite 0x015dba thru 0x018b26 (0x2d6d bytes) 000784 1772176410.574573 mapread 0x023b5c thru 0x0245a3 (0xa48 bytes) 000785 1772176410.583432 read 0x00f60e thru 0x01912b (0x9b1e bytes) 000787 1772176410.585532 fallocate 0x03a0a5 thru 0x03ffff (0x5f5b bytes) 000788 1772176410.590390 trunc from 0x0274dc to 0x02dcbc (0x67e1 bytes) 000789 1772176410.597384 write 0x024d2d thru 0x026962 (0x1c36 bytes) 000790 1772176410.600407 fallocate 0x0170b1 thru 0x01c809 (0x5759 bytes) 000791 1772176410.608591 fallocate 0x01f9d7 thru 0x023c69 (0x4293 bytes) 000792 1772176410.620039 write 0x01e1cc thru 0x01f6a7 (0x14dc bytes) 000793 1772176410.621164 read 0x01c9cc thru 0x029f52 (0xd587 bytes) 000794 1772176410.633383 punch from 0x00be43 to 0x00d760 (0x191e bytes) 000795 1772176410.646157 punch from 0x013a71 to 0x0178e5 (0x3e75 bytes) 000796 1772176410.653442 read 0x0026e4 thru 0x00d196 (0xaab3 bytes) 000797 1772176410.664099 mapread 0x02c137 thru 0x02dcbc (0x1b86 bytes) 000798 1772176410.664900 trunc from 0x00e007 to 0x00fd57 (0x1d51 bytes) 000799 1772176410.670761 mapread 0x006582 thru 0x00ae29 (0x48a8 bytes) 000800 1772176410.672229 trunc from 0x004661 to 0x01330a (0xecaa bytes) 000801 1772176410.676692 mapwrite 0x03ce36 thru 0x03ffff (0x31ca bytes) 000802 1772176410.718853 fallocate 0x018c3f thru 0x02686d (0xdc2f bytes) 000803 1772176410.723947 write 0x032925 thru 0x039ade (0x71ba bytes) 000804 1772176410.732010 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x99ce876b (stale) Mirror 3: 0x99ce876b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a952055 Mirror 4: 0x8a952055 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000805 1772176411.494096 trunc from 0x040000 to 0x02ce62 (0x1319d bytes) 000806 1772176411.536316 mapread 0x003a0a thru 0x00c987 (0x8f7e bytes) 000807 1772176411.555434 punch from 0x01b879 to 0x01f01d (0x37a5 bytes) 000808 1772176411.561720 trunc from 0x00df59 to 0x01041b (0x24c3 bytes) 000809 1772176411.568952 trunc from 0x0285f9 to 0x02ce62 (0x486a bytes) 000810 1772176411.572760 fallocate 0x009380 thru 0x017f8d (0xec0e bytes) 000811 1772176411.577340 read 0x001611 thru 0x00e441 (0xce31 bytes) 000812 1772176411.582315 read 0x003552 thru 0x0053d3 (0x1e82 bytes) 000813 1772176411.582998 trunc from 0x01514d to 0x01b072 (0x5f26 bytes) 000814 1772176411.587957 punch from 0x01a86f to 0x01bb4f (0x12e1 bytes) 000815 1772176411.592648 mapwrite 0x011c5c thru 0x01724d (0x55f2 bytes) 000816 1772176411.620047 mapwrite 0x000a2c thru 0x0100c6 (0xf69b bytes) 000817 1772176411.640257 mapread 0x00f099 thru 0x01b97a (0xc8e2 bytes) 000818 1772176411.641197 write 0x006e20 thru 0x013e48 (0xd029 bytes) 000819 1772176411.643173 trunc from 0x02ce63 to 0x023e7f (0x8fe3 bytes) 000820 1772176411.659876 read 0x0154d3 thru 0x01769f (0x21cd bytes) 000821 1772176411.660712 punch from 0x00d8a0 to 0x013c5a (0x63bb bytes) 000823 1772176411.682456 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23e80): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23e80): Mirror 2: 0xdf7ce635 (stale) Mirror 3: 0xdf7ce635 (stale) Mirror 4: 0x8a093d26 (stale) CRC-32 checksum value for chunk [0, 0x23e80): Mirror 1: 0x4dc4f1e1 Mirror 5: 0x4dc4f1e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23e80: skipped 000824 1772176412.468384 write 0x00f2e3 thru 0x01778d (0x84ab bytes) 000825 1772176412.514109 write 0x030c52 thru 0x037e2b (0x71da bytes) 000826 1772176412.517174 trunc from 0x037e2c to 0x01ef0a (0x18f21 bytes) 000827 1772176412.543773 punch from 0x014abf to 0x01e94b (0x9e8d bytes) 000829 1772176412.556848 punch from 0x016c01 to 0x01ef0a (0x830a bytes) 000830 1772176412.561671 trunc from 0x014d56 to 0x01b3a3 (0x664e bytes) 000832 1772176412.569703 punch from 0x00f096 to 0x014303 (0x526e bytes) 000833 1772176412.580197 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ef0b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef0b): Mirror 2: 0xfe51ca (stale) Mirror 3: 0xfe51ca (stale) Mirror 4: 0xef79aeb7 (stale) Mirror 5: 0x14d0e5a6 (stale) CRC-32 checksum value for chunk [0, 0x1ef0b): Mirror 1: 0x85ac4229 Mirror 6: 0x85ac4229 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef0b: skipped 000834 1772176413.451496 mapread 0x013157 thru 0x01ef0a (0xbdb4 bytes) 000836 1772176413.480677 fallocate 0x003106 thru 0x00fcc9 (0xcbc4 bytes) 000837 1772176413.522370 punch from 0x014efc to 0x0171fc (0x2301 bytes) 000838 1772176413.529754 trunc from 0x01ef0b to 0x037bfd (0x18cf3 bytes) 000839 1772176413.558676 mapread 0x02163a thru 0x028983 (0x734a bytes) 000840 1772176413.567180 MIRROR_RESYNC 6 mirrors 000841 1772176414.759669 mapwrite 0x03eb4f thru 0x03ffff (0x14b1 bytes) 000842 1772176414.829247 write 0x02e292 thru 0x037ee0 (0x9c4f bytes) 000843 1772176414.847637 write 0x0195fa thru 0x027156 (0xdb5d bytes) 000844 1772176414.862789 mapread 0x0145f2 thru 0x0213ac (0xcdbb bytes) 000845 1772176414.864726 punch from 0x014907 to 0x01e135 (0x982f bytes) 000847 1772176414.899644 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8dd9df91 (stale) Mirror 3: 0x8dd9df91 (stale) Mirror 4: 0x8dd9df91 (stale) Mirror 5: 0x8dd9df91 (stale) Mirror 6: 0x8dd9df91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8895d9b9 Mirror 7: 0x8895d9b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000848 1772176415.835033 punch from 0x03d7c4 to 0x03ffff (0x283c bytes) 000849 1772176415.887152 write 0x02e042 thru 0x031aeb (0x3aaa bytes) 000850 1772176415.906257 mapread 0x03f888 thru 0x03ffff (0x778 bytes) 000851 1772176415.922199 mapwrite 0x028b68 thru 0x02e49d (0x5936 bytes) 000852 1772176415.965086 punch from 0x0235d5 to 0x026239 (0x2c65 bytes) 000853 1772176415.982251 write 0x03ea66 thru 0x03f790 (0xd2b bytes) 000854 1772176415.986858 read 0x02a46b thru 0x02c4b7 (0x204d bytes) 000855 1772176415.987612 read 0x01c9eb thru 0x023717 (0x6d2d bytes) 000856 1772176416.002359 write 0x02264f thru 0x0235a0 (0xf52 bytes) 000857 1772176416.003762 write 0x01a7b0 thru 0x025308 (0xab59 bytes) 000858 1772176416.008205 mapread 0x02f5c9 thru 0x03b8cf (0xc307 bytes) 000859 1772176416.010280 trunc from 0x040000 to 0x034fe2 (0xb01d bytes) 000860 1772176416.038237 mapread 0x0196d4 thru 0x0284fb (0xee28 bytes) 000861 1772176416.041070 mapread 0x02f5cb thru 0x034fe2 (0x5a18 bytes) 000862 1772176416.042669 punch from 0x00f2c8 to 0x015c88 (0x69c1 bytes) 000864 1772176416.048416 mapread 0x032fb2 thru 0x034fe2 (0x2031 bytes) 000865 1772176416.049649 fallocate 0x006ade thru 0x00d871 (0x6d94 bytes) 000866 1772176416.054706 mapread 0x023c99 thru 0x02821c (0x4584 bytes) 000867 1772176416.056016 fallocate 0x016580 thru 0x022411 (0xbe92 bytes) 000868 1772176416.060180 mapread 0x0182ac thru 0x0232ce (0xb023 bytes) 000869 1772176416.061660 mapread 0x02f084 thru 0x034fe2 (0x5f5f bytes) 000870 1772176416.062705 mapread 0x01851a thru 0x026906 (0xe3ed bytes) 000871 1772176416.063954 punch from 0x02a877 to 0x034fe2 (0xa76c bytes) 000872 1772176416.067739 fallocate 0x007c69 thru 0x01046a (0x8802 bytes) 000873 1772176416.071945 fallocate 0x03234f thru 0x037331 (0x4fe3 bytes) 000874 1772176416.077064 punch from 0x01dfc4 to 0x01f0d1 (0x110e bytes) 000875 1772176416.090328 trunc from 0x034fe3 to 0x02f8d5 (0x570d bytes) 000876 1772176416.104320 write 0x03ed80 thru 0x03ffff (0x1280 bytes) 000877 1772176416.105461 mapread 0x03366b thru 0x0388e0 (0x5276 bytes) 000878 1772176416.130054 punch from 0x021d07 to 0x0245fa (0x28f4 bytes) 000879 1772176416.135504 mapwrite 0x018f62 thru 0x01a640 (0x16df bytes) 000880 1772176416.154394 trunc from 0x0066bd to 0x009765 (0x30a9 bytes) 000881 1772176416.160041 trunc from 0x040000 to 0x02813e (0x17ec1 bytes) 000882 1772176416.172955 trunc from 0x02813f to 0x03befc (0x13dbe bytes) 000883 1772176416.187916 read 0x024769 thru 0x0292bf (0x4b57 bytes) 000884 1772176416.196935 read 0x02d037 thru 0x0367e3 (0x97ad bytes) 000885 1772176416.198185 trunc from 0x03befd to 0x01c78b (0x1f771 bytes) 000887 1772176416.219203 trunc from 0x0020a3 to 0x00bb7f (0x9add bytes) 000889 1772176416.231059 mapread 0x00fb2a thru 0x019623 (0x9afa bytes) 000891 1772176416.233276 mapread 0x00322d thru 0x00fbff (0xc9d3 bytes) 000892 1772176416.248824 punch from 0x00dcb4 to 0x017d40 (0xa08d bytes) 000893 1772176416.267940 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c78c): Mirror 3: 0x477138a1 (stale) Mirror 4: 0x477138a1 (stale) Mirror 5: 0x477138a1 (stale) Mirror 6: 0x477138a1 (stale) Mirror 7: 0xc7bb3c1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c78c): Mirror 3: 0x477138a1 (stale) Mirror 4: 0x477138a1 (stale) Mirror 5: 0x477138a1 (stale) Mirror 6: 0x477138a1 (stale) Mirror 7: 0xc7bb3c1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c78c: skipped 000894 1772176416.669091 write 0x013d0d thru 0x018e12 (0x5106 bytes) 000895 1772176416.680010 trunc from 0x016771 to 0x01c78b (0x601b bytes) 000896 1772176416.689754 mapwrite 0x01821d thru 0x01c145 (0x3f29 bytes) 000897 1772176416.708806 punch from 0x01a7fc to 0x01c78b (0x1f90 bytes) 000898 1772176416.716763 punch from 0x0182ef to 0x0195c7 (0x12d9 bytes) 000899 1772176416.723982 write 0x004549 thru 0x013aa7 (0xf55f bytes) 000900 1772176416.738230 write 0x01f724 thru 0x0220fb (0x29d8 bytes) 000901 1772176416.739738 mapread 0x0175a2 thru 0x01ce35 (0x5894 bytes) 000902 1772176416.750642 read 0x01c090 thru 0x0220fb (0x606c bytes) 000903 1772176416.751560 mapwrite 0x02ea83 thru 0x034dc7 (0x6345 bytes) 000904 1772176416.792908 punch from 0x024697 to 0x024fba (0x924 bytes) 000905 1772176416.797344 fallocate 0x0160bb thru 0x0243d8 (0xe31e bytes) 000906 1772176416.802721 mapwrite 0x01eb08 thru 0x02350b (0x4a04 bytes) 000907 1772176416.826835 trunc from 0x00ea47 to 0x00f1be (0x778 bytes) 000908 1772176416.831995 write 0x0125a9 thru 0x01e434 (0xbe8c bytes) 000909 1772176416.833547 trunc from 0x034dc8 to 0x01379d (0x2162a bytes) 000910 1772176416.850196 fallocate 0x020cf6 thru 0x02a4f5 (0x9800 bytes) 000911 1772176416.854686 mapwrite 0x011bb8 thru 0x01d326 (0xb76f bytes) 000912 1772176416.916985 trunc from 0x01d327 to 0x02e8d1 (0x115ab bytes) 000913 1772176416.945937 trunc from 0x02d6f7 to 0x02e8d1 (0x11db bytes) 000914 1772176416.952109 mapread 0x001f0d thru 0x009940 (0x7a34 bytes) 000915 1772176416.953319 mapread 0x00ab58 thru 0x00e138 (0x35e1 bytes) 000916 1772176416.958175 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e8d2): Mirror 3: 0x1d469f0f (stale) Mirror 4: 0x1d469f0f (stale) Mirror 5: 0x1d469f0f (stale) Mirror 7: 0x7a0ab5e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8d2): Mirror 3: 0x1d469f0f (stale) Mirror 4: 0x1d469f0f (stale) Mirror 5: 0x1d469f0f (stale) Mirror 7: 0x7a0ab5e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8d2: skipped 000917 1772176417.289308 read 0x0046f8 thru 0x0069c9 (0x22d2 bytes) 000918 1772176417.297116 write 0x031fa8 thru 0x03e73f (0xc798 bytes) 000920 1772176417.304717 punch from 0x02e444 to 0x03ab99 (0xc756 bytes) 000921 1772176417.322920 trunc from 0x03e740 to 0x03da5f (0xce0 bytes) 000922 1772176417.348325 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da60): Mirror 4: 0xddb3f003 (stale) Mirror 5: 0xddb3f003 (stale) Mirror 7: 0x32bd912f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da60): Mirror 4: 0xddb3f003 (stale) Mirror 5: 0xddb3f003 (stale) Mirror 7: 0x32bd912f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da60: skipped 000923 1772176417.721024 read 0x0057b1 thru 0x00678b (0xfdb bytes) 000924 1772176417.733145 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3da60): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da60): Mirror 4: 0xddb3f003 (stale) Mirror 5: 0xddb3f003 (stale) Mirror 7: 0x32bd912f (stale) CRC-32 checksum value for chunk [0, 0x3da60): Mirror 1: 0x64cc4b0b Mirror 8: 0x64cc4b0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da60: skipped 000927 1772176418.383668 trunc from 0x002f07 to 0x00e0c4 (0xb1be bytes) 000928 1772176418.415696 write 0x01ce12 thru 0x026654 (0x9843 bytes) 000929 1772176418.427789 punch from 0x026f60 to 0x03255a (0xb5fb bytes) 000930 1772176418.437605 fallocate 0x02343d thru 0x0277fb (0x43bf bytes) 000932 1772176418.442609 mapread 0x034f3c thru 0x03da5f (0x8b24 bytes) 000933 1772176418.456352 trunc from 0x03da60 to 0x0067c2 (0x3729d bytes) 000934 1772176418.473291 trunc from 0x0067c3 to 0x034ae8 (0x2e326 bytes) 000935 1772176418.486113 mapwrite 0x02f412 thru 0x03e0ee (0xecdd bytes) 000936 1772176418.531093 trunc from 0x00ae0e to 0x0101c9 (0x53bc bytes) 000937 1772176418.536306 write 0x03df62 thru 0x03ffff (0x209e bytes) 000939 1772176418.537899 trunc from 0x006ce0 to 0x00e4ac (0x77cd bytes) 000940 1772176418.542339 read 0x01a69e thru 0x0243d5 (0x9d38 bytes) 000942 1772176418.554620 punch from 0x003e5b to 0x0051da (0x1380 bytes) 000943 1772176418.563359 punch from 0x02f191 to 0x035a27 (0x6897 bytes) 000945 1772176418.568847 mapread 0x024716 thru 0x02df25 (0x9810 bytes) 000946 1772176418.573792 fallocate 0x0077d8 thru 0x007b85 (0x3ae bytes) 000947 1772176418.578142 trunc from 0x039443 to 0x03da54 (0x4612 bytes) 000948 1772176418.584948 trunc from 0x040000 to 0x0278c5 (0x1873a bytes) 000949 1772176418.610052 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x278c6): Mirror 4: 0xda9a383 (stale) Mirror 7: 0x52358bc6 (stale) Mirror 8: 0xc1d56ff3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278c6): Mirror 4: 0xda9a383 (stale) Mirror 7: 0x52358bc6 (stale) Mirror 8: 0xc1d56ff3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278c6: skipped 000950 1772176418.850691 read 0x026736 thru 0x0278c5 (0x1190 bytes) 000952 1772176418.855169 punch from 0x020d64 to 0x0278c5 (0x6b62 bytes) 000953 1772176418.858482 punch from 0x01b5b8 to 0x024d21 (0x976a bytes) 000954 1772176418.861699 trunc from 0x0057cf to 0x012362 (0xcb94 bytes) 000955 1772176418.865362 trunc from 0x0278c6 to 0x01983e (0xe087 bytes) 000956 1772176418.880918 trunc from 0x007d33 to 0x00d3cd (0x569b bytes) 000957 1772176418.884735 fallocate 0x021b58 thru 0x025ae2 (0x3f8b bytes) 000958 1772176418.888224 fallocate 0x027311 thru 0x036aa7 (0xf797 bytes) 000959 1772176418.891572 trunc from 0x036aa8 to 0x030062 (0x6a45 bytes) 000960 1772176418.901023 punch from 0x02b59a to 0x02c831 (0x1298 bytes) 000962 1772176418.905349 mapread 0x006524 thru 0x015a8d (0xf56a bytes) 000963 1772176418.921846 read 0x019cca thru 0x029782 (0xfab9 bytes) 000964 1772176418.927904 read 0x027552 thru 0x02bf09 (0x49b8 bytes) 000965 1772176418.928650 trunc from 0x030063 to 0x00703a (0x29028 bytes) 000966 1772176418.942328 MIRROR_RESYNC 4 mirrors 000968 1772176419.559203 trunc from 0x00703b to 0x032776 (0x2b73c bytes) 000969 1772176419.606835 mapwrite 0x034897 thru 0x03766a (0x2dd4 bytes) 000970 1772176419.691420 trunc from 0x020e51 to 0x02e058 (0xd208 bytes) 000971 1772176419.702079 trunc from 0x03766b to 0x03377e (0x3eec bytes) 000972 1772176419.721417 punch from 0x008a98 to 0x014132 (0xb69b bytes) 000974 1772176419.731373 trunc from 0x012722 to 0x019177 (0x6a56 bytes) 000975 1772176419.738440 punch from 0x00817e to 0x01566b (0xd4ee bytes) 000976 1772176419.744928 punch from 0x00598e to 0x015728 (0xfd9b bytes) 000977 1772176419.748220 punch from 0x0065e8 to 0x00761c (0x1035 bytes) 000979 1772176419.751369 punch from 0x02a0b2 to 0x02e4a8 (0x43f7 bytes) 000982 1772176419.763067 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3377f): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3377f): Mirror 4: 0xb058a7d1 (stale) Mirror 7: 0xb058a7d1 (stale) Mirror 8: 0xb058a7d1 (stale) CRC-32 checksum value for chunk [0, 0x3377f): Mirror 1: 0xb058a7d1 Mirror 9: 0xb058a7d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3377f: skipped 000983 1772176420.411438 trunc from 0x027fea to 0x032254 (0xa26b bytes) 000984 1772176420.454413 fallocate 0x0080d8 thru 0x015277 (0xd1a0 bytes) 000986 1772176420.461416 trunc from 0x03377f to 0x02d631 (0x614d bytes) 000987 1772176420.483724 fallocate 0x015d9c thru 0x0218b4 (0xbb19 bytes) 000989 1772176420.492129 punch from 0x02944c to 0x02ba29 (0x25de bytes) 000990 1772176420.497925 fallocate 0x01cab8 thru 0x02bafa (0xf043 bytes) 000991 1772176420.502756 mapwrite 0x00071c thru 0x00327b (0x2b60 bytes) 000992 1772176420.534660 trunc from 0x02d632 to 0x025ff2 (0x763f bytes) 000994 1772176420.554380 punch from 0x00ad6b to 0x0132f2 (0x8588 bytes) 000995 1772176420.567242 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25ff3): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25ff3): Mirror 4: 0xf9162d8f (stale) Mirror 7: 0xf9162d8f (stale) Mirror 8: 0xf9162d8f (stale) Mirror 9: 0xf9162d8f (stale) CRC-32 checksum value for chunk [0, 0x25ff3): Mirror 1: 0x40813bc7 Mirror 10: 0x40813bc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25ff3: skipped 000996 1772176421.428890 MIRROR_RESYNC 6 mirrors 000998 1772176422.304371 read 0x0237f1 thru 0x025ff2 (0x2802 bytes) 000999 1772176422.317391 mapwrite 0x00ae54 thru 0x013949 (0x8af6 bytes) 001000 1772176422.432976 fallocate 0x0000ec thru 0x0099f0 (0x9905 bytes) All operations completed A-OK! PASS 70a (53s) == sanity-flr test 71: check mirror extend parent fid ==== 02:13:45 (1772176425) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x2a7:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 02:13:52 (1772176432) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156493 s, 13.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106457 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 02:13:58 (1772176438) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4264992 .. Extending file size to 3768928 .. Extending file size to 7669600 .. Extending file size to 8347232 .. Extending file size to 8431744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8458336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8813856 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9008576 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9124288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9120288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 60283 60284 60285 60288 60289 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:15:08 (1772176508) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5171584 .. Extending file size to 6957536 .. failed Extending file size to 8610816 .. Extending file size to 8153312 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8863392 .. Extending file size to 8633600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8752000 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9087904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8921152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62822 62823 62824 62827 62829 failed 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:16:17 (1772176577) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1268796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24624 3582396 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17736 3589284 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42360 7171680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1268796 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24624 3582396 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17736 3589284 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42360 7171680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0695686 s, 30.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17102 s, 12.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3dcdcf29 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:16:29 (1772176589) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521422 s, 40.2 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 02:16:34 (1772176594) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:16:38 (1772176598) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0595901 s, 35.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x367:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x367:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x368:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:16:42 (1772176602) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0503108 s, 41.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:16:47 (1772176607) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0645495 s, 32.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:17:14 (1772176634) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064394 s, 16.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:17:58 (1772176678) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0498836 s, 42.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x368:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x368:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:18:43 (1772176723) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg323-server: Pool lustre.test_204f created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.052679 s, 39.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg323-server: Pool lustre.test_204f destroyed PASS 204f (54s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:19:37 (1772176777) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:19:41 (1772176781) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:19:45 (1772176785) oleg323-server: Pool lustre.test_206 created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg323-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:20:02 (1772176802) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405573 s, 25.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179545 s, 58.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4666bc63 Mirror 3: 0x4666bc63 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:20:07 (1772176807) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:20:10 (1772176810) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:20:12 (1772176812) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.030017 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd80000 Read this (4 bytes): BBBB 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:20:34 (1772176834) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0100277 s, 817 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:20:38 (1772176838) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292637 s, 35.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x92 lmm_fid: [0x200000405:0x92:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x92 lmm_fid: [0x200000405:0x92:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611282 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-27 02:20:39.000000000 -0500 Modify: 2026-02-27 02:20:39.000000000 -0500 Change: 2026-02-27 02:20:40.000000000 -0500 Birth: 2026-02-27 02:20:39.000000000 -0500 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:20:42 (1772176842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221889 s, 47.3 MB/s oleg323-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg323-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg323-server: O/280000401/d18/d2: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:21:00 (1772176860) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.134201 s, 305 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0115923 s, 353 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 27 02:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x97:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 27 02:21 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:21:04 (1772176864) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:21:08 (1772176868) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1359 sec =========== 02:21:13 (1772176873) === sanity-flr: start cleanup 02:21:14 (1772176874) === === sanity-flr: finish cleanup 02:21:18 (1772176878) ===