-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 10:03:21 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg447-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:03:42 (1789049022) === oleg447-client.virtnet: executing check_config_client /mnt/lustre oleg447-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg447-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91a15186f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91a15186f000.idle_timeout=debug disable quota as required oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:04:22 (1789049062) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8237] == sanity-flr test 0a: lfs mirror create with -N option == 10:04:24 (1789049064) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:04:43 (1789049083) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:04:46 (1789049086) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:04:49 (1789049089) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:05:16 (1789049116) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:05:19 (1789049119) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:05:23 (1789049123) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91a15186f000.stats=clear osc.lustre-OST0001-osc-ffff91a15186f000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:05:31 (1789049131) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:05:43 (1789049143) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:05:52 (1789049152) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:05:55 (1789049155) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:06:03 (1789049163) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 10:06:13 (1789049173) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:06:20 (1789049180) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:06:27 (1789049187) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:06:35 (1789049195) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:06:48 (1789049208) PASS 7 (10s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:06:58 (1789049218) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.13279 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194498 s, 5.4 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:07:12 (1789049232) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.412865 s, 45.7 MB/s PASS 22 (13s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:07:24 (1789049244) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:07:37 (1789049257) Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:08:58 (1789049338) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.406116 s, 15.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:09:42 (1789049382) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:11:08 (1789049468) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.39482 s, 15.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 10:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.169s user 0m0.001s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.045s user 0m0.001s sys 0m0.192s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.373s user 0m0.002s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a15186f000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:12:48 (1789049568) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:12:52 (1789049572) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:12:56 (1789049576) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:13:00 (1789049580) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 10:13:07 (1789049587) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.745319 s, 16.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:13:21 (1789049601) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329207 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030105 s, 34.8 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:13:28 (1789049608) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:13:38 (1789049618) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107897 s, 949 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130059 s, 787 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012091 s, 847 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00308102 s, 332 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 10:13:50 (1789049630) create test files with size 15746902 .. d5ea166460afcaabd5cdb7a8297a1237 - 0b7d4f2e0017c91a9bc9d3814461af32 - b28f835c8475714ddaccd658173de895 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 10:14:14 (1789049654) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:14:27 (1789049667) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:14:34 (1789049674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731327 s, 14.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326249 s, 32.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 10:14:45 (1789049685) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120199 s, 17.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.315451 s, 13.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157467 s, 13.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 10:15:01 (1789049701) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (6s) == sanity-flr test 43a: mirror pick on write ============= 10:15:06 (1789049706) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:15:11 (1789049711) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602795 s, 17.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 10:15:20 (1789049720) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.533571 s, 5.9 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:15:32 (1789049732) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:15:42 (1789049742) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.572632 s, 18.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:15:56 (1789049756) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff91a16018d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.428765 s, 24.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff91a1602da000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:16:15 (1789049775) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:17:05 (1789049825) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:17:49 (1789049869) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 10:17:54 (1789049874) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.249226 s, 12.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.175181 s, 18.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x49b3f73a (nosync,stale) Mirror 2: 0x49b3f73a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8bd230b5f519e0470868c2b9be23b0cf - mirror 1 checksum: f5d83c001894b1c21441038586a66b3f - mirror 2 checksum: 8bd230b5f519e0470868c2b9be23b0cf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T10:17:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T10:17:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8bd230b5f519e0470868c2b9be23b0cf - mirror 1 checksum: f5d83c001894b1c21441038586a66b3f - mirror 2 checksum: f5d83c001894b1c21441038586a66b3f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T10:17:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T10:17:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:18:06 (1789049886) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550061 s, 19.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:18:14 (1789049894) Failing mds1 on oleg447-server Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 10:18:26 (1789049906) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server Failover mds1 to oleg447-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 10:19:03 (1789049943) targets are mounted 10:19:03 (1789049943) facet_failover done oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576232 s, 18.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 50A (69s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:19:23 (1789049963) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0629738 s, 325 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228072 s, 898 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (16s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:19:39 (1789049979) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0735129 s, 279 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0129638 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (13s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:19:52 (1789049992) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0189196 s, 866 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:20:00 (1789050000) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0878946 s, 233 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:20:12 (1789050012) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143738 s, 285 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0582176 s, 70.4 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:20:24 (1789050024) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:20:53 (1789050053) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:21:25 (1789050085) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:21:51 (1789050111) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:22:17 (1789050137) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 10:22:25 (1789050145) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57594 mirror split pid 57596 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57594 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57596 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 10:23:33 (1789050213) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 215 000001 1789050215.472750 write 0x03cb87 thru 0x03ffff (0x3479 bytes) zero_range to largest ever: 0x29daa 000002 1789050215.487772 trunc from 0x022d70 to 0x029da9 (0x703a bytes) 000003 1789050215.499938 read 0x01d45c thru 0x028e15 (0xb9ba bytes) 000004 1789050215.513056 fallocate 0x00f544 thru 0x012802 (0x32bf bytes) 000005 1789050215.523339 mapread 0x017b30 thru 0x020b36 (0x9007 bytes) 000006 1789050215.555994 punch from 0x039164 to 0x03fd40 (0x6bdd bytes) 000007 1789050215.580046 trunc from 0x040000 to 0x0175db (0x28a24 bytes) 000008 1789050215.603263 punch from 0x015262 to 0x016f4b (0x1cea bytes) fallocating to largest ever: 0x40000 000009 1789050215.612201 fallocate 0x03e0e0 thru 0x03ffff (0x1f20 bytes) 000010 1789050215.619807 trunc from 0x00a090 to 0x012cc3 (0x8c34 bytes) 000011 1789050215.626047 mapread 0x009981 thru 0x00a62b (0xcab bytes) 000013 1789050215.643151 write 0x03bcbd thru 0x03ffff (0x4343 bytes) 000014 1789050215.645367 read 0x032f57 thru 0x03c437 (0x94e1 bytes) 000015 1789050215.687882 trunc from 0x033b07 to 0x035072 (0x156c bytes) 000016 1789050215.707618 punch from 0x0298e2 to 0x02fdb8 (0x64d7 bytes) 000017 1789050215.713731 punch from 0x000ddf to 0x0056ff (0x4921 bytes) 000019 1789050215.721551 mapwrite 0x02ba02 thru 0x02fa6c (0x406b bytes) 000020 1789050215.815115 mapwrite 0x00a8e4 thru 0x01679a (0xbeb7 bytes) 000021 1789050215.875083 fallocate 0x03cb65 thru 0x03ffff (0x349b bytes) 000022 1789050215.896675 write 0x0364a9 thru 0x038575 (0x20cd bytes) 000023 1789050215.901384 read 0x024e7e thru 0x034928 (0xfaab bytes) 000024 1789050215.916267 punch from 0x03335f to 0x036d64 (0x3a06 bytes) 000025 1789050215.936352 fallocate 0x00a92f thru 0x00d46d (0x2b3f bytes) 000027 1789050215.949245 mapread 0x02005a thru 0x027e96 (0x7e3d bytes) 000028 1789050215.950440 read 0x02c711 thru 0x03923d (0xcb2d bytes) 000029 1789050215.958700 trunc from 0x040000 to 0x003433 (0x3cbcc bytes) 000031 1789050215.983006 punch from 0x001c2b to 0x003433 (0x1809 bytes) 000034 1789050215.999672 mapwrite 0x01c902 thru 0x02a13c (0xd83b bytes) 000038 1789050216.190588 read 0x001657 thru 0x0099e7 (0x8391 bytes) 000039 1789050216.208166 trunc from 0x015d56 to 0x02217b (0xc426 bytes) 000040 1789050216.219217 write 0x00927a thru 0x012fae (0x9d35 bytes) 000041 1789050216.226356 mapwrite 0x03bafa thru 0x03ffff (0x4506 bytes) 000042 1789050216.336626 read 0x02c9d2 thru 0x03b33e (0xe96d bytes) 000043 1789050216.337639 fallocate 0x033290 thru 0x037627 (0x4398 bytes) 000044 1789050216.352390 mapwrite 0x01718d thru 0x01df3e (0x6db2 bytes) 000045 1789050216.407269 mapwrite 0x0071a1 thru 0x00eeb6 (0x7d16 bytes) 000047 1789050216.427288 fallocate 0x0036ad thru 0x00ef96 (0xb8ea bytes) 000048 1789050216.433117 mapread 0x025359 thru 0x025ec7 (0xb6f bytes) 000049 1789050216.434276 fallocate 0x005419 thru 0x00b4f9 (0x60e1 bytes) 000050 1789050216.440640 punch from 0x016f1a to 0x0239e1 (0xcac8 bytes) 000054 1789050216.450261 write 0x02e6df thru 0x032baf (0x44d1 bytes) 000055 1789050216.452210 punch from 0x016a09 to 0x01a44c (0x3a44 bytes) 000056 1789050216.457419 mapwrite 0x01dca1 thru 0x02887a (0xabda bytes) 000057 1789050216.571446 mapread 0x01957a thru 0x01e2c2 (0x4d49 bytes) 000059 1789050216.624492 mapwrite 0x02320a thru 0x028250 (0x5047 bytes) 000060 1789050216.675698 mapwrite 0x0221f6 thru 0x02a280 (0x808b bytes) 000061 1789050216.713197 read 0x00dde6 thru 0x010f3e (0x3159 bytes) 000062 1789050216.713999 trunc from 0x040000 to 0x0337e7 (0xc818 bytes) 000063 1789050216.733129 read 0x020413 thru 0x020c9e (0x88c bytes) 000065 1789050216.734408 trunc from 0x01c406 to 0x027b8a (0xb785 bytes) 000066 1789050216.749233 trunc from 0x00e98a to 0x01683e (0x7eb5 bytes) 000067 1789050216.757964 read 0x007ce7 thru 0x00a667 (0x2981 bytes) 000068 1789050216.758691 read 0x02d87d thru 0x02fd3c (0x24c0 bytes) 000069 1789050216.759245 trunc from 0x0337e8 to 0x01b558 (0x1828f bytes) 000070 1789050216.795953 write 0x02add2 thru 0x0323a6 (0x75d5 bytes) 000071 1789050216.799219 fallocate 0x00cff4 thru 0x00f5e8 (0x25f5 bytes) 000072 1789050216.804902 mapread 0x0239f5 thru 0x027282 (0x388e bytes) 000073 1789050216.822109 trunc from 0x0323a7 to 0x006e98 (0x2b50e bytes) 000074 1789050216.863517 trunc from 0x005b66 to 0x006e98 (0x1333 bytes) 000075 1789050216.876872 trunc from 0x006e99 to 0x03dbd0 (0x36d38 bytes) 000076 1789050216.904895 mapread 0x00d381 thru 0x0116a6 (0x4326 bytes) 000078 1789050216.969295 MIRROR_EXTEND to 2 mirrors 000079 1789050218.010755 mapread 0x0181c4 thru 0x022b19 (0xa956 bytes) 000080 1789050218.064500 punch from 0x00d3c2 to 0x011309 (0x3f48 bytes) 000081 1789050218.213125 fallocate 0x012b4d thru 0x0215aa (0xea5e bytes) 000082 1789050218.241099 punch from 0x02fa86 to 0x03a661 (0xabdc bytes) 000083 1789050218.267442 fallocate 0x019519 thru 0x01cc6c (0x3754 bytes) 000085 1789050218.288327 fallocate 0x009daf thru 0x00cffb (0x324d bytes) 000087 1789050218.297544 punch from 0x0147af to 0x01677c (0x1fce bytes) 000088 1789050218.302750 trunc from 0x0009f7 to 0x00fdf4 (0xf3fe bytes) 000089 1789050218.312201 mapread 0x01fa2d thru 0x02aa61 (0xb035 bytes) 000090 1789050218.353350 trunc from 0x000a7b to 0x007e3d (0x73c3 bytes) 000091 1789050218.379591 read 0x0094be thru 0x00fc52 (0x6795 bytes) 000092 1789050218.413377 punch from 0x00c74e to 0x016137 (0x99ea bytes) 000093 1789050218.429298 fallocate 0x03b14c thru 0x03ffff (0x4eb4 bytes) 000094 1789050218.449188 fallocate 0x03c214 thru 0x03ffff (0x3dec bytes) 000095 1789050218.477612 mapread 0x0383f4 thru 0x03ffff (0x7c0c bytes) 000097 1789050218.499332 fallocate 0x001485 thru 0x004641 (0x31bd bytes) 000098 1789050218.520933 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000099 1789050218.816010 mapread 0x011ddf thru 0x014869 (0x2a8b bytes) 000101 1789050218.884849 write 0x01d82c thru 0x02a3aa (0xcb7f bytes) 000102 1789050218.926795 punch from 0x008f96 to 0x013302 (0xa36d bytes) 000103 1789050218.938810 read 0x002eff thru 0x008164 (0x5266 bytes) 000104 1789050218.954400 punch from 0x03a466 to 0x03ffff (0x5b9a bytes) 000105 1789050218.972366 trunc from 0x026537 to 0x028604 (0x20ce bytes) 000106 1789050219.023461 mapread 0x032137 thru 0x03a364 (0x822e bytes) 000108 1789050219.081917 trunc from 0x010bbc to 0x0193ec (0x8831 bytes) 000109 1789050219.105826 trunc from 0x040000 to 0x00708b (0x38f74 bytes) 000110 1789050219.129159 read 0x00639d thru 0x00708b (0xcef bytes) 000111 1789050219.129938 write 0x0074e5 thru 0x01736a (0xfe86 bytes) 000113 1789050219.134207 mapwrite 0x03d4e3 thru 0x03ffff (0x2b1d bytes) 000114 1789050219.242199 mapwrite 0x0255af thru 0x02c10a (0x6b5c bytes) 000116 1789050219.282284 trunc from 0x03c108 to 0x03ffff (0x3ef8 bytes) 000117 1789050219.294607 read 0x01135c thru 0x01c253 (0xaef8 bytes) 000118 1789050219.296115 fallocate 0x014a39 thru 0x0196aa (0x4c72 bytes) 000120 1789050219.304843 mapwrite 0x022e8e thru 0x029e81 (0x6ff4 bytes) 000121 1789050219.341446 mapwrite 0x02840c thru 0x02dcf9 (0x58ee bytes) 000122 1789050219.400061 fallocate 0x01c49f thru 0x02b2ce (0xee30 bytes) 000123 1789050219.416090 punch from 0x00ae97 to 0x016136 (0xb2a0 bytes) 000124 1789050219.429903 mapread 0x016d06 thru 0x01eab8 (0x7db3 bytes) 000126 1789050219.457434 read 0x00b11d thru 0x013863 (0x8747 bytes) 000127 1789050219.458164 punch from 0x00d903 to 0x00e4d8 (0xbd6 bytes) 000128 1789050219.471930 write 0x00a951 thru 0x00e013 (0x36c3 bytes) 000129 1789050219.491415 mapread 0x02698e thru 0x02d581 (0x6bf4 bytes) 000130 1789050219.492620 mapwrite 0x00f160 thru 0x01471f (0x55c0 bytes) 000131 1789050219.550193 punch from 0x037a23 to 0x03ffff (0x85dd bytes) 000132 1789050219.557356 trunc from 0x040000 to 0x00bd86 (0x34279 bytes) 000133 1789050219.584399 trunc from 0x0021c0 to 0x006823 (0x4664 bytes) 000134 1789050219.590456 trunc from 0x00bd87 to 0x01e315 (0x1258f bytes) 000135 1789050219.617125 punch from 0x001d66 to 0x00ac69 (0x8f04 bytes) 000137 1789050219.637521 punch from 0x001a9d to 0x006e2d (0x5391 bytes) 000138 1789050219.662028 mapwrite 0x012189 thru 0x0206da (0xe552 bytes) 000141 1789050219.742784 trunc from 0x012015 to 0x016a5b (0x4a47 bytes) 000142 1789050219.750777 trunc from 0x01faa8 to 0x01ffad (0x506 bytes) 000143 1789050219.762076 mapread 0x01295c thru 0x01a0de (0x7783 bytes) 000144 1789050219.807820 trunc from 0x0206db to 0x02b024 (0xa94a bytes) 000145 1789050219.861277 mapread 0x023ad1 thru 0x02b024 (0x7554 bytes) 000147 1789050219.909037 write 0x030497 thru 0x038186 (0x7cf0 bytes) 000148 1789050219.915231 read 0x00958d thru 0x01703f (0xdab3 bytes) 000149 1789050219.939100 read 0x0089c4 thru 0x017cde (0xf31b bytes) 000150 1789050219.943303 write 0x00eafc thru 0x011ffe (0x3503 bytes) 000152 1789050219.956943 trunc from 0x00982d to 0x00b890 (0x2064 bytes) 000153 1789050219.968217 trunc from 0x038187 to 0x037c5c (0x52a bytes) 000154 1789050220.016484 fallocate 0x0056d5 thru 0x006a8d (0x13b9 bytes) 000155 1789050220.030790 trunc from 0x037c5d to 0x003e8d (0x33dcf bytes) 000158 1789050220.080293 punch from 0x001a82 to 0x003e8d (0x240c bytes) 000159 1789050220.089865 fallocate 0x032517 thru 0x03435a (0x1e44 bytes) 000160 1789050220.104113 read 0x002bbf thru 0x003e8d (0x12cf bytes) 000161 1789050220.129314 trunc from 0x003e8e to 0x0038ef (0x59e bytes) 000163 1789050220.222971 mapread 0x00107f thru 0x0038ef (0x2871 bytes) 000164 1789050220.225691 mapread 0x000331 thru 0x0038ef (0x35bf bytes) 000166 1789050220.232407 mapwrite 0x00e891 thru 0x019bac (0xb31c bytes) 000167 1789050220.334488 read 0x001732 thru 0x007386 (0x5c55 bytes) 000168 1789050220.336361 punch from 0x005544 to 0x01155e (0xc01b bytes) 000169 1789050220.355337 read 0x01930c thru 0x019bac (0x8a1 bytes) 000171 1789050220.356778 mapwrite 0x03df2f thru 0x03ffff (0x20d1 bytes) 000172 1789050220.458787 mapread 0x0154b4 thru 0x01eba0 (0x96ed bytes) 000173 1789050220.459850 mapread 0x00741e thru 0x009510 (0x20f3 bytes) 000174 1789050220.466981 trunc from 0x040000 to 0x02eb12 (0x114ed bytes) 000175 1789050220.505229 fallocate 0x00ec02 thru 0x01b37e (0xc77d bytes) 000176 1789050220.520292 punch from 0x0192e7 to 0x024db2 (0xbacc bytes) 000177 1789050220.533685 read 0x015a09 thru 0x019f26 (0x451e bytes) 000178 1789050220.570415 mapwrite 0x01a457 thru 0x01ef40 (0x4aea bytes) 000179 1789050220.607155 trunc from 0x01a28d to 0x01f5c3 (0x5337 bytes) 000180 1789050220.617168 trunc from 0x02eb13 to 0x01dea0 (0x10c72 bytes) 000181 1789050220.663981 fallocate 0x03f1cd thru 0x03ffff (0xe33 bytes) 000182 1789050220.678164 read 0x005b12 thru 0x010c26 (0xb115 bytes) 000183 1789050220.681200 trunc from 0x040000 to 0x0098cf (0x36730 bytes) 000184 1789050220.719978 mapread 0x0081a8 thru 0x0098cf (0x1728 bytes) 000185 1789050220.721273 trunc from 0x0098d0 to 0x02d1f6 (0x23927 bytes) 000186 1789050220.755197 trunc from 0x02d1f7 to 0x01058f (0x1cc67 bytes) 000187 1789050220.809249 read 0x00ced7 thru 0x01058f (0x36b9 bytes) 000188 1789050220.831181 trunc from 0x010590 to 0x0296f3 (0x19164 bytes) 000189 1789050220.878451 mapwrite 0x020729 thru 0x02e46d (0xdd45 bytes) 000190 1789050220.976330 mapread 0x0092d5 thru 0x0187df (0xf50b bytes) 000192 1789050220.980547 trunc from 0x02e46e to 0x02d03d (0x1430 bytes) 000193 1789050221.003649 mapread 0x029d6a thru 0x02d03d (0x32d4 bytes) 000195 1789050221.005186 write 0x03fca9 thru 0x03ffff (0x357 bytes) 000196 1789050221.009471 trunc from 0x01408e to 0x01e330 (0xa2a3 bytes) 000197 1789050221.022870 trunc from 0x040000 to 0x00d7f4 (0x3280b bytes) 000198 1789050221.075464 write 0x03fea3 thru 0x03ffff (0x15d bytes) 000199 1789050221.082379 read 0x03e6a4 thru 0x03ffff (0x195c bytes) 000200 1789050221.113926 read 0x0031bf thru 0x004a6f (0x18b1 bytes) 000201 1789050221.115005 write 0x03c2b7 thru 0x03ffff (0x3d49 bytes) 000202 1789050221.122327 trunc from 0x00c737 to 0x013187 (0x6a51 bytes) 000203 1789050221.131542 trunc from 0x040000 to 0x021ea4 (0x1e15b bytes) 000204 1789050221.168119 write 0x01574b thru 0x01f008 (0x98be bytes) 000205 1789050221.175022 mapwrite 0x03e1d9 thru 0x03ffff (0x1e27 bytes) 000206 1789050221.296716 MIRROR_EXTEND to 2 mirrors 000207 1789050222.396946 trunc from 0x023713 to 0x032c27 (0xf515 bytes) 000208 1789050222.515239 trunc from 0x040000 to 0x024073 (0x1bf8c bytes) 000209 1789050222.562254 trunc from 0x0122c8 to 0x01a8bb (0x85f4 bytes) 000210 1789050222.567510 trunc from 0x024074 to 0x03d8d3 (0x19860 bytes) 000211 1789050222.597926 read 0x010ab8 thru 0x0208dc (0xfe25 bytes) 000212 1789050222.641205 trunc from 0x03d8d4 to 0x01d45d (0x20476 bytes) 000213 1789050222.685026 trunc from 0x00db86 to 0x0179a9 (0x9e24 bytes) 000214 1789050222.692233 mapwrite 0x0191df thru 0x0213b3 (0x81d5 bytes) 000215 1789050222.850449 trunc from 0x00ea7a to 0x0127d0 (0x3d57 bytes) 000216 1789050222.870874 mapread 0x00a41f thru 0x00f842 (0x5424 bytes) 000217 1789050222.882605 fallocate 0x036469 thru 0x03eaf3 (0x868b bytes) 000218 1789050222.889605 read 0x009b77 thru 0x012d5c (0x91e6 bytes) 000219 1789050222.890234 trunc from 0x0213b4 to 0x0079f0 (0x199c3 bytes) 000220 1789050222.925130 write 0x01a025 thru 0x025d60 (0xbd3c bytes) 000221 1789050222.934357 trunc from 0x025d61 to 0x03221c (0xc4bc bytes) 000222 1789050222.977061 fallocate 0x03e766 thru 0x03ffff (0x189a bytes) 000223 1789050222.983367 write 0x00054b thru 0x004da3 (0x4859 bytes) 000224 1789050222.999181 MIRROR_RESYNC 2 mirrors 000225 1789050223.903308 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000226 1789050224.089951 read 0x00fbcc thru 0x0167ae (0x6be3 bytes) 000227 1789050224.102329 mapread 0x039da2 thru 0x03ffff (0x625e bytes) 000228 1789050224.120132 mapwrite 0x01e107 thru 0x01e721 (0x61b bytes) 000230 1789050224.156374 read 0x022b79 thru 0x02ff7c (0xd404 bytes) 000231 1789050224.175273 punch from 0x020f57 to 0x0289ff (0x7aa9 bytes) 000232 1789050224.180933 trunc from 0x040000 to 0x0060d4 (0x39f2b bytes) 000234 1789050224.204165 mapread 0x0030bf thru 0x0060d4 (0x3016 bytes) 000235 1789050224.206080 punch from 0x002585 to 0x0058bc (0x3338 bytes) 000237 1789050224.216619 mapread 0x0030cf thru 0x0060d4 (0x3006 bytes) 000238 1789050224.244186 mapwrite 0x029214 thru 0x038439 (0xf226 bytes) 000240 1789050224.521872 mapwrite 0x00ca42 thru 0x016f7d (0xa53c bytes) 000241 1789050224.688935 fallocate 0x0228d1 thru 0x02577b (0x2eab bytes) 000242 1789050224.694263 mapread 0x021650 thru 0x030b9b (0xf54c bytes) 000244 1789050224.742089 write 0x0364e5 thru 0x038d56 (0x2872 bytes) 000245 1789050224.744081 punch from 0x031c87 to 0x038d56 (0x70d0 bytes) 000246 1789050224.756905 trunc from 0x038d57 to 0x004c1a (0x3413c bytes) 000247 1789050224.779779 mapread 0x002927 thru 0x004c1a (0x22f4 bytes) 000248 1789050224.786065 mapread 0x002153 thru 0x004c1a (0x2ac8 bytes) 000250 1789050224.793498 MIRROR_EXTEND to 2 mirrors 000251 1789050225.936845 punch from 0x0027ea to 0x004c1a (0x2431 bytes) 000252 1789050225.977556 mapwrite 0x02cc06 thru 0x03a3fa (0xd7f5 bytes) 000253 1789050226.085688 mapwrite 0x01edf7 thru 0x022063 (0x326d bytes) 000256 1789050226.125971 punch from 0x014d7a to 0x018a8c (0x3d13 bytes) 000257 1789050226.132175 fallocate 0x00636e thru 0x00d94c (0x75df bytes) 000258 1789050226.155662 MIRROR_RESYNC 2 mirrors 000259 1789050227.351822 mapread 0x032d27 thru 0x0367fd (0x3ad7 bytes) 000260 1789050227.370479 fallocate 0x01ab32 thru 0x02148a (0x6959 bytes) 000261 1789050227.423338 fallocate 0x02c079 thru 0x0302ee (0x4276 bytes) 000262 1789050227.434389 mapread 0x022402 thru 0x028848 (0x6447 bytes) 000264 1789050227.447682 trunc from 0x03a3fb to 0x004fff (0x353fb bytes) 000265 1789050227.486450 read 0x002b5b thru 0x004fff (0x24a5 bytes) 000266 1789050227.504953 MIRROR_RESYNC 2 mirrors 000267 1789050228.617150 write 0x01cfd6 thru 0x024bc5 (0x7bf0 bytes) 000268 1789050228.731041 read 0x0074e3 thru 0x0087f0 (0x130e bytes) 000269 1789050228.741494 mapwrite 0x024af0 thru 0x025f3e (0x144f bytes) 000270 1789050228.851508 mapwrite 0x006040 thru 0x00f652 (0x9613 bytes) 000271 1789050228.920131 punch from 0x00c418 to 0x0118af (0x5498 bytes) 000272 1789050228.924850 read 0x008b8b thru 0x014501 (0xb977 bytes) 000273 1789050228.937005 trunc from 0x00f0ad to 0x01e3e2 (0xf336 bytes) 000274 1789050228.951126 trunc from 0x025f3f to 0x039fa7 (0x14069 bytes) 000275 1789050228.981763 write 0x010686 thru 0x0122e9 (0x1c64 bytes) 000277 1789050229.001460 trunc from 0x00f7d1 to 0x0180a4 (0x88d4 bytes) 000279 1789050229.025507 trunc from 0x039fa8 to 0x028269 (0x11d3e bytes) 000280 1789050229.059665 MIRROR_RESYNC 2 mirrors 000281 1789050230.027709 write 0x03bcc6 thru 0x03ffff (0x433a bytes) 000282 1789050230.104786 mapread 0x022f58 thru 0x0251a9 (0x2252 bytes) 000283 1789050230.141693 MIRROR_RESYNC 2 mirrors 000284 1789050231.379741 mapread 0x02b34d thru 0x03411b (0x8dcf bytes) 000285 1789050231.408769 read 0x028cbc thru 0x03690b (0xdc50 bytes) 000286 1789050231.426868 trunc from 0x015781 to 0x0207b5 (0xb035 bytes) 000287 1789050231.486845 write 0x01ad2d thru 0x023a3e (0x8d12 bytes) 000289 1789050231.528475 MIRROR_RESYNC 2 mirrors 000290 1789050232.506644 mapwrite 0x00b024 thru 0x00c898 (0x1875 bytes) 000291 1789050232.614907 fallocate 0x02e6a8 thru 0x036ad5 (0x842e bytes) 000292 1789050232.622272 read 0x01e472 thru 0x0266f5 (0x8284 bytes) 000293 1789050232.638005 trunc from 0x004e6b to 0x007ae5 (0x2c7b bytes) 000294 1789050232.646272 read 0x01091b thru 0x01a9c8 (0xa0ae bytes) 000295 1789050232.663683 punch from 0x017ded to 0x022f7b (0xb18f bytes) 000296 1789050232.670051 fallocate 0x0297f8 thru 0x02ac53 (0x145c bytes) 000298 1789050232.677391 mapwrite 0x00291f thru 0x006a34 (0x4116 bytes) 000300 1789050232.699435 write 0x005f23 thru 0x006a0b (0xae9 bytes) 000301 1789050232.712210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000302 1789050232.989101 read 0x024c31 thru 0x026486 (0x1856 bytes) 000303 1789050233.014635 MIRROR_EXTEND to 2 mirrors 000304 1789050234.491320 trunc from 0x040000 to 0x022058 (0x1dfa7 bytes) 000305 1789050234.592721 mapwrite 0x01c6dc thru 0x01f835 (0x315a bytes) 000306 1789050234.695215 mapread 0x019ab8 thru 0x01c790 (0x2cd9 bytes) 000307 1789050234.707749 MIRROR_RESYNC 2 mirrors 000308 1789050235.540661 trunc from 0x022059 to 0x0131a0 (0xeeb8 bytes) 000309 1789050235.612248 punch from 0x012463 to 0x0131a0 (0xd3e bytes) 000310 1789050235.625193 write 0x011511 thru 0x01ddb1 (0xc8a1 bytes) 000311 1789050235.631234 mapread 0x00f635 thru 0x01334f (0x3d1b bytes) 000312 1789050235.652627 mapread 0x01aced thru 0x01ddb1 (0x30c5 bytes) 000313 1789050235.653589 trunc from 0x00b690 to 0x013968 (0x82d9 bytes) 000314 1789050235.681820 trunc from 0x01ddb2 to 0x01a16e (0x3c43 bytes) 000315 1789050235.720305 trunc from 0x01a16f to 0x03ce73 (0x22d05 bytes) 000316 1789050235.766933 write 0x019e49 thru 0x02845e (0xe616 bytes) 000317 1789050235.777661 read 0x01e199 thru 0x020dca (0x2c32 bytes) 000318 1789050235.778844 mapread 0x03b3fb thru 0x03b90d (0x513 bytes) 000319 1789050235.793325 write 0x02da30 thru 0x037c16 (0xa1e7 bytes) 000320 1789050235.802836 mapwrite 0x03ebf1 thru 0x03ffff (0x140f bytes) 000321 1789050235.916839 read 0x021bdf thru 0x030399 (0xe7bb bytes) 000322 1789050235.929944 mapread 0x026c33 thru 0x02dba3 (0x6f71 bytes) 000323 1789050235.932482 mapread 0x0221d5 thru 0x02c1bf (0x9feb bytes) 000324 1789050235.933797 fallocate 0x028b34 thru 0x02b987 (0x2e54 bytes) 000325 1789050235.939493 mapread 0x03ed48 thru 0x03ffff (0x12b8 bytes) 000326 1789050235.940465 write 0x024e90 thru 0x034878 (0xf9e9 bytes) 000327 1789050235.944307 fallocate 0x00baab thru 0x01662d (0xab83 bytes) 000328 1789050235.954811 trunc from 0x0071f6 to 0x00739f (0x1aa bytes) 000329 1789050235.982310 punch from 0x01612f to 0x0176d9 (0x15ab bytes) 000330 1789050235.992043 fallocate 0x0051d6 thru 0x00e1ba (0x8fe5 bytes) 000331 1789050236.001669 fallocate 0x031d45 thru 0x03f089 (0xd345 bytes) 000332 1789050236.020593 trunc from 0x040000 to 0x00393d (0x3c6c2 bytes) 000333 1789050236.066885 read 0x003272 thru 0x00393d (0x6cc bytes) 000334 1789050236.073348 trunc from 0x000259 to 0x00393d (0x36e5 bytes) 000335 1789050236.090371 write 0x00048f thru 0x0061ec (0x5d5e bytes) 000336 1789050236.105915 punch from 0x0014af to 0x0061ec (0x4d3e bytes) 000337 1789050236.146093 trunc from 0x0018fd to 0x0061ec (0x48f0 bytes) 000338 1789050236.165629 MIRROR_RESYNC 2 mirrors 000339 1789050237.021337 trunc from 0x0061ed to 0x00a15c (0x3f70 bytes) 000340 1789050237.095363 punch from 0x008ab9 to 0x00a15c (0x16a4 bytes) 000341 1789050237.101926 mapwrite 0x033e17 thru 0x034988 (0xb72 bytes) 000342 1789050237.161829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34989): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34989): Mirror 2: 0x964534b0 (stale) CRC-32 checksum value for chunk [0, 0x34989): Mirror 1: 0x5e478bac Mirror 3: 0x5e478bac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34989: skipped 000343 1789050238.395059 mapread 0x01e35e thru 0x027f0e (0x9bb1 bytes) 000344 1789050238.439797 read 0x0127f2 thru 0x01a849 (0x8058 bytes) 000345 1789050238.451526 trunc from 0x00940b to 0x012855 (0x944b bytes) 000346 1789050238.494801 trunc from 0x000282 to 0x00d51e (0xd29d bytes) 000348 1789050238.511531 mapread 0x031aab thru 0x034988 (0x2ede bytes) 000349 1789050238.523750 punch from 0x006df5 to 0x0077d5 (0x9e1 bytes) 000350 1789050238.529714 fallocate 0x025679 thru 0x02b027 (0x59af bytes) 000351 1789050238.541177 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34989): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34989): Mirror 2: 0x964534b0 (stale) Mirror 3: 0x5e478bac (stale) CRC-32 checksum value for chunk [0, 0x34989): Mirror 1: 0xe6740004 Mirror 4: 0xe6740004 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34989: skipped 000352 1789050239.749522 fallocate 0x0060ee thru 0x013613 (0xd526 bytes) 000353 1789050239.788487 fallocate 0x02a28d thru 0x03366d (0x93e1 bytes) 000354 1789050239.794295 punch from 0x0329d3 to 0x034988 (0x1fb6 bytes) 000356 1789050239.808538 fallocate 0x0336b8 thru 0x035acf (0x2418 bytes) 000357 1789050239.827925 mapread 0x013242 thru 0x01bc70 (0x8a2f bytes) 000358 1789050239.858242 read 0x00ebc0 thru 0x019245 (0xa686 bytes) 000359 1789050239.859021 write 0x000766 thru 0x0009ff (0x29a bytes) 000360 1789050239.869181 punch from 0x01771c to 0x022613 (0xaef8 bytes) 000362 1789050239.875506 fallocate 0x03122a thru 0x03ff07 (0xecde bytes) 000363 1789050239.880522 punch from 0x00f172 to 0x01ee87 (0xfd16 bytes) 000364 1789050239.886687 read 0x0281fb thru 0x02be57 (0x3c5d bytes) 000365 1789050239.924972 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff08): Mirror 3: 0x8de560e8 (stale) Mirror 4: 0x3334e6e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff08): Mirror 3: 0x8de560e8 (stale) Mirror 4: 0x3334e6e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff08: skipped 000366 1789050240.404851 fallocate 0x004dff thru 0x00daa6 (0x8ca8 bytes) 000367 1789050240.415023 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff08): Mirror 4: 0x3334e6e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff08): Mirror 4: 0x3334e6e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff08: skipped 000368 1789050240.709517 trunc from 0x03ff08 to 0x020a57 (0x1f4b0 bytes) 000369 1789050240.735744 write 0x026653 thru 0x02fa9b (0x9449 bytes) 000370 1789050240.739019 trunc from 0x00ad7b to 0x01a803 (0xfa89 bytes) 000371 1789050240.744495 trunc from 0x0124f4 to 0x0190dd (0x6bea bytes) 000372 1789050240.749832 fallocate 0x020a40 thru 0x02843d (0x79fe bytes) 000373 1789050240.755417 mapwrite 0x00422e thru 0x00a158 (0x5f2b bytes) 000374 1789050240.796291 trunc from 0x02fa9c to 0x0184de (0x175bd bytes) 000375 1789050240.815600 read 0x002f80 thru 0x00b07e (0x80ff bytes) 000376 1789050240.816639 mapwrite 0x02799a thru 0x02c57b (0x4be2 bytes) 000377 1789050240.868821 read 0x0006a4 thru 0x00ea83 (0xe3e0 bytes) 000379 1789050240.883211 trunc from 0x0204d0 to 0x023015 (0x2b46 bytes) 000380 1789050240.889435 write 0x008329 thru 0x013846 (0xb51e bytes) 000381 1789050240.896704 MIRROR_RESYNC 2 mirrors 000382 1789050241.740528 trunc from 0x023a16 to 0x02bfe5 (0x85d0 bytes) 000383 1789050241.826549 MIRROR_RESYNC 2 mirrors 000384 1789050242.825803 punch from 0x00d0b6 to 0x00dfb1 (0xefc bytes) 000385 1789050242.938394 trunc from 0x02acd3 to 0x02c57b (0x18a9 bytes) 000386 1789050242.955949 trunc from 0x01b031 to 0x029f82 (0xef52 bytes) 000387 1789050242.962547 trunc from 0x02c57c to 0x01d7b7 (0xedc4 bytes) 000388 1789050242.980433 mapwrite 0x026c4f thru 0x02d7c1 (0x6b73 bytes) 000389 1789050243.040454 fallocate 0x0288a8 thru 0x03317b (0xa8d4 bytes) 000390 1789050243.047930 mapwrite 0x02067c thru 0x024f9c (0x4921 bytes) 000391 1789050243.101433 trunc from 0x0180c2 to 0x01d4df (0x541e bytes) 000392 1789050243.106662 read 0x02d167 thru 0x02d7c1 (0x65b bytes) 000393 1789050243.107290 punch from 0x019dc4 to 0x02655c (0xc799 bytes) 000394 1789050243.117793 mapwrite 0x004304 thru 0x00c789 (0x8486 bytes) 000395 1789050243.157546 punch from 0x0205e5 to 0x02194d (0x1369 bytes) 000396 1789050243.164730 read 0x018e9c thru 0x01a2be (0x1423 bytes) 000397 1789050243.178383 trunc from 0x02d7c2 to 0x02db39 (0x378 bytes) 000398 1789050243.194605 mapwrite 0x00174a thru 0x00fb22 (0xe3d9 bytes) 000399 1789050243.223167 fallocate 0x020059 thru 0x02f854 (0xf7fc bytes) 000400 1789050243.233742 MIRROR_RESYNC 2 mirrors 000401 1789050244.052218 trunc from 0x02f855 to 0x02240d (0xd447 bytes) 000402 1789050244.102326 mapread 0x007c30 thru 0x00f733 (0x7b04 bytes) 000403 1789050244.125517 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000405 1789050244.335703 trunc from 0x02240e to 0x036dfb (0x149ee bytes) 000406 1789050244.375075 write 0x028272 thru 0x029511 (0x12a0 bytes) 000407 1789050244.403824 write 0x01d98f thru 0x029ebe (0xc530 bytes) 000408 1789050244.412663 write 0x0065b9 thru 0x015a37 (0xf47f bytes) 000409 1789050244.432752 fallocate 0x003491 thru 0x011204 (0xdd74 bytes) 000410 1789050244.440991 trunc from 0x024528 to 0x03373d (0xf216 bytes) 000412 1789050244.482479 mapwrite 0x03b206 thru 0x03ffff (0x4dfa bytes) 000413 1789050244.565786 mapwrite 0x03840b thru 0x03ffff (0x7bf5 bytes) 000414 1789050244.592944 trunc from 0x026056 to 0x035ed1 (0xfe7c bytes) 000415 1789050244.600390 trunc from 0x040000 to 0x013990 (0x2c66f bytes) 000416 1789050244.617134 trunc from 0x00b5dd to 0x013990 (0x83b4 bytes) 000417 1789050244.623479 punch from 0x0117c2 to 0x013990 (0x21cf bytes) 000418 1789050244.635239 read 0x000f8c thru 0x010f1c (0xff91 bytes) 000419 1789050244.648763 write 0x016276 thru 0x01af26 (0x4cb1 bytes) 000420 1789050244.653416 punch from 0x019a3e to 0x01af26 (0x14e9 bytes) 000422 1789050244.692424 mapread 0x00b9e8 thru 0x018bc1 (0xd1da bytes) 000423 1789050244.707486 fallocate 0x02cd9b thru 0x037e71 (0xb0d7 bytes) 000425 1789050244.715086 mapwrite 0x03b008 thru 0x03e555 (0x354e bytes) 000426 1789050244.757380 fallocate 0x03001b thru 0x03213b (0x2121 bytes) 000427 1789050244.763138 punch from 0x0311a5 to 0x03226e (0x10ca bytes) 000428 1789050244.773803 punch from 0x034e8b to 0x03e555 (0x96cb bytes) 000429 1789050244.784602 fallocate 0x038163 thru 0x03ffff (0x7e9d bytes) 000430 1789050244.791219 trunc from 0x03e556 to 0x009ea5 (0x346b0 bytes) 000431 1789050244.807707 trunc from 0x009ea6 to 0x0134c4 (0x961f bytes) 000432 1789050244.821902 mapread 0x00edbe thru 0x012442 (0x3685 bytes) 000433 1789050244.830939 fallocate 0x02ea75 thru 0x03c441 (0xd9cd bytes) 000435 1789050244.839341 read 0x02d58b thru 0x03376d (0x61e3 bytes) 000436 1789050244.853028 punch from 0x01e283 to 0x01f615 (0x1393 bytes) 000437 1789050244.857412 punch from 0x032d0a to 0x037541 (0x4838 bytes) 000438 1789050244.863553 read 0x03141d thru 0x032325 (0xf09 bytes) 000440 1789050244.877736 mapread 0x01dd7c thru 0x02138d (0x3612 bytes) 000441 1789050244.879124 read 0x002c25 thru 0x008a34 (0x5e10 bytes) 000442 1789050244.888065 MIRROR_EXTEND to 2 mirrors 000443 1789050246.217120 trunc from 0x0192b2 to 0x01f915 (0x6664 bytes) 000444 1789050246.265786 trunc from 0x007ed2 to 0x00abe5 (0x2d14 bytes) 000445 1789050246.274911 fallocate 0x03c277 thru 0x03f34b (0x30d5 bytes) 000446 1789050246.290956 write 0x0275c2 thru 0x031371 (0x9db0 bytes) 000447 1789050246.313991 trunc from 0x03f34c to 0x023a4a (0x1b901 bytes) 000448 1789050246.341483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000449 1789050246.593743 punch from 0x01ce52 to 0x020a00 (0x3baf bytes) 000450 1789050246.605381 read 0x0227d3 thru 0x023a4a (0x1278 bytes) 000451 1789050246.614440 write 0x034ebf thru 0x037877 (0x29b9 bytes) 000452 1789050246.622484 fallocate 0x007fa5 thru 0x00d276 (0x52d2 bytes) 000453 1789050246.633786 mapwrite 0x02875e thru 0x02b8eb (0x318e bytes) 000454 1789050246.682875 trunc from 0x0317be to 0x037877 (0x60ba bytes) 000455 1789050246.689984 write 0x0024f1 thru 0x00ff11 (0xda21 bytes) 000456 1789050246.710852 fallocate 0x0227af thru 0x023f9f (0x17f1 bytes) 000458 1789050246.727526 mapwrite 0x018017 thru 0x01fcda (0x7cc4 bytes) 000459 1789050246.781736 punch from 0x007825 to 0x00930c (0x1ae8 bytes) 000460 1789050246.788495 trunc from 0x01f4ae to 0x027b72 (0x86c5 bytes) 000461 1789050246.795507 trunc from 0x037878 to 0x001081 (0x367f6 bytes) 000462 1789050246.816514 fallocate 0x0052b0 thru 0x00a8a0 (0x55f1 bytes) 000463 1789050246.823079 write 0x01cb96 thru 0x02c627 (0xfa92 bytes) 000464 1789050246.826707 punch from 0x027d49 to 0x02c627 (0x48df bytes) 000465 1789050246.847761 write 0x018928 thru 0x01f767 (0x6e40 bytes) 000466 1789050246.856093 read 0x008d37 thru 0x01002f (0x72f9 bytes) 000467 1789050246.875127 read 0x00252e thru 0x004e23 (0x28f6 bytes) 000468 1789050246.876602 write 0x00c637 thru 0x01a0b3 (0xda7d bytes) 000469 1789050246.879970 mapwrite 0x006add thru 0x00a491 (0x39b5 bytes) 000470 1789050246.909694 trunc from 0x001dbd to 0x00841c (0x6660 bytes) 000471 1789050246.917690 trunc from 0x011aa8 to 0x016ecd (0x5426 bytes) 000473 1789050246.925391 punch from 0x00df5d to 0x01049d (0x2541 bytes) 000475 1789050246.933500 fallocate 0x035c45 thru 0x03cb75 (0x6f31 bytes) 000476 1789050246.939783 mapwrite 0x001796 thru 0x00ea1d (0xd288 bytes) 000477 1789050246.975329 fallocate 0x036578 thru 0x03df1f (0x79a8 bytes) 000478 1789050246.980863 mapread 0x00da0b thru 0x01c306 (0xe8fc bytes) 000479 1789050246.982750 fallocate 0x008bf0 thru 0x008c92 (0xa3 bytes) 000480 1789050246.989405 punch from 0x00f82c to 0x015da7 (0x657c bytes) 000481 1789050246.995524 fallocate 0x038ed3 thru 0x03ffff (0x712d bytes) 000482 1789050247.001097 trunc from 0x040000 to 0x00e658 (0x319a7 bytes) 000483 1789050247.020777 fallocate 0x03a724 thru 0x03ffff (0x58dc bytes) 000484 1789050247.035744 mapwrite 0x019bec thru 0x01bd1b (0x2130 bytes) 000485 1789050247.089037 mapwrite 0x0223b1 thru 0x025799 (0x33e9 bytes) 000487 1789050247.167874 mapread 0x001310 thru 0x002207 (0xef8 bytes) 000488 1789050247.174132 write 0x026107 thru 0x02bd9b (0x5c95 bytes) 000489 1789050247.177595 read 0x01f880 thru 0x02718b (0x790c bytes) 000490 1789050247.192465 trunc from 0x02bd9c to 0x00cade (0x1f2bd bytes) 000491 1789050247.216520 trunc from 0x005c09 to 0x00cade (0x6ed6 bytes) 000492 1789050247.224996 mapread 0x001bb8 thru 0x008a98 (0x6ee1 bytes) 000493 1789050247.247508 read 0x00572b thru 0x0064ca (0xda0 bytes) 000495 1789050247.249235 trunc from 0x000961 to 0x00a407 (0x9aa7 bytes) 000496 1789050247.260606 MIRROR_EXTEND to 2 mirrors 000497 1789050248.329475 MIRROR_EXTEND to 3 mirrors 000498 1789050249.568446 mapread 0x006377 thru 0x00b7cf (0x5459 bytes) 000501 1789050249.665487 trunc from 0x00cadf to 0x02bd89 (0x1f2ab bytes) 000502 1789050249.801895 trunc from 0x02bd8a to 0x007036 (0x24d53 bytes) 000503 1789050249.849124 mapread 0x004271 thru 0x00468b (0x41b bytes) 000504 1789050249.872057 mapwrite 0x005a0e thru 0x009296 (0x3889 bytes) 000505 1789050249.990268 MIRROR_RESYNC 3 mirrors 000506 1789050251.141427 write 0x02da96 thru 0x0342b6 (0x6821 bytes) 000507 1789050251.180450 fallocate 0x01a3c3 thru 0x01d8c7 (0x3505 bytes) 000508 1789050251.188435 fallocate 0x005e92 thru 0x01280e (0xc97d bytes) 000509 1789050251.194497 read 0x02faca thru 0x0342b6 (0x47ed bytes) 000510 1789050251.195525 trunc from 0x0342b7 to 0x03bff7 (0x7d41 bytes) 000511 1789050251.234711 read 0x00091c thru 0x00428a (0x396f bytes) 000512 1789050251.262404 trunc from 0x005a3e to 0x013c10 (0xe1d3 bytes) 000513 1789050251.281337 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bff8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff8): Mirror 2: 0xc1b2c57d (stale) Mirror 3: 0xc1b2c57d (stale) CRC-32 checksum value for chunk [0, 0x3bff8): Mirror 1: 0xc0cbb92b Mirror 4: 0xc0cbb92b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff8: skipped 000514 1789050253.156800 write 0x01d85e thru 0x020dc7 (0x356a bytes) 000515 1789050253.276139 punch from 0x033df5 to 0x0395f3 (0x57ff bytes) 000516 1789050253.282461 trunc from 0x0224d7 to 0x0283d1 (0x5efb bytes) 000517 1789050253.292504 mapwrite 0x026cf7 thru 0x030209 (0x9513 bytes) 000518 1789050253.364801 punch from 0x01d2de to 0x020e49 (0x3b6c bytes) 000519 1789050253.372888 trunc from 0x03bff8 to 0x0239e7 (0x18610 bytes) 000520 1789050253.401994 read 0x01877a thru 0x018a11 (0x298 bytes) 000521 1789050253.414377 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x239e8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x239e8): Mirror 2: 0xbe7f5e01 (stale) Mirror 3: 0xbe7f5e01 (stale) Mirror 4: 0x426d80f0 (stale) CRC-32 checksum value for chunk [0, 0x239e8): Mirror 1: 0x426d80f0 Mirror 5: 0x426d80f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x239e8: skipped 000522 1789050255.296141 punch from 0x0063d6 to 0x00ea68 (0x8693 bytes) 000523 1789050255.353785 write 0x03a03f thru 0x03e82c (0x47ee bytes) 000525 1789050255.365433 MIRROR_RESYNC 5 mirrors 000526 1789050257.988524 mapread 0x025507 thru 0x02e3f7 (0x8ef1 bytes) 000527 1789050258.048405 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) 000528 1789050258.571752 punch from 0x01bb8d to 0x026b50 (0xafc4 bytes) 000529 1789050258.625010 punch from 0x01d39a to 0x02618a (0x8df1 bytes) 000530 1789050258.631527 write 0x024730 thru 0x02846e (0x3d3f bytes) 000532 1789050258.653130 write 0x01913f thru 0x022803 (0x96c5 bytes) 000533 1789050258.678781 MIRROR_RESYNC 4 mirrors 000534 1789050260.898674 mapread 0x0137ff thru 0x01aa68 (0x726a bytes) 000535 1789050260.975467 trunc from 0x03e82d to 0x011fe2 (0x2c84a bytes) 000536 1789050261.057899 mapread 0x009a2b thru 0x00c280 (0x2856 bytes) 000537 1789050261.069987 fallocate 0x00a645 thru 0x00f5e0 (0x4f9c bytes) 000538 1789050261.089415 punch from 0x0028f5 to 0x011fe2 (0xf6ee bytes) 000539 1789050261.103378 fallocate 0x01aad5 thru 0x01cd9e (0x22ca bytes) 000540 1789050261.111411 mapwrite 0x00219a thru 0x010920 (0xe787 bytes) 000541 1789050261.180298 trunc from 0x011fe3 to 0x01d894 (0xb8b2 bytes) 000542 1789050261.221641 mapwrite 0x0178e9 thru 0x01913e (0x1856 bytes) 000543 1789050261.258444 read 0x01a78d thru 0x01d894 (0x3108 bytes) 000544 1789050261.259166 fallocate 0x0205cd thru 0x02cded (0xc821 bytes) 000545 1789050261.271668 MIRROR_RESYNC 4 mirrors 000546 1789050262.971193 read 0x019ca8 thru 0x028457 (0xe7b0 bytes) 000547 1789050263.014387 MIRROR_RESYNC 4 mirrors 000548 1789050263.795240 trunc from 0x02cdee to 0x0146c4 (0x18729 bytes) 000549 1789050263.858654 mapwrite 0x0360f6 thru 0x038dcb (0x2cd6 bytes) 000550 1789050263.985374 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 4: 0xe7313679 (stale) Mirror 5: 0xe7313679 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 4: 0xe7313679 (stale) Mirror 5: 0xe7313679 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38dcc: skipped 000551 1789050264.713067 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 4: 0xe7313679 (stale) Mirror 5: 0xe7313679 (stale) CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 1: 0x396c7257 Mirror 6: 0x396c7257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38dcc: skipped 000552 1789050266.202523 trunc from 0x038dcc to 0x01b6ac (0x1d71f bytes) 000553 1789050266.280150 mapread 0x0032ba thru 0x00cee3 (0x9c2a bytes) 000554 1789050266.307182 read 0x015a88 thru 0x01b6ac (0x5c25 bytes) 000555 1789050266.316226 trunc from 0x0043c7 to 0x0076d0 (0x330a bytes) 000556 1789050266.325291 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b6ad): Mirror 5: 0x46c8e419 (stale) Mirror 6: 0x9a75ec63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b6ad): Mirror 5: 0x46c8e419 (stale) Mirror 6: 0x9a75ec63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b6ad: skipped 000557 1789050266.769230 trunc from 0x010d2b to 0x01ae79 (0xa14f bytes) 000558 1789050266.774376 write 0x03e593 thru 0x03ffff (0x1a6d bytes) 000559 1789050266.782357 write 0x017474 thru 0x0199bd (0x254a bytes) 000560 1789050266.806210 mapread 0x0326eb thru 0x03bfaf (0x98c5 bytes) 000561 1789050266.850106 write 0x031f4a thru 0x03ee95 (0xcf4c bytes) 000562 1789050266.857070 mapwrite 0x03d1a8 thru 0x03ffff (0x2e58 bytes) 000563 1789050266.899522 mapwrite 0x0243ba thru 0x030e8d (0xcad4 bytes) 000565 1789050266.970394 punch from 0x02e7d6 to 0x032541 (0x3d6c bytes) 000566 1789050266.989520 mapwrite 0x02a8b2 thru 0x02f7fa (0x4f49 bytes) 000567 1789050267.053916 mapread 0x02b124 thru 0x0349a0 (0x987d bytes) 000568 1789050267.056006 punch from 0x01741f to 0x01f5ae (0x8190 bytes) 000569 1789050267.064399 write 0x03f362 thru 0x03ffff (0xc9e bytes) 000570 1789050267.067370 trunc from 0x001ca9 to 0x00e005 (0xc35d bytes) 000571 1789050267.088332 trunc from 0x0120da to 0x020199 (0xe0c0 bytes) 000572 1789050267.108170 mapwrite 0x0311bc thru 0x03a5ab (0x93f0 bytes) 000573 1789050267.158719 mapread 0x0364de thru 0x037cb6 (0x17d9 bytes) 000574 1789050267.159742 fallocate 0x011fbd thru 0x014c4a (0x2c8e bytes) 000575 1789050267.188169 fallocate 0x03656b thru 0x03c198 (0x5c2e bytes) 000576 1789050267.195892 write 0x002df3 thru 0x00be82 (0x9090 bytes) 000577 1789050267.213043 mapread 0x03030b thru 0x039ec7 (0x9bbd bytes) 000578 1789050267.214289 read 0x025404 thru 0x026194 (0xd91 bytes) 000579 1789050267.215507 trunc from 0x026677 to 0x031db9 (0xb743 bytes) 000580 1789050267.232504 punch from 0x01dfc9 to 0x02dd87 (0xfdbf bytes) 000581 1789050267.239764 read 0x028661 thru 0x029964 (0x1304 bytes) 000582 1789050267.260483 mapwrite 0x002362 thru 0x00711b (0x4dba bytes) 000583 1789050267.297377 read 0x0005f4 thru 0x00bef7 (0xb904 bytes) 000584 1789050267.298903 trunc from 0x0151ef to 0x022540 (0xd352 bytes) 000585 1789050267.304944 read 0x03af9d thru 0x03ffff (0x5063 bytes) 000586 1789050267.306198 fallocate 0x009349 thru 0x00db41 (0x47f9 bytes) 000588 1789050267.314270 punch from 0x015016 to 0x017f8b (0x2f76 bytes) 000589 1789050267.319060 trunc from 0x00cfc7 to 0x018287 (0xb2c1 bytes) 000591 1789050267.324090 read 0x033fc1 thru 0x03ffff (0xc03f bytes) 000592 1789050267.341548 MIRROR_RESYNC 3 mirrors 000593 1789050268.178311 read 0x02594d thru 0x0279ed (0x20a1 bytes) 000594 1789050268.190042 fallocate 0x03d87a thru 0x03ffff (0x2786 bytes) 000595 1789050268.297050 read 0x03f93d thru 0x03ffff (0x6c3 bytes) 000596 1789050268.335658 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2822e996 (stale) Mirror 6: 0x2822e996 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2822e996 Mirror 7: 0x2822e996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000599 1789050269.713548 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2822e996 (stale) Mirror 6: 0x2822e996 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2822e996 Mirror 7: 0x2822e996 Mirror 8: 0x2822e996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1789050270.976050 trunc from 0x040000 to 0x006b28 (0x394d7 bytes) 000601 1789050271.064711 read 0x0037a9 thru 0x006b28 (0x3380 bytes) 000602 1789050271.080647 mapread 0x005383 thru 0x006b28 (0x17a6 bytes) 000603 1789050271.083552 mapwrite 0x002d7a thru 0x004981 (0x1c08 bytes) 000606 1789050271.130950 mapread 0x00135c thru 0x006b28 (0x57cd bytes) 000607 1789050271.159266 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x6b29): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b29): Mirror 5: 0x80c7b851 (stale) Mirror 6: 0x80c7b851 (stale) Mirror 7: 0x80c7b851 (stale) Mirror 8: 0x80c7b851 (stale) CRC-32 checksum value for chunk [0, 0x6b29): Mirror 1: 0xb248084b Mirror 9: 0xb248084b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b29: skipped 000608 1789050272.900145 trunc from 0x006b29 to 0x027485 (0x2095d bytes) 000609 1789050273.009470 mapread 0x012187 thru 0x019693 (0x750d bytes) 000610 1789050273.075910 fallocate 0x01145c thru 0x012281 (0xe26 bytes) 000611 1789050273.107277 fallocate 0x02771b thru 0x02a34b (0x2c31 bytes) 000612 1789050273.120893 punch from 0x013795 to 0x01380a (0x76 bytes) 000613 1789050273.128638 read 0x01c2f5 thru 0x026418 (0xa124 bytes) 000614 1789050273.148275 trunc from 0x02a34c to 0x003047 (0x27304 bytes) 000615 1789050273.213230 read 0x0011b0 thru 0x003047 (0x1e98 bytes) 000616 1789050273.242055 MIRROR_RESYNC 6 mirrors 000617 1789050275.144764 mapwrite 0x012ccc thru 0x0201ff (0xd534 bytes) 000618 1789050275.277472 write 0x00afbd thru 0x014a90 (0x9ad4 bytes) 000619 1789050275.288971 write 0x0215ca thru 0x025d13 (0x474a bytes) 000620 1789050275.290909 trunc from 0x025d14 to 0x03f898 (0x19b85 bytes) 000621 1789050275.333890 trunc from 0x03f899 to 0x013c63 (0x2bc35 bytes) 000622 1789050275.366427 write 0x0354cb thru 0x039db6 (0x48ec bytes) 000623 1789050275.369483 trunc from 0x003a36 to 0x00beb6 (0x8481 bytes) 000624 1789050275.388752 mapread 0x0305df thru 0x039db6 (0x97d8 bytes) 000625 1789050275.403477 write 0x02b8a5 thru 0x03561d (0x9d79 bytes) 000626 1789050275.405800 read 0x016744 thru 0x02547e (0xed3b bytes) 000627 1789050275.420441 mapread 0x00d99a thru 0x012ea7 (0x550e bytes) 000628 1789050275.421407 mapread 0x023daf thru 0x0269a5 (0x2bf7 bytes) 000629 1789050275.428990 mapwrite 0x008125 thru 0x0172d2 (0xf1ae bytes) 000630 1789050275.510432 mapread 0x01a801 thru 0x0207e3 (0x5fe3 bytes) 000631 1789050275.511342 write 0x01b851 thru 0x029460 (0xdc10 bytes) 000632 1789050275.521165 mapwrite 0x008eb0 thru 0x0093f0 (0x541 bytes) 000633 1789050275.578131 read 0x005929 thru 0x0078d9 (0x1fb1 bytes) 000634 1789050275.579166 mapread 0x01787b thru 0x018ba7 (0x132d bytes) 000635 1789050275.584979 trunc from 0x039db7 to 0x01d63a (0x1c77c bytes) 000636 1789050275.622527 read 0x01a41b thru 0x01d63a (0x3220 bytes) 000637 1789050275.623678 write 0x011388 thru 0x01ed40 (0xd9b9 bytes) 000638 1789050275.653084 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ed41): Mirror 5: 0x1ca30494 (stale) Mirror 6: 0x1ca30494 (stale) Mirror 7: 0x1ca30494 (stale) Mirror 9: 0x1ca30494 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed41): Mirror 5: 0x1ca30494 (stale) Mirror 6: 0x1ca30494 (stale) Mirror 7: 0x1ca30494 (stale) Mirror 9: 0x1ca30494 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed41: skipped 000640 1789050276.429978 mapread 0x0034ae thru 0x006536 (0x3089 bytes) 000641 1789050276.504988 write 0x009c5d thru 0x015f41 (0xc2e5 bytes) 000642 1789050276.563772 trunc from 0x01ed41 to 0x00d30f (0x11a31 bytes) 000643 1789050276.597091 trunc from 0x00c735 to 0x00d30f (0xbdb bytes) 000644 1789050276.672548 MIRROR_RESYNC 5 mirrors 000645 1789050278.582713 mapread 0x00b8ef thru 0x00d06b (0x177d bytes) 000646 1789050278.607384 mapwrite 0x00ed4f thru 0x0138b6 (0x4b68 bytes) 000647 1789050278.690782 mapwrite 0x032565 thru 0x035a5a (0x34f6 bytes) 000648 1789050278.780906 trunc from 0x035a5b to 0x016773 (0x1f2e7 bytes) 000649 1789050278.804835 trunc from 0x004f4f to 0x006d5b (0x1e0d bytes) 000650 1789050278.824100 MIRROR_RESYNC 5 mirrors 000651 1789050280.534723 trunc from 0x016774 to 0x02f85d (0x190ea bytes) 000653 1789050280.611641 trunc from 0x02f85e to 0x010e50 (0x1ea0d bytes) 000654 1789050280.667017 mapread 0x00c11e thru 0x010e50 (0x4d33 bytes) 000655 1789050280.723859 trunc from 0x004d93 to 0x00789f (0x2b0d bytes) 000656 1789050280.746844 mapread 0x00cbac thru 0x010e50 (0x42a5 bytes) 000657 1789050280.752091 mapread 0x006de8 thru 0x010e50 (0xa069 bytes) 000658 1789050280.797840 MIRROR_RESYNC 5 mirrors 000659 1789050282.486705 fallocate 0x00f308 thru 0x0108b3 (0x15ac bytes) 000660 1789050282.527092 write 0x00f279 thru 0x012ef5 (0x3c7d bytes) 000661 1789050282.535359 trunc from 0x012ef6 to 0x01b0d3 (0x81de bytes) 000662 1789050282.557885 trunc from 0x01b0d4 to 0x01b336 (0x263 bytes) 000663 1789050282.584399 punch from 0x002dea to 0x00e799 (0xb9b0 bytes) 000664 1789050282.595691 trunc from 0x006b51 to 0x00966f (0x2b1f bytes) 000665 1789050282.603690 mapwrite 0x01aac0 thru 0x025c56 (0xb197 bytes) 000666 1789050282.695348 fallocate 0x028cdd thru 0x02f5a7 (0x68cb bytes) 000668 1789050282.703455 trunc from 0x02f5a8 to 0x03cae8 (0xd541 bytes) 000669 1789050282.735089 fallocate 0x03beff thru 0x03ffff (0x4101 bytes) 000670 1789050282.743042 trunc from 0x03cae9 to 0x03bc7a (0xe6e bytes) 000672 1789050282.773485 fallocate 0x034a38 thru 0x03ffff (0xb5c8 bytes) 000673 1789050282.781471 read 0x01f67b thru 0x02d728 (0xe0ae bytes) 000674 1789050282.800938 punch from 0x004fc4 to 0x00882c (0x3869 bytes) 000675 1789050282.806834 write 0x01448e thru 0x023fc3 (0xfb36 bytes) 000676 1789050282.815266 punch from 0x02d3a9 to 0x02eb69 (0x17c1 bytes) 000677 1789050282.821898 trunc from 0x040000 to 0x00604a (0x39fb5 bytes) 000678 1789050282.851452 mapwrite 0x01722e thru 0x0188de (0x16b1 bytes) 000679 1789050282.906106 trunc from 0x0188df to 0x01b6da (0x2dfc bytes) 000680 1789050282.934975 fallocate 0x012aef thru 0x0178f6 (0x4e08 bytes) 000681 1789050282.942019 fallocate 0x03b3a2 thru 0x03fdef (0x4a4e bytes) 000682 1789050282.949663 mapwrite 0x03ea16 thru 0x03ffff (0x15ea bytes) 000683 1789050283.003273 trunc from 0x040000 to 0x025608 (0x1a9f7 bytes) 000684 1789050283.025544 trunc from 0x0122b0 to 0x01817a (0x5ecb bytes) 000685 1789050283.032972 punch from 0x005a63 to 0x01104e (0xb5ec bytes) 000686 1789050283.039466 trunc from 0x01910e to 0x02300e (0x9f01 bytes) 000687 1789050283.045701 trunc from 0x012a4f to 0x0190a5 (0x6657 bytes) 000688 1789050283.051958 punch from 0x01403a to 0x01bfbe (0x7f85 bytes) 000689 1789050283.058014 write 0x024948 thru 0x026950 (0x2009 bytes) 000690 1789050283.065383 write 0x021b7d thru 0x02e3df (0xc863 bytes) 000691 1789050283.073618 mapwrite 0x00f879 thru 0x0174ca (0x7c52 bytes) 000692 1789050283.112701 punch from 0x01f1de to 0x0293a4 (0xa1c7 bytes) 000693 1789050283.121191 read 0x0066eb thru 0x0150c6 (0xe9dc bytes) 000694 1789050283.137485 mapread 0x0095ba thru 0x016221 (0xcc68 bytes) 000695 1789050283.140004 punch from 0x00e33e to 0x00eff5 (0xcb8 bytes) 000696 1789050283.147291 write 0x03be13 thru 0x03bf76 (0x164 bytes) 000698 1789050283.151407 mapread 0x010803 thru 0x01ff21 (0xf71f bytes) 000699 1789050283.154523 punch from 0x00c486 to 0x01aaab (0xe626 bytes) 000700 1789050283.163988 read 0x037b42 thru 0x038eea (0x13a9 bytes) 000701 1789050283.182372 trunc from 0x007113 to 0x00bf28 (0x4e16 bytes) 000703 1789050283.188369 trunc from 0x00288f to 0x00bbb4 (0x9326 bytes) 000704 1789050283.194021 read 0x028da0 thru 0x036295 (0xd4f6 bytes) 000705 1789050283.195326 trunc from 0x03bf77 to 0x00dccd (0x2e2a9 bytes) 000706 1789050283.212188 trunc from 0x0085c7 to 0x00dccd (0x5707 bytes) 000707 1789050283.225678 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xdcce): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdcce): Mirror 5: 0xad016a1d (stale) Mirror 6: 0xad016a1d (stale) Mirror 7: 0xad016a1d (stale) Mirror 9: 0xad016a1d (stale) CRC-32 checksum value for chunk [0, 0xdcce): Mirror 1: 0x8dd4e398 Mirror 10: 0x8dd4e398 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdcce: skipped 000708 1789050284.315937 fallocate 0x03cb68 thru 0x03dfee (0x1487 bytes) 000710 1789050284.379247 trunc from 0x00dcce to 0x018f89 (0xb2bc bytes) 000711 1789050284.450929 mapwrite 0x01140b thru 0x017048 (0x5c3e bytes) 000712 1789050284.499975 trunc from 0x016723 to 0x018b28 (0x2406 bytes) 000713 1789050284.508007 mapread 0x001724 thru 0x009a16 (0x82f3 bytes) 000715 1789050284.523769 write 0x00fb57 thru 0x017afb (0x7fa5 bytes) 000716 1789050284.541972 mapwrite 0x021472 thru 0x026192 (0x4d21 bytes) 000718 1789050284.620831 fallocate 0x02e957 thru 0x02fdf2 (0x149c bytes) 000719 1789050284.630015 trunc from 0x02fdf3 to 0x01306f (0x1cd83 bytes) 000720 1789050284.656601 punch from 0x006763 to 0x00976c (0x300a bytes) 000721 1789050284.663290 trunc from 0x005b4d to 0x007b6b (0x201f bytes) 000722 1789050284.669746 mapread 0x003307 thru 0x00d12f (0x9e29 bytes) 000723 1789050284.681079 trunc from 0x00bd5d to 0x01306f (0x7313 bytes) 000724 1789050284.690178 punch from 0x011fef to 0x01306f (0x1081 bytes) 000725 1789050284.699944 fallocate 0x03835b thru 0x03ffff (0x7ca5 bytes) 000726 1789050284.709146 mapwrite 0x004ed8 thru 0x0121b5 (0xd2de bytes) 000727 1789050284.762802 punch from 0x017fe5 to 0x01bce6 (0x3d02 bytes) 000728 1789050284.768366 read 0x004bf9 thru 0x009c46 (0x504e bytes) 000729 1789050284.769055 trunc from 0x040000 to 0x02fbd7 (0x10428 bytes) 000730 1789050284.791081 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fbd8): Mirror 6: 0xba6d6cf1 (stale) Mirror 7: 0xba6d6cf1 (stale) Mirror 9: 0xba6d6cf1 (stale) Mirror 10: 0x137c51c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fbd8): Mirror 6: 0xba6d6cf1 (stale) Mirror 7: 0xba6d6cf1 (stale) Mirror 9: 0xba6d6cf1 (stale) Mirror 10: 0x137c51c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fbd8: skipped 000731 1789050285.333793 write 0x0335f8 thru 0x03e8c2 (0xb2cb bytes) 000732 1789050285.356316 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8c3): Mirror 7: 0xcc129223 (stale) Mirror 9: 0xcc129223 (stale) Mirror 10: 0xa2e3e22f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8c3): Mirror 7: 0xcc129223 (stale) Mirror 9: 0xcc129223 (stale) Mirror 10: 0xa2e3e22f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8c3: skipped 000733 1789050285.838318 trunc from 0x0105c6 to 0x014f20 (0x495b bytes) 000734 1789050285.845604 mapread 0x026066 thru 0x02f50b (0x94a6 bytes) 000735 1789050285.884178 trunc from 0x007dfc to 0x009471 (0x1676 bytes) 000736 1789050285.891113 trunc from 0x03e1c8 to 0x03e8c2 (0x6fb bytes) 000737 1789050285.897022 mapread 0x0171ed thru 0x02338e (0xc1a2 bytes) 000738 1789050285.924590 read 0x0125ef thru 0x01a0d0 (0x7ae2 bytes) 000739 1789050285.951672 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8c3): Mirror 7: 0xcc129223 (stale) Mirror 10: 0xa2e3e22f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8c3): Mirror 7: 0xcc129223 (stale) Mirror 10: 0xa2e3e22f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8c3: skipped 000740 1789050286.533856 punch from 0x01c9e6 to 0x01cd6b (0x386 bytes) 000741 1789050286.543741 trunc from 0x03e8c3 to 0x01cf30 (0x21992 bytes) 000742 1789050286.582812 trunc from 0x013048 to 0x015554 (0x250d bytes) 000743 1789050286.592665 punch from 0x00678e to 0x0148fb (0xe16e bytes) 000744 1789050286.612850 write 0x01bdea thru 0x01dfaf (0x21c6 bytes) 000745 1789050286.627864 fallocate 0x018bd3 thru 0x01b9c9 (0x2df7 bytes) 000746 1789050286.642255 mapwrite 0x004716 thru 0x012201 (0xdaec bytes) 000748 1789050286.838064 fallocate 0x0167c7 thru 0x0251bb (0xe9f5 bytes) 000749 1789050286.871043 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x251bc): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251bc): Mirror 7: 0x5e90248e (stale) Mirror 10: 0x413ebcd2 (stale) CRC-32 checksum value for chunk [0, 0x251bc): Mirror 1: 0x4412fbf9 Mirror 11: 0x4412fbf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251bc: skipped 000750 1789050288.270404 trunc from 0x01ebd7 to 0x0251bb (0x65e5 bytes) 000752 1789050288.329177 trunc from 0x00662f to 0x00bfed (0x59bf bytes) 000753 1789050288.337743 mapwrite 0x0164a8 thru 0x01d828 (0x7381 bytes) 000754 1789050288.395124 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x251bc): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251bc): Mirror 7: 0x5e90248e (stale) Mirror 10: 0x413ebcd2 (stale) Mirror 11: 0x4412fbf9 (stale) CRC-32 checksum value for chunk [0, 0x251bc): Mirror 1: 0x6dc8b856 Mirror 12: 0x6dc8b856 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251bc: skipped 000755 1789050290.110918 mapwrite 0x012609 thru 0x01a383 (0x7d7b bytes) 000756 1789050290.315651 mapread 0x00c90e thru 0x00ca52 (0x145 bytes) 000757 1789050290.324504 MIRROR_RESYNC 5 mirrors 000758 1789050292.527837 mapwrite 0x01827c thru 0x022c91 (0xaa16 bytes) 000759 1789050292.708757 fallocate 0x009495 thru 0x00e5b2 (0x511e bytes) 000761 1789050292.720256 mapwrite 0x022e4f thru 0x0310bc (0xe26e bytes) 000762 1789050292.839475 mapwrite 0x02703a thru 0x0361ab (0xf172 bytes) 000765 1789050292.919250 write 0x00c06b thru 0x01a7c8 (0xe75e bytes) 000766 1789050292.935289 MIRROR_RESYNC 5 mirrors 000767 1789050294.852444 read 0x0251e0 thru 0x02d10c (0x7f2d bytes) 000768 1789050294.864779 punch from 0x026a32 to 0x0345f1 (0xdbc0 bytes) 000770 1789050294.938528 trunc from 0x02a58b to 0x0359c9 (0xb43f bytes) 000771 1789050294.964726 mapread 0x024e7f thru 0x02cba1 (0x7d23 bytes) 000772 1789050295.005450 MIRROR_RESYNC 5 mirrors 000773 1789050297.736308 punch from 0x018d02 to 0x01adb6 (0x20b5 bytes) 000774 1789050297.809056 mapwrite 0x021884 thru 0x02f976 (0xe0f3 bytes) 000775 1789050297.886266 punch from 0x02a89e to 0x0358b2 (0xb015 bytes) 000776 1789050297.921470 write 0x02fbd2 thru 0x0357c1 (0x5bf0 bytes) 000778 1789050297.952677 trunc from 0x033c18 to 0x0361ab (0x2594 bytes) 000780 1789050297.988039 write 0x02ee45 thru 0x0318d3 (0x2a8f bytes) 000781 1789050298.008098 fallocate 0x01c80b thru 0x026cbf (0xa4b5 bytes) 000782 1789050298.026024 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x361ac): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361ac): Mirror 7: 0xde58bc32 (stale) Mirror 10: 0xde58bc32 (stale) Mirror 11: 0xde58bc32 (stale) Mirror 12: 0xde58bc32 (stale) CRC-32 checksum value for chunk [0, 0x361ac): Mirror 1: 0xf1c663a1 Mirror 13: 0xf1c663a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361ac: skipped 000783 1789050299.995537 write 0x029e66 thru 0x031f72 (0x810d bytes) 000784 1789050300.079486 write 0x0212b5 thru 0x0310a1 (0xfded bytes) 000785 1789050300.096197 write 0x039b89 thru 0x03e28f (0x4707 bytes) 000786 1789050300.102412 write 0x02e515 thru 0x03b5d0 (0xd0bc bytes) 000787 1789050300.109789 write 0x0141ef thru 0x018679 (0x448b bytes) 000788 1789050300.123677 trunc from 0x03e290 to 0x01c859 (0x21a36 bytes) 000789 1789050300.176403 MIRROR_RESYNC 6 mirrors 000790 1789050302.564413 fallocate 0x01a850 thru 0x01b260 (0xa11 bytes) 000791 1789050302.653794 mapwrite 0x0263f7 thru 0x02f043 (0x8c4d bytes) 000792 1789050302.838268 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f044): Mirror 7: 0x89db76eb (stale) Mirror 10: 0x89db76eb (stale) Mirror 11: 0x89db76eb (stale) Mirror 12: 0x89db76eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f044): Mirror 7: 0x89db76eb (stale) Mirror 10: 0x89db76eb (stale) Mirror 11: 0x89db76eb (stale) Mirror 12: 0x89db76eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f044: skipped 000793 1789050303.410182 punch from 0x003f4d to 0x00bdfc (0x7eb0 bytes) 000794 1789050303.416447 mapread 0x001761 thru 0x006395 (0x4c35 bytes) 000796 1789050303.440540 read 0x02d635 thru 0x02dfe7 (0x9b3 bytes) 000797 1789050303.446579 punch from 0x01a885 to 0x01dc7a (0x33f6 bytes) 000798 1789050303.453327 punch from 0x008fb1 to 0x009ed8 (0xf28 bytes) 000799 1789050303.464123 MIRROR_RESYNC 5 mirrors 000800 1789050305.549357 trunc from 0x02f044 to 0x016393 (0x18cb0 bytes) 000801 1789050305.611012 trunc from 0x016394 to 0x02229d (0xbf0a bytes) 000802 1789050305.629476 mapwrite 0x0349b4 thru 0x036ddf (0x242c bytes) 000803 1789050305.702471 read 0x001efd thru 0x007717 (0x581b bytes) 000804 1789050305.713665 trunc from 0x036de0 to 0x01b632 (0x1b7ad bytes) 000806 1789050305.739237 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b633): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b633): Mirror 7: 0x5e2c2751 (stale) Mirror 10: 0x5e2c2751 (stale) Mirror 11: 0x5e2c2751 (stale) Mirror 12: 0x5e2c2751 (stale) CRC-32 checksum value for chunk [0, 0x1b633): Mirror 1: 0x25c2fed6 Mirror 13: 0x25c2fed6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b633: skipped 000807 1789050307.431847 read 0x013a9f thru 0x01b632 (0x7b94 bytes) 000810 1789050307.450938 write 0x02900c thru 0x02e4ff (0x54f4 bytes) 000811 1789050307.496615 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e500): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e500): Mirror 7: 0x17bab76b (stale) Mirror 10: 0x17bab76b (stale) Mirror 11: 0x17bab76b (stale) Mirror 12: 0x17bab76b (stale) Mirror 13: 0x83d80a43 (stale) CRC-32 checksum value for chunk [0, 0x2e500): Mirror 1: 0xdb5d4471 Mirror 14: 0xdb5d4471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e500: skipped 000812 1789050309.059455 trunc from 0x02e500 to 0x03ffde (0x11adf bytes) 000814 1789050309.147870 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ffdf): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffdf): Mirror 7: 0x2e54e36d (stale) Mirror 10: 0x2e54e36d (stale) Mirror 11: 0x2e54e36d (stale) Mirror 12: 0x2e54e36d (stale) Mirror 13: 0xf8dff1da (stale) Mirror 14: 0xc4cf6fed (stale) CRC-32 checksum value for chunk [0, 0x3ffdf): Mirror 1: 0xc4cf6fed Mirror 15: 0xc4cf6fed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffdf: skipped 000815 1789050311.162454 punch from 0x0208d3 to 0x02e0ef (0xd81d bytes) 000816 1789050311.214441 write 0x0366a5 thru 0x03ffff (0x995b bytes) 000817 1789050311.231826 mapread 0x022737 thru 0x02d681 (0xaf4b bytes) 000818 1789050311.282387 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa49e4826 (stale) Mirror 10: 0xa49e4826 (stale) Mirror 12: 0xa49e4826 (stale) Mirror 13: 0x62d5bbd7 (stale) Mirror 14: 0xb8522a64 (stale) Mirror 15: 0xb8522a64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa49e4826 (stale) Mirror 10: 0xa49e4826 (stale) Mirror 12: 0xa49e4826 (stale) Mirror 13: 0x62d5bbd7 (stale) Mirror 14: 0xb8522a64 (stale) Mirror 15: 0xb8522a64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000819 1789050312.234537 trunc from 0x040000 to 0x02b267 (0x14d98 bytes) 000820 1789050312.266841 fallocate 0x02cc72 thru 0x039daa (0xd139 bytes) 000822 1789050312.284445 punch from 0x02e275 to 0x03766c (0x93f8 bytes) 000823 1789050312.295359 punch from 0x022772 to 0x02f1a4 (0xca33 bytes) 000824 1789050312.308307 trunc from 0x0199e3 to 0x01fb0b (0x6129 bytes) 000825 1789050312.314600 mapread 0x00724f thru 0x00c317 (0x50c9 bytes) 000826 1789050312.369197 read 0x01bdd4 thru 0x02439a (0x85c7 bytes) 000827 1789050312.400060 punch from 0x011ce4 to 0x01d1ba (0xb4d7 bytes) 000828 1789050312.417485 read 0x00ff09 thru 0x01b8a2 (0xb99a bytes) 000829 1789050312.444512 punch from 0x006294 to 0x0119e1 (0xb74e bytes) 000830 1789050312.452744 mapread 0x02dbbf thru 0x039daa (0xc1ec bytes) 000831 1789050312.454082 mapread 0x02e733 thru 0x030b5c (0x242a bytes) 000832 1789050312.457099 mapread 0x026c7d thru 0x02fac5 (0x8e49 bytes) 000833 1789050312.458480 read 0x02ddbe thru 0x038c70 (0xaeb3 bytes) 000834 1789050312.467785 read 0x0296b6 thru 0x02a0fe (0xa49 bytes) 000835 1789050312.468390 fallocate 0x023dac thru 0x02817b (0x43d0 bytes) 000836 1789050312.478258 fallocate 0x02f49f thru 0x03a1e1 (0xad43 bytes) 000837 1789050312.483914 mapwrite 0x02fa1b thru 0x036cad (0x7293 bytes) 000838 1789050312.520486 write 0x008251 thru 0x013061 (0xae11 bytes) 000839 1789050312.543060 read 0x0314e8 thru 0x039daa (0x88c3 bytes) 000840 1789050312.558913 MIRROR_RESYNC 7 mirrors 000841 1789050315.758053 trunc from 0x026764 to 0x0360d6 (0xf973 bytes) 000842 1789050315.795269 trunc from 0x01ab35 to 0x01d64a (0x2b16 bytes) 000843 1789050315.802018 write 0x02df5a thru 0x0327a9 (0x4850 bytes) 000844 1789050315.814837 punch from 0x00e66f to 0x01ccc8 (0xe65a bytes) 000845 1789050315.822219 read 0x019f43 thru 0x023d32 (0x9df0 bytes) 000847 1789050315.835896 trunc from 0x00a913 to 0x0186b5 (0xdda3 bytes) 000848 1789050315.841611 read 0x008794 thru 0x00bce0 (0x354d bytes) 000849 1789050315.852755 write 0x03aab0 thru 0x03ffff (0x5550 bytes) 000850 1789050315.855064 punch from 0x02ae65 to 0x02cf54 (0x20f0 bytes) 000851 1789050315.859710 trunc from 0x040000 to 0x0012c8 (0x3ed37 bytes) 000853 1789050315.879955 punch from 0x0008bb to 0x0012c8 (0xa0e bytes) 000854 1789050315.887091 write 0x029d61 thru 0x02e93e (0x4bde bytes) 000855 1789050315.890264 punch from 0x02760d to 0x02a75b (0x314f bytes) 000856 1789050315.907608 write 0x00bf16 thru 0x015f97 (0xa082 bytes) 000857 1789050315.920155 fallocate 0x009540 thru 0x012fa5 (0x9a66 bytes) 000858 1789050315.925540 trunc from 0x02e93f to 0x028d3a (0x5c04 bytes) 000859 1789050315.943406 trunc from 0x0117a5 to 0x013008 (0x1864 bytes) 000860 1789050315.958170 punch from 0x01b7e4 to 0x023820 (0x803d bytes) 000863 1789050315.965298 fallocate 0x014451 thru 0x021b20 (0xd6d0 bytes) 000864 1789050315.971273 read 0x023804 thru 0x028d3a (0x5537 bytes) 000865 1789050315.985437 write 0x01e07a thru 0x02b244 (0xd1cb bytes) 000866 1789050315.991399 punch from 0x02af42 to 0x02b244 (0x303 bytes) 000867 1789050316.012168 mapread 0x009ab3 thru 0x00e375 (0x48c3 bytes) 000868 1789050316.013335 read 0x005c6e thru 0x005e55 (0x1e8 bytes) 000869 1789050316.014799 punch from 0x008e43 to 0x00a23a (0x13f8 bytes) 000870 1789050316.019313 mapwrite 0x01e927 thru 0x02d00f (0xe6e9 bytes) 000871 1789050316.115218 mapwrite 0x01dcd2 thru 0x02cf13 (0xf242 bytes) 000872 1789050316.165626 mapread 0x01e03d thru 0x01eb4a (0xb0e bytes) 000873 1789050316.169102 fallocate 0x021a40 thru 0x02d7d8 (0xbd99 bytes) 000874 1789050316.175540 mapwrite 0x03bcf9 thru 0x03ffff (0x4307 bytes) 000875 1789050316.229173 write 0x021d64 thru 0x0290eb (0x7388 bytes) 000876 1789050316.233134 mapwrite 0x01d752 thru 0x023c3a (0x64e9 bytes) 000877 1789050316.260420 mapwrite 0x00a527 thru 0x0147d3 (0xa2ad bytes) 000878 1789050316.299001 mapwrite 0x00422d thru 0x00c9a3 (0x8777 bytes) 000879 1789050316.329385 trunc from 0x040000 to 0x01a5f3 (0x25a0c bytes) 000880 1789050316.358458 fallocate 0x02c88f thru 0x02f673 (0x2de5 bytes) 000881 1789050316.375098 fallocate 0x00a01c thru 0x00f2c6 (0x52ab bytes) 000882 1789050316.396620 trunc from 0x01a5f4 to 0x02abf6 (0x10603 bytes) 000883 1789050316.444083 trunc from 0x00672f to 0x00c8b7 (0x6189 bytes) 000884 1789050316.464306 MIRROR_RESYNC 7 mirrors 000886 1789050318.934977 fallocate 0x023376 thru 0x030afa (0xd785 bytes) 000887 1789050318.979766 mapread 0x01e526 thru 0x02d0b6 (0xeb91 bytes) 000888 1789050319.021264 trunc from 0x02ec4d to 0x030afa (0x1eae bytes) 000889 1789050319.030061 write 0x0068f9 thru 0x0096eb (0x2df3 bytes) 000890 1789050319.044714 read 0x00802c thru 0x009db5 (0x1d8a bytes) 000891 1789050319.045775 punch from 0x017dbc to 0x01d0a9 (0x52ee bytes) 000892 1789050319.051767 mapread 0x01477a thru 0x01ae4c (0x66d3 bytes) 000893 1789050319.063162 punch from 0x017b6b to 0x0234bf (0xb955 bytes) 000894 1789050319.069336 read 0x002e0f thru 0x00960f (0x6801 bytes) 000895 1789050319.089066 write 0x016248 thru 0x01f641 (0x93fa bytes) 000896 1789050319.092392 punch from 0x008a88 to 0x010de2 (0x835b bytes) 000897 1789050319.116556 mapread 0x02ed9e thru 0x030afa (0x1d5d bytes) 000899 1789050319.118971 mapwrite 0x03bb46 thru 0x03ffff (0x44ba bytes) 000900 1789050319.178422 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x89e3b3ae (stale) Mirror 10: 0x89e3b3ae (stale) Mirror 12: 0x89e3b3ae (stale) Mirror 14: 0x89e3b3ae (stale) Mirror 15: 0x89e3b3ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x89e3b3ae (stale) Mirror 10: 0x89e3b3ae (stale) Mirror 12: 0x89e3b3ae (stale) Mirror 14: 0x89e3b3ae (stale) Mirror 15: 0x89e3b3ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1789050319.926096 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x89e3b3ae (stale) Mirror 12: 0x89e3b3ae (stale) Mirror 14: 0x89e3b3ae (stale) Mirror 15: 0x89e3b3ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x89e3b3ae (stale) Mirror 12: 0x89e3b3ae (stale) Mirror 14: 0x89e3b3ae (stale) Mirror 15: 0x89e3b3ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000903 1789050320.465895 write 0x006d09 thru 0x00c8d0 (0x5bc8 bytes) 000904 1789050320.493176 mapread 0x01c00f thru 0x0215ea (0x55dc bytes) 000905 1789050320.527383 mapwrite 0x023218 thru 0x02499d (0x1786 bytes) 000906 1789050320.586169 MIRROR_RESYNC 5 mirrors 000908 1789050322.721543 trunc from 0x017a6f to 0x021dad (0xa33f bytes) 000910 1789050322.757714 write 0x02e019 thru 0x036ed4 (0x8ebc bytes) 000911 1789050322.769337 mapread 0x01d25f thru 0x02afbb (0xdd5d bytes) 000912 1789050322.804912 fallocate 0x00f79b thru 0x010550 (0xdb6 bytes) 000913 1789050322.809950 write 0x013014 thru 0x01e074 (0xb061 bytes) 000914 1789050322.815737 mapread 0x0277ef thru 0x02bf25 (0x4737 bytes) 000915 1789050322.816804 punch from 0x024aad to 0x02ccdd (0x8231 bytes) 000916 1789050322.823094 trunc from 0x028e76 to 0x032375 (0x9500 bytes) 000917 1789050322.840260 fallocate 0x0048da thru 0x014377 (0xfa9e bytes) 000918 1789050322.846234 punch from 0x01c4b0 to 0x0201b8 (0x3d09 bytes) 000919 1789050322.852139 fallocate 0x010b84 thru 0x01c0d2 (0xb54f bytes) 000920 1789050322.856860 read 0x02622b thru 0x033214 (0xcfea bytes) 000921 1789050322.870241 mapwrite 0x028f46 thru 0x037d3e (0xedf9 bytes) 000922 1789050322.900395 read 0x03877e thru 0x03d493 (0x4d16 bytes) 000923 1789050322.901268 punch from 0x03e89e to 0x03ffff (0x1762 bytes) 000924 1789050322.907168 mapread 0x00a402 thru 0x0183f1 (0xdff0 bytes) 000925 1789050322.918025 punch from 0x00defe to 0x01a5d9 (0xc6dc bytes) 000926 1789050322.924653 read 0x0139e7 thru 0x01ca26 (0x9040 bytes) 000927 1789050322.935310 fallocate 0x00125e thru 0x00472d (0x34d0 bytes) 000928 1789050322.941280 read 0x03414f thru 0x03993a (0x57ec bytes) 000929 1789050322.942268 trunc from 0x040000 to 0x0301fd (0xfe02 bytes) 000930 1789050322.966117 write 0x00e5aa thru 0x01b44c (0xcea3 bytes) 000931 1789050322.969315 fallocate 0x001a98 thru 0x003785 (0x1cee bytes) 000932 1789050322.974675 trunc from 0x0301fe to 0x03cbdf (0xc9e2 bytes) 000933 1789050323.021117 write 0x03c0f7 thru 0x03f026 (0x2f30 bytes) 000934 1789050323.050975 MIRROR_RESYNC 5 mirrors 000935 1789050325.435993 trunc from 0x03f027 to 0x006eed (0x38139 bytes) 000936 1789050325.486834 trunc from 0x003f8c to 0x006eed (0x2f62 bytes) 000937 1789050325.495300 read 0x000a6d thru 0x00150a (0xa9e bytes) 000938 1789050325.503273 write 0x003579 thru 0x00bc6a (0x86f2 bytes) 000939 1789050325.516196 read 0x00b79e thru 0x00bc6a (0x4cd bytes) 000940 1789050325.517446 mapwrite 0x03875c thru 0x03c5b9 (0x3e5e bytes) 000941 1789050325.589525 mapread 0x01e28e thru 0x021b60 (0x38d3 bytes) 000942 1789050325.601211 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c5ba): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5ba): Mirror 7: 0x56d8addf (stale) Mirror 12: 0x56d8addf (stale) Mirror 14: 0x56d8addf (stale) Mirror 15: 0x56d8addf (stale) CRC-32 checksum value for chunk [0, 0x3c5ba): Mirror 1: 0x9c5d74a2 Mirror 16: 0x9c5d74a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5ba: skipped 000943 1789050326.807919 mapread 0x01d5af thru 0x02bbfa (0xe64c bytes) 000944 1789050326.840040 mapread 0x03228d thru 0x0372d5 (0x5049 bytes) 000945 1789050326.847624 read 0x035173 thru 0x03c5b9 (0x7447 bytes) 000946 1789050326.852402 trunc from 0x01d5f5 to 0x021702 (0x410e bytes) 000947 1789050326.901245 fallocate 0x03cdd1 thru 0x03ffff (0x322f bytes) 000948 1789050326.907994 mapwrite 0x03c64a thru 0x03ffff (0x39b6 bytes) 000950 1789050326.934655 mapread 0x02b87d thru 0x0366c8 (0xae4c bytes) 000951 1789050326.955461 trunc from 0x040000 to 0x038dc8 (0x7237 bytes) 000952 1789050326.988174 write 0x02d28c thru 0x030654 (0x33c9 bytes) 000953 1789050326.989269 trunc from 0x038dc9 to 0x021926 (0x174a2 bytes) 000954 1789050327.012496 trunc from 0x0028b9 to 0x00e435 (0xbb7d bytes) 000955 1789050327.024068 MIRROR_RESYNC 6 mirrors 000956 1789050328.474882 write 0x01b045 thru 0x0291b4 (0xe170 bytes) 000957 1789050328.527886 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x291b5): Mirror 7: 0x74194591 (stale) Mirror 14: 0x74194591 (stale) Mirror 15: 0x74194591 (stale) Mirror 16: 0x74194591 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x291b5): Mirror 7: 0x74194591 (stale) Mirror 14: 0x74194591 (stale) Mirror 15: 0x74194591 (stale) Mirror 16: 0x74194591 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x291b5: skipped 000958 1789050329.223555 mapwrite 0x02d2ef thru 0x02fe69 (0x2b7b bytes) 000959 1789050329.306217 trunc from 0x02fe6a to 0x03414a (0x42e1 bytes) 000961 1789050329.351253 mapread 0x02b0c3 thru 0x03414a (0x9088 bytes) 000962 1789050329.378982 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3414b): Mirror 7: 0x90611bea (stale) Mirror 15: 0x90611bea (stale) Mirror 16: 0x90611bea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3414b): Mirror 7: 0x90611bea (stale) Mirror 15: 0x90611bea (stale) Mirror 16: 0x90611bea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3414b: skipped 000964 1789050329.828523 punch from 0x025701 to 0x026084 (0x984 bytes) 000965 1789050329.836402 write 0x01841e thru 0x027201 (0xede4 bytes) 000966 1789050329.857551 mapwrite 0x03ae67 thru 0x03ffff (0x5199 bytes) 000967 1789050329.917359 punch from 0x01416e to 0x019823 (0x56b6 bytes) 000968 1789050329.924813 mapwrite 0x017378 thru 0x0257a6 (0xe42f bytes) 000969 1789050329.968311 trunc from 0x02e9dd to 0x03ad30 (0xc354 bytes) 000970 1789050329.987053 trunc from 0x040000 to 0x009561 (0x36a9e bytes) 000971 1789050330.044713 fallocate 0x017e6d thru 0x0199b9 (0x1b4d bytes) 000972 1789050330.059426 punch from 0x0031a2 to 0x009561 (0x63c0 bytes) 000973 1789050330.067114 fallocate 0x00001f thru 0x006250 (0x6232 bytes) 000974 1789050330.082406 fallocate 0x039a0a thru 0x03ffff (0x65f6 bytes) 000975 1789050330.093331 write 0x016eb7 thru 0x01abff (0x3d49 bytes) 000977 1789050330.113334 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ac00): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac00): Mirror 7: 0xf99f300f (stale) Mirror 15: 0xf99f300f (stale) Mirror 16: 0xf99f300f (stale) CRC-32 checksum value for chunk [0, 0x1ac00): Mirror 1: 0xa96d0567 Mirror 17: 0xa96d0567 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac00: skipped 000978 1789050331.478646 MIRROR_RESYNC 5 mirrors 000979 1789050332.577422 write 0x03527a thru 0x03ffff (0xad86 bytes) 000980 1789050332.623280 read 0x014d94 thru 0x023fc7 (0xf234 bytes) 000982 1789050332.632512 fallocate 0x013aad thru 0x017a61 (0x3fb5 bytes) 000983 1789050332.638250 punch from 0x00d3e5 to 0x00d8e5 (0x501 bytes) 000984 1789050332.642506 mapread 0x01753f thru 0x026549 (0xf00b bytes) 000985 1789050332.662394 mapwrite 0x02ba69 thru 0x03ada1 (0xf339 bytes) 000986 1789050332.690181 fallocate 0x0017e2 thru 0x00723e (0x5a5d bytes) 000987 1789050332.695123 write 0x012a04 thru 0x020933 (0xdf30 bytes) 000988 1789050332.702887 fallocate 0x002fdf thru 0x010aee (0xdb10 bytes) 000990 1789050332.726501 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdd4ee64f (stale) Mirror 15: 0xdd4ee64f (stale) Mirror 16: 0xdd4ee64f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdd4ee64f (stale) Mirror 15: 0xdd4ee64f (stale) Mirror 16: 0xdd4ee64f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1789050333.352502 punch from 0x006680 to 0x012729 (0xc0aa bytes) 000992 1789050333.366683 read 0x014a3b thru 0x01b583 (0x6b49 bytes) 000993 1789050333.388918 mapwrite 0x036f26 thru 0x037666 (0x741 bytes) 000994 1789050333.434285 write 0x01e852 thru 0x027fa3 (0x9752 bytes) 000995 1789050333.451082 read 0x02d0f3 thru 0x030aa3 (0x39b1 bytes) 000996 1789050333.468811 trunc from 0x025bf2 to 0x02ce14 (0x7223 bytes) 000998 1789050333.498461 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdd4ee64f (stale) Mirror 15: 0xdd4ee64f (stale) Mirror 16: 0xdd4ee64f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d87d8b1 Mirror 17: 0x2d87d8b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000999 1789050334.836276 read 0x02c406 thru 0x03be60 (0xfa5b bytes) 001000 1789050334.862402 write 0x004747 thru 0x00c70b (0x7fc5 bytes) All operations completed A-OK! PASS 70a (128s) == sanity-flr test 71: check mirror extend parent fid ==== 10:25:41 (1789050341) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1b2:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 10:25:53 (1789050353) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263011 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204812 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 10:26:04 (1789050364) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 929184 .. Extending file size to 6426784 .. done Extending file size to 8072192 .. Extending file size to 6536800 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7267328 .. Extending file size to 7318080 .. Extending file size to 8192800 .. Extending file size to 8194208 .. Extending file size to 8700896 .. Extending file size to 8659936 .. Extending file size to 9138880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8925024 .. done Extending file size to 9083424 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9116800 .. Extending file size to 9254560 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61359 61360 61362 61364 61366 done 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:27:27 (1789050447) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 717632 .. Extending file size to 5370144 .. Extending file size to 7358720 .. Extending file size to 5698048 .. Extending file size to 6932992 .. Extending file size to 8320064 .. Extending file size to 8490624 .. Extending file size to 7184512 .. Extending file size to 8250112 .. Extending file size to 8664480 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8651424 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8890560 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8671744 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8823904 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8993472 .. Waiting 63105 63106 63107 63109 63110 done 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:28:53 (1789050533) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2172 1269132 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24076 3582944 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18604 3588416 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42680 7171360 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2172 1269132 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24076 3582944 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18604 3588416 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42680 7171360 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0968369 s, 21.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198936 s, 10.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x942dbe17 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:29:15 (1789050555) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0884072 s, 23.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 10:29:24 (1789050564) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:29:32 (1789050572) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139823 s, 15.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:29:40 (1789050580) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106859 s, 19.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:29:50 (1789050590) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140095 s, 15.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:30:26 (1789050626) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540672 s, 19.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x220:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:31:20 (1789050680) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236446 s, 8.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:32:15 (1789050735) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg447-server: Pool lustre.test_204f created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21066 s, 10.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg447-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:33:41 (1789050821) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:33:49 (1789050829) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:33:58 (1789050838) oleg447-server: Pool lustre.test_206 created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg447-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:34:29 (1789050869) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740874 s, 14.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456209 s, 23.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7a8fc0d8 Mirror 3: 0x7a8fc0d8 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:34:41 (1789050881) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:34:46 (1789050886) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:34:49 (1789050889) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0839344 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1bf0000 Read this (4 bytes): BBBB 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:35:31 (1789050931) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0361978 s, 226 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:35:40 (1789050940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932874 s, 11.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 10:35:41.000000000 -0400 Modify: 2026-09-10 10:35:42.000000000 -0400 Change: 2026-09-10 10:35:43.779000000 -0400 Birth: 2026-09-10 10:35:42.336000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:35:49 (1789050949) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353677 s, 29.6 MB/s oleg447-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg447-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg447-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:36:15 (1789050975) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.322834 s, 127 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0520021 s, 78.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 10:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 10:36 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:36:24 (1789050984) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:36:32 (1789050992) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1999 sec =========== 10:36:42 (1789051002) === sanity-flr: start cleanup 10:36:43 (1789051003) === === sanity-flr: finish cleanup 10:36:53 (1789051013) ===