-----============= acceptance-small: sanity-flr ============----- Mon Aug 31 11:13:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg127-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:14:26 (1788189266) === oleg127-client.virtnet: executing check_config_client /mnt/lustre oleg127-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg127-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97dcc3209800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97dcc3209800.idle_timeout=debug disable quota as required oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:15:10 (1788189310) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8264] == sanity-flr test 0a: lfs mirror create with -N option == 11:15:13 (1788189313) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:15:28 (1788189328) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:15:32 (1788189332) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:15:35 (1788189335) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:15:59 (1788189359) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:16:02 (1788189362) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 11:16:05 (1788189365) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97dcc3209800.stats=clear osc.lustre-OST0001-osc-ffff97dcc3209800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:16:14 (1788189374) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:16:25 (1788189385) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:16:35 (1788189395) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:16:41 (1788189401) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:16:51 (1788189411) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 11:17:03 (1788189423) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:17:10 (1788189430) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:17:17 (1788189437) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:17:25 (1788189445) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:17:37 (1788189457) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:17:45 (1788189465) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.903295 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732023 s, 14.3 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:17:55 (1788189475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433105 s, 24.2 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:18:03 (1788189483) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:18:16 (1788189496) Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (91s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:19:47 (1788189587) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.65786 s, 15.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:20:36 (1788189636) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:22:03 (1788189723) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.7059 s, 19.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 11:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.624s user 0m0.000s sys 0m0.138s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.102s user 0m0.000s sys 0m0.139s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.561s user 0m0.000s sys 0m0.159s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97dcc3209800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:23:45 (1788189825) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:23:50 (1788189830) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:23:53 (1788189833) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 11:23:57 (1788189837) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 11:24:07 (1788189847) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.494271 s, 25.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:24:18 (1788189858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380842 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241215 s, 43.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:24:25 (1788189865) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:24:34 (1788189874) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000806192 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00154853 s, 661 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000838297 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158171 s, 647 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 11:24:45 (1788189885) create test files with size 15743423 .. 3766e36f90048f3e992bbff0138f6ff6 - 2024e966b5bf278b447dcfbac2ebf01f - 770e9a935598ac911d78644472325e2f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 11:25:11 (1788189911) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:25:25 (1788189925) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:25:32 (1788189932) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12285 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347497 s, 30.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 11:25:41 (1788189941) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920074 s, 22.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.281131 s, 14.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236171 s, 8.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 11:25:57 (1788189957) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 11:26:02 (1788189962) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:26:06 (1788189966) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269457 s, 38.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 11:26:13 (1788189973) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.32807 s, 9.6 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:26:25 (1788189985) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:26:36 (1788189996) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.298265 s, 35.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:26:47 (1788190007) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff97dcc933a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.370829 s, 28.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff97dcc845b800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:27:06 (1788190026) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:27:49 (1788190069) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:28:32 (1788190112) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:28:35 (1788190115) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201405 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.292943 s, 10.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5906f6fc (nosync,stale) Mirror 2: 0x5906f6fc (nosync,stale) ** verify mirror 2 doesn't change original checksum: e98836954f4770e75e2f3bec70b7eb3f - mirror 1 checksum: 0cb91dd9654b4fe1994bdc88992baed3 - mirror 2 checksum: e98836954f4770e75e2f3bec70b7eb3f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T11:28:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T11:28:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e98836954f4770e75e2f3bec70b7eb3f - mirror 1 checksum: 0cb91dd9654b4fe1994bdc88992baed3 - mirror 2 checksum: 0cb91dd9654b4fe1994bdc88992baed3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T11:28:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T11:28:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:28:46 (1788190126) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314756 s, 33.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:28:55 (1788190135) Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:29:06 (1788190146) shut down facet: mds1 facet_host: oleg127-server facet_failover_host: oleg127-server Failover mds1 to oleg127-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:29:40 (1788190180) targets are mounted 11:29:40 (1788190180) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227237 s, 46.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:30:01 (1788190201) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0252728 s, 810 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0101213 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:30:10 (1788190210) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0641355 s, 319 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0191407 s, 1.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:30:20 (1788190220) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00700841 s, 2.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:30:28 (1788190228) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0866984 s, 236 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:30:37 (1788190237) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00937162 s, 437 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.043401 s, 94.4 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:30:46 (1788190246) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (31s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:31:17 (1788190277) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:31:45 (1788190305) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:32:09 (1788190329) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:32:36 (1788190356) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:32:44 (1788190364) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57609 mirror split pid 57610 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57609 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57610 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 11:33:52 (1788190432) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 434 skipping zero size read skipping zero length punch hole truncating to largest ever: 0xd592 000003 1788190434.933376 trunc from 00000000 to 0x00d591 (0xd592 bytes) 000004 1788190434.957962 mapread 0x008f81 thru 0x00d591 (0x4611 bytes) 000005 1788190434.980278 trunc from 0x000f4c to 0x00cba4 (0xbc59 bytes) 000006 1788190434.986317 read 0x003597 thru 0x00d591 (0x9ffb bytes) 000007 1788190435.000933 trunc from 0x00b932 to 0x00d591 (0x1c60 bytes) fallocating to largest ever: 0x2d96e 000009 1788190435.011789 fallocate 0x028297 thru 0x02d96d (0x56d7 bytes) 000010 1788190435.020930 write 0x016281 thru 0x0246c3 (0xe443 bytes) 000011 1788190435.045268 trunc from 0x02d96e to 0x0196ea (0x14283 bytes) truncating to largest ever: 0x3520a 000012 1788190435.059234 trunc from 0x0196eb to 0x035209 (0x1bb1f bytes) 000013 1788190435.073417 fallocate 0x022357 thru 0x02e947 (0xc5f1 bytes) 000015 1788190435.082491 mapread 0x0150d2 thru 0x015b14 (0xa43 bytes) 000016 1788190435.091896 write 0x039fd2 thru 0x03ffff (0x602e bytes) 000017 1788190435.093966 trunc from 0x0059b0 to 0x008fed (0x363e bytes) 000018 1788190435.098351 mapwrite 0x025a6b thru 0x034cd2 (0xf268 bytes) 000019 1788190435.140491 fallocate 0x00f9f2 thru 0x014b84 (0x5193 bytes) 000020 1788190435.145578 write 0x0155d6 thru 0x021eac (0xc8d7 bytes) fallocating to largest ever: 0x3bb0c 000021 1788190435.147880 fallocate 0x030da7 thru 0x03bb0b (0xad65 bytes) 000022 1788190435.152731 fallocate 0x02b50e thru 0x02c1fc (0xcef bytes) 000023 1788190435.157016 mapwrite 0x030db3 thru 0x0329da (0x1c28 bytes) 000024 1788190435.177065 fallocate 0x016d56 thru 0x01fc56 (0x8f01 bytes) 000026 1788190435.182876 read 0x037acb thru 0x03ffff (0x8535 bytes) 000027 1788190435.193072 punch from 0x024374 to 0x031d4b (0xd9d8 bytes) 000029 1788190435.199149 trunc from 0x040000 to 0x018d00 (0x272ff bytes) 000030 1788190435.220838 mapread 0x00acaf thru 0x018d00 (0xe052 bytes) 000031 1788190435.222378 trunc from 0x009d93 to 0x017b0d (0xdd7b bytes) fallocating to largest ever: 0x40000 000033 1788190435.231601 fallocate 0x03bc97 thru 0x03ffff (0x4369 bytes) 000034 1788190435.238890 mapread 0x009f9b thru 0x012ef1 (0x8f57 bytes) 000035 1788190435.295250 mapread 0x009c3e thru 0x0145c6 (0xa989 bytes) 000036 1788190435.307951 trunc from 0x00bcd4 to 0x015846 (0x9b73 bytes) 000037 1788190435.312918 write 0x008177 thru 0x00d1ed (0x5077 bytes) 000038 1788190435.318950 mapwrite 0x02176c thru 0x030d46 (0xf5db bytes) 000039 1788190435.486575 trunc from 0x0305b7 to 0x030d46 (0x790 bytes) 000040 1788190435.494381 punch from 0x02e51d to 0x030d46 (0x282a bytes) 000041 1788190435.502925 mapread 0x02f522 thru 0x030d46 (0x1825 bytes) 000042 1788190435.517988 trunc from 0x027bd9 to 0x030d46 (0x916e bytes) 000043 1788190435.525034 punch from 0x02a531 to 0x02e5db (0x40ab bytes) 000044 1788190435.530946 trunc from 0x02269a to 0x02ca1d (0xa384 bytes) 000045 1788190435.540221 punch from 0x0286a9 to 0x030d46 (0x869e bytes) 000046 1788190435.547206 fallocate 0x012323 thru 0x01759e (0x527c bytes) 000047 1788190435.553359 write 0x026b43 thru 0x033450 (0xc90e bytes) 000049 1788190435.563282 write 0x03d9c3 thru 0x03fc64 (0x22a2 bytes) 000050 1788190435.565282 mapwrite 0x03eb34 thru 0x03ffff (0x14cc bytes) 000051 1788190435.598400 mapwrite 0x01b3e8 thru 0x0228ef (0x7508 bytes) 000052 1788190435.657991 read 0x03a8bf thru 0x03e681 (0x3dc3 bytes) 000053 1788190435.669798 mapread 0x03cace thru 0x03ffff (0x3532 bytes) 000054 1788190435.674193 trunc from 0x032350 to 0x03676c (0x441d bytes) 000055 1788190435.681659 read 0x00bc97 thru 0x00fdec (0x4156 bytes) 000056 1788190435.682490 mapread 0x0297a5 thru 0x037156 (0xd9b2 bytes) 000057 1788190435.699363 mapread 0x015625 thru 0x01e8ae (0x928a bytes) 000058 1788190435.702116 punch from 0x0134e2 to 0x01f437 (0xbf56 bytes) 000059 1788190435.708862 trunc from 0x040000 to 0x01475e (0x2b8a1 bytes) 000061 1788190435.729900 punch from 0x00e189 to 0x01475e (0x65d6 bytes) 000062 1788190435.737259 write 0x03f733 thru 0x03ffff (0x8cd bytes) 000063 1788190435.739687 punch from 0x0364f0 to 0x03a54f (0x4060 bytes) 000064 1788190435.744946 trunc from 0x03e097 to 0x03ffff (0x1f69 bytes) 000066 1788190435.758369 punch from 0x02d6f5 to 0x03bdec (0xe6f8 bytes) 000067 1788190435.764738 mapread 0x00d8ae thru 0x00dd77 (0x4ca bytes) 000068 1788190435.766362 fallocate 0x018500 thru 0x019a73 (0x1574 bytes) 000069 1788190435.773993 mapread 0x01f5db thru 0x02b745 (0xc16b bytes) 000070 1788190435.843675 read 0x00688e thru 0x01052a (0x9c9d bytes) 000071 1788190435.855670 trunc from 0x040000 to 0x00a89c (0x35763 bytes) 000072 1788190435.878765 read 0x007115 thru 0x008c75 (0x1b61 bytes) 000073 1788190435.880228 trunc from 0x00a89d to 0x013b12 (0x9276 bytes) 000074 1788190435.904310 punch from 0x00a59e to 0x00b6e9 (0x114c bytes) 000075 1788190435.911114 punch from 0x00a2d4 to 0x00ce7a (0x2ba7 bytes) 000076 1788190435.917834 mapwrite 0x03edc0 thru 0x03ffff (0x1240 bytes) 000077 1788190435.980908 mapwrite 0x03e6f1 thru 0x03ffff (0x190f bytes) 000079 1788190436.013157 MIRROR_EXTEND to 2 mirrors 000080 1788190436.646675 write 0x032d89 thru 0x03ea85 (0xbcfd bytes) 000081 1788190436.703419 write 0x033fb6 thru 0x035cb7 (0x1d02 bytes) 000082 1788190436.705386 mapread 0x02db38 thru 0x039f6e (0xc437 bytes) 000083 1788190436.727213 write 0x02e0b3 thru 0x0382a6 (0xa1f4 bytes) 000084 1788190436.730781 read 0x0234d9 thru 0x02ee83 (0xb9ab bytes) 000085 1788190436.745816 write 0x02d04a thru 0x02d189 (0x140 bytes) 000086 1788190436.747792 fallocate 0x0057b6 thru 0x00ecfb (0x9546 bytes) 000088 1788190436.755311 mapwrite 0x0286b2 thru 0x02e89b (0x61ea bytes) 000089 1788190436.789703 mapwrite 0x029c2f thru 0x0308bd (0x6c8f bytes) 000090 1788190436.833459 trunc from 0x040000 to 0x01cda8 (0x23257 bytes) 000091 1788190436.859126 mapread 0x00fe10 thru 0x010f0e (0x10ff bytes) 000092 1788190436.877974 punch from 0x01a7d3 to 0x01cda8 (0x25d6 bytes) 000093 1788190436.886933 trunc from 0x003f65 to 0x00664b (0x26e7 bytes) 000094 1788190436.895552 trunc from 0x01cda9 to 0x02edba (0x12012 bytes) 000095 1788190436.911477 mapread 0x0227dc thru 0x02cb0c (0xa331 bytes) 000096 1788190436.943870 punch from 0x0247ed to 0x02edba (0xa5ce bytes) 000097 1788190436.952802 read 0x00b893 thru 0x011d32 (0x64a0 bytes) 000098 1788190436.953596 mapread 0x001aa8 thru 0x0067da (0x4d33 bytes) 000099 1788190436.989575 mapwrite 0x01160b thru 0x01db00 (0xc4f6 bytes) 000100 1788190437.020894 mapwrite 0x028916 thru 0x031bd1 (0x92bc bytes) 000101 1788190437.085293 MIRROR_RESYNC 2 mirrors 000102 1788190437.883744 read 0x0045f8 thru 0x00aa15 (0x641e bytes) 000103 1788190437.896146 fallocate 0x01dc19 thru 0x026dac (0x9194 bytes) 000104 1788190437.946045 read 0x02d7f8 thru 0x02f042 (0x184b bytes) 000105 1788190437.956281 trunc from 0x031bd2 to 0x0317a2 (0x42f bytes) 000106 1788190438.007978 MIRROR_RESYNC 2 mirrors 000107 1788190438.757180 punch from 0x0080f9 to 0x00b370 (0x3278 bytes) 000108 1788190438.803087 mapwrite 0x009af3 thru 0x00f4e1 (0x59ef bytes) 000112 1788190438.843866 trunc from 0x0317a3 to 0x02dae0 (0x3cc2 bytes) 000114 1788190438.873239 punch from 0x000fa0 to 0x009e24 (0x8e85 bytes) 000115 1788190438.882730 trunc from 0x02dae1 to 0x025109 (0x89d7 bytes) 000116 1788190438.899005 read 0x018385 thru 0x01fa73 (0x76ef bytes) 000118 1788190438.912711 mapread 0x019282 thru 0x025109 (0xbe88 bytes) 000119 1788190438.922052 trunc from 0x02510a to 0x02bc8d (0x6b84 bytes) 000120 1788190438.946352 punch from 0x00a8a1 to 0x016282 (0xb9e2 bytes) 000122 1788190438.954196 read 0x0179da thru 0x01d999 (0x5fc0 bytes) 000123 1788190438.964418 punch from 0x00dde4 to 0x017c27 (0x9e44 bytes) 000124 1788190438.970006 fallocate 0x01e075 thru 0x02a96f (0xc8fb bytes) 000125 1788190438.974432 write 0x03942c thru 0x03ffff (0x6bd4 bytes) 000126 1788190438.976029 trunc from 0x01cb8c to 0x0263e9 (0x985e bytes) 000127 1788190438.981006 write 0x0233e0 thru 0x02aef5 (0x7b16 bytes) 000128 1788190438.999085 write 0x01d169 thru 0x0231dc (0x6074 bytes) 000129 1788190439.005890 mapwrite 0x005853 thru 0x00e4f8 (0x8ca6 bytes) 000130 1788190439.040299 fallocate 0x02efda thru 0x03d3c0 (0xe3e7 bytes) 000131 1788190439.045729 fallocate 0x003c59 thru 0x0111ea (0xd592 bytes) 000132 1788190439.051231 fallocate 0x0391ef thru 0x03ffff (0x6e11 bytes) 000133 1788190439.055787 fallocate 0x017c73 thru 0x01c3d4 (0x4762 bytes) 000134 1788190439.060450 write 0x031d8b thru 0x031ff5 (0x26b bytes) 000135 1788190439.061501 trunc from 0x015326 to 0x01d6a8 (0x8383 bytes) 000136 1788190439.067268 trunc from 0x025201 to 0x026afd (0x18fd bytes) 000137 1788190439.072758 mapread 0x00e942 thru 0x012864 (0x3f23 bytes) 000138 1788190439.091061 fallocate 0x0055d5 thru 0x01314e (0xdb7a bytes) 000139 1788190439.096122 trunc from 0x040000 to 0x03d328 (0x2cd7 bytes) 000140 1788190439.110205 trunc from 0x02f237 to 0x03bdec (0xcbb6 bytes) 000143 1788190439.123828 mapwrite 0x032c71 thru 0x039839 (0x6bc9 bytes) 000144 1788190439.182616 write 0x0315c7 thru 0x03ac91 (0x96cb bytes) 000145 1788190439.197145 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d329): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d329): Mirror 2: 0x16d290a3 (stale) CRC-32 checksum value for chunk [0, 0x3d329): Mirror 1: 0xb425cecf Mirror 3: 0xb425cecf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d329: skipped 000146 1788190439.975895 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000147 1788190440.350692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000149 1788190440.515056 read 0x02b3a0 thru 0x031415 (0x6076 bytes) 000150 1788190440.526463 mapwrite 0x02b9ca thru 0x037e78 (0xc4af bytes) 000153 1788190440.611137 mapwrite 0x007ce8 thru 0x012a8e (0xada7 bytes) 000154 1788190440.651968 write 0x005f59 thru 0x013929 (0xd9d1 bytes) 000155 1788190440.659315 write 0x00d2dc thru 0x019435 (0xc15a bytes) 000156 1788190440.660833 mapread 0x010279 thru 0x012802 (0x258a bytes) 000158 1788190440.666649 fallocate 0x00cd5d thru 0x01662b (0x98cf bytes) 000159 1788190440.673445 write 0x0140f0 thru 0x01ba7d (0x798e bytes) 000161 1788190440.681910 mapwrite 0x03b8eb thru 0x03ffff (0x4715 bytes) 000162 1788190440.760931 trunc from 0x019490 to 0x01d011 (0x3b82 bytes) 000163 1788190440.767641 punch from 0x02b30c to 0x0337ca (0x84bf bytes) 000164 1788190440.774109 trunc from 0x03edde to 0x03ffff (0x1222 bytes) 000165 1788190440.782004 mapread 0x030363 thru 0x0317a8 (0x1446 bytes) 000166 1788190440.794013 trunc from 0x020b60 to 0x02de2a (0xd2cb bytes) 000167 1788190440.800906 write 0x0367a1 thru 0x03aa33 (0x4293 bytes) 000168 1788190440.802728 write 0x0009e4 thru 0x008075 (0x7692 bytes) 000169 1788190440.809928 fallocate 0x00c1f4 thru 0x01621d (0xa02a bytes) 000170 1788190440.816198 fallocate 0x03f8e2 thru 0x03ffff (0x71e bytes) 000171 1788190440.823085 trunc from 0x040000 to 0x02eaed (0x11512 bytes) 000173 1788190440.839152 trunc from 0x0203c7 to 0x024a4d (0x4687 bytes) 000175 1788190440.851905 trunc from 0x0255f7 to 0x02eaed (0x94f7 bytes) 000176 1788190440.858650 trunc from 0x014c6f to 0x02448d (0xf81f bytes) 000177 1788190440.864998 fallocate 0x03dfef thru 0x03ffff (0x2011 bytes) 000178 1788190440.870725 write 0x013a1d thru 0x020738 (0xcd1c bytes) 000179 1788190440.877938 read 0x004695 thru 0x0139ed (0xf359 bytes) 000180 1788190440.878637 read 0x025f0d thru 0x02b5fa (0x56ee bytes) 000181 1788190440.895103 mapread 0x0266bc thru 0x02e77c (0x80c1 bytes) 000183 1788190440.899938 fallocate 0x02c51f thru 0x031e9c (0x597e bytes) 000184 1788190440.905798 trunc from 0x02eaee to 0x032c38 (0x414b bytes) 000185 1788190440.926183 punch from 0x0323b3 to 0x032c38 (0x886 bytes) 000186 1788190440.931412 fallocate 0x036948 thru 0x03ffff (0x96b8 bytes) 000187 1788190440.939635 trunc from 0x040000 to 0x0057e3 (0x3a81c bytes) 000188 1788190440.963438 write 0x0355eb thru 0x03b97a (0x6390 bytes) 000189 1788190440.968051 trunc from 0x03b97b to 0x029ca3 (0x11cd7 bytes) 000190 1788190440.992672 punch from 0x011754 to 0x017e11 (0x66be bytes) 000191 1788190441.000443 mapwrite 0x017f38 thru 0x01ace7 (0x2db0 bytes) 000192 1788190441.050011 write 0x00f7f2 thru 0x019dba (0xa5c9 bytes) 000193 1788190441.056541 trunc from 0x01d028 to 0x0294e0 (0xc4b9 bytes) 000194 1788190441.064617 mapwrite 0x02be5a thru 0x03a1a3 (0xe34a bytes) 000195 1788190441.205447 trunc from 0x01e408 to 0x02c953 (0xe54c bytes) 000196 1788190441.214187 punch from 0x011c77 to 0x018a49 (0x6dd3 bytes) 000197 1788190441.223732 read 0x0079dc thru 0x00a2c1 (0x28e6 bytes) 000198 1788190441.252681 write 0x013c15 thru 0x01dc29 (0xa015 bytes) 000200 1788190441.255951 fallocate 0x0017c4 thru 0x006a42 (0x527f bytes) 000201 1788190441.260931 write 0x02fc24 thru 0x03b429 (0xb806 bytes) 000202 1788190441.272904 read 0x003a9f thru 0x00f48d (0xb9ef bytes) 000203 1788190441.275012 mapwrite 0x00f44b thru 0x01d9c0 (0xe576 bytes) 000204 1788190441.328764 mapwrite 0x0195c8 thru 0x0288e7 (0xf320 bytes) 000205 1788190441.390482 mapread 0x024198 thru 0x031c86 (0xdaef bytes) 000206 1788190441.391895 mapwrite 0x01442e thru 0x0202df (0xbeb2 bytes) 000208 1788190441.466971 write 0x01a7ca thru 0x025181 (0xa9b8 bytes) 000209 1788190441.486180 MIRROR_EXTEND to 2 mirrors 000211 1788190442.740276 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000213 1788190442.900214 trunc from 0x0383b2 to 0x03b429 (0x3078 bytes) 000214 1788190442.916084 fallocate 0x01957b thru 0x01f274 (0x5cfa bytes) 000215 1788190442.923665 write 0x0313bd thru 0x039fb5 (0x8bf9 bytes) 000216 1788190442.943997 mapread 0x0210d8 thru 0x02dae9 (0xca12 bytes) 000217 1788190442.977498 trunc from 0x018f7e to 0x01e0fa (0x517d bytes) 000219 1788190442.984957 write 0x01a041 thru 0x029a93 (0xfa53 bytes) 000221 1788190442.994350 punch from 0x020d2a to 0x026729 (0x5a00 bytes) 000222 1788190443.013954 punch from 0x019cf7 to 0x01ec83 (0x4f8d bytes) 000223 1788190443.019549 read 0x0325e0 thru 0x03b429 (0x8e4a bytes) 000224 1788190443.038846 mapread 0x00610d thru 0x00d918 (0x780c bytes) 000225 1788190443.048920 trunc from 0x01db26 to 0x024841 (0x6d1c bytes) 000226 1788190443.053587 trunc from 0x03b42a to 0x006816 (0x34c13 bytes) 000227 1788190443.076225 trunc from 0x006817 to 0x013c58 (0xd442 bytes) 000228 1788190443.100995 write 0x00fbb1 thru 0x013476 (0x38c6 bytes) 000229 1788190443.126175 MIRROR_EXTEND to 2 mirrors 000230 1788190443.936993 mapread 0x00b37e thru 0x013c58 (0x88db bytes) 000231 1788190443.963837 read 0x00c63b thru 0x013a97 (0x745d bytes) 000233 1788190443.965207 read 0x010509 thru 0x013c58 (0x3750 bytes) 000234 1788190443.966097 mapread 0x00c786 thru 0x013c58 (0x74d3 bytes) 000235 1788190443.970745 mapread 0x0091a8 thru 0x009422 (0x27b bytes) 000236 1788190443.971746 trunc from 0x01124d to 0x013c58 (0x2a0c bytes) 000237 1788190444.025785 fallocate 0x0208a1 thru 0x02ae80 (0xa5e0 bytes) 000238 1788190444.033143 mapread 0x00aab8 thru 0x00e13f (0x3688 bytes) 000239 1788190444.046199 mapread 0x005279 thru 0x0123c9 (0xd151 bytes) 000240 1788190444.049085 mapread 0x010abc thru 0x013487 (0x29cc bytes) 000241 1788190444.050946 mapread 0x000dae thru 0x0071a9 (0x63fc bytes) 000243 1788190444.053994 fallocate 0x01e0a3 thru 0x022f8b (0x4ee9 bytes) 000244 1788190444.059767 punch from 0x0075d2 to 0x013b01 (0xc530 bytes) 000245 1788190444.066123 write 0x016073 thru 0x01fa0d (0x999b bytes) 000246 1788190444.084677 fallocate 0x0273c9 thru 0x033f2d (0xcb65 bytes) 000247 1788190444.090762 read 0x0074a6 thru 0x00be3c (0x4997 bytes) 000248 1788190444.105425 mapwrite 0x021572 thru 0x02c48b (0xaf1a bytes) 000249 1788190444.179928 MIRROR_RESYNC 2 mirrors 000250 1788190445.035089 read 0x01b665 thru 0x023b60 (0x84fc bytes) 000251 1788190445.053075 punch from 0x011706 to 0x013963 (0x225e bytes) 000252 1788190445.101070 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: 0xdbe3d6d9 (stale) CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 1: 0xdbe3d6d9 Mirror 3: 0xdbe3d6d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c48c: skipped 000253 1788190445.865108 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: 0xdbe3d6d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: 0xdbe3d6d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c48c: skipped 000254 1788190446.092912 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 2: 0xdbe3d6d9 (stale) CRC-32 checksum value for chunk [0, 0x2c48c): Mirror 1: 0xdbe3d6d9 Mirror 3: 0xdbe3d6d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c48c: skipped 000255 1788190447.205842 write 0x00e11d thru 0x01ab60 (0xca44 bytes) 000257 1788190447.283094 MIRROR_RESYNC 3 mirrors 000258 1788190448.289088 mapwrite 0x02165a thru 0x024579 (0x2f20 bytes) 000259 1788190448.406041 read 0x020003 thru 0x02c48b (0xc489 bytes) 000260 1788190448.413034 trunc from 0x02c48c to 0x02e647 (0x21bc bytes) 000261 1788190448.457107 read 0x0217f5 thru 0x02645e (0x4c6a bytes) 000262 1788190448.462291 write 0x0157b7 thru 0x018bbf (0x3409 bytes) 000263 1788190448.479619 write 0x000fb0 thru 0x002515 (0x1566 bytes) 000264 1788190448.481829 trunc from 0x02e648 to 0x010bbd (0x1da8a bytes) 000265 1788190448.508848 trunc from 0x008eb6 to 0x010bbd (0x7d08 bytes) 000266 1788190448.514483 punch from 0x003aa9 to 0x008da9 (0x5301 bytes) 000267 1788190448.525012 write 0x03e0b3 thru 0x03ffff (0x1f4d bytes) 000268 1788190448.526167 trunc from 0x03ab9d to 0x03ffff (0x5463 bytes) 000269 1788190448.557503 punch from 0x033c2a to 0x03732e (0x3705 bytes) 000270 1788190448.575058 mapwrite 0x0037dc thru 0x0133ee (0xfc13 bytes) 000271 1788190448.853177 punch from 0x02fa14 to 0x03cef6 (0xd4e3 bytes) 000272 1788190448.858641 fallocate 0x014a1e thru 0x017d20 (0x3303 bytes) 000273 1788190448.863779 trunc from 0x020b64 to 0x02f2af (0xe74c bytes) 000275 1788190448.871274 trunc from 0x02cf14 to 0x031cde (0x4dcb bytes) 000276 1788190448.875887 read 0x0171fd thru 0x01abee (0x39f2 bytes) 000277 1788190448.889080 mapwrite 0x007175 thru 0x0092c5 (0x2151 bytes) 000278 1788190448.908401 mapwrite 0x0083c8 thru 0x00f876 (0x74af bytes) 000279 1788190448.948337 write 0x02fa68 thru 0x03cf2e (0xd4c7 bytes) 000280 1788190448.950012 trunc from 0x040000 to 0x00578f (0x3a870 bytes) 000281 1788190448.970850 trunc from 0x005790 to 0x02400c (0x1e87d bytes) 000282 1788190448.988805 punch from 0x00a87d to 0x01a63c (0xfdc0 bytes) 000283 1788190449.007901 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2400d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2400d): Mirror 2: 0x3f160ae1 (stale) Mirror 3: 0x3f160ae1 (stale) CRC-32 checksum value for chunk [0, 0x2400d): Mirror 1: 0xdfd8880f Mirror 4: 0xdfd8880f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2400d: skipped 000284 1788190449.909063 write 0x035350 thru 0x03d8d3 (0x8584 bytes) 000285 1788190449.990762 mapwrite 0x0145e9 thru 0x015b90 (0x15a8 bytes) 000286 1788190450.036159 trunc from 0x03d8d4 to 0x0177d3 (0x26100 bytes) 000287 1788190450.086450 trunc from 0x0176b1 to 0x0177d3 (0x123 bytes) 000288 1788190450.091773 read 0x008c09 thru 0x01509f (0xc497 bytes) 000289 1788190450.108018 MIRROR_RESYNC 4 mirrors 000291 1788190450.903586 mapread 0x008f1f thru 0x00943a (0x51c bytes) 000292 1788190450.915845 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000293 1788190451.313919 mapwrite 0x001e3d thru 0x005924 (0x3ae8 bytes) 000294 1788190451.518651 mapread 0x013d68 thru 0x0177d3 (0x3a6c bytes) 000295 1788190451.537712 trunc from 0x00bfb2 to 0x014e57 (0x8ea6 bytes) 000296 1788190451.543246 mapread 0x00564e thru 0x011887 (0xc23a bytes) 000297 1788190451.551029 read 0x0156c4 thru 0x0177d3 (0x2110 bytes) 000298 1788190451.552462 write 0x035026 thru 0x03cfad (0x7f88 bytes) 000299 1788190451.557493 mapread 0x037808 thru 0x03cfad (0x57a6 bytes) 000300 1788190451.558630 fallocate 0x03334a thru 0x03f191 (0xbe48 bytes) 000302 1788190451.570989 read 0x034b63 thru 0x03ec8b (0xa129 bytes) 000303 1788190451.584835 trunc from 0x03f192 to 0x02ca3a (0x12757 bytes) 000305 1788190451.642287 fallocate 0x00e9d1 thru 0x01be2b (0xd45b bytes) 000306 1788190451.649064 punch from 0x00fd16 to 0x016320 (0x660b bytes) 000307 1788190451.654984 trunc from 0x00cd35 to 0x01a74e (0xda1a bytes) 000308 1788190451.660776 mapwrite 0x02cdf2 thru 0x030566 (0x3775 bytes) 000309 1788190451.720531 trunc from 0x030567 to 0x03cb75 (0xc60f bytes) 000311 1788190451.743038 fallocate 0x010184 thru 0x01ac79 (0xaaf6 bytes) 000314 1788190451.752070 write 0x00d4d1 thru 0x01b7f3 (0xe323 bytes) 000315 1788190451.761404 fallocate 0x03c527 thru 0x03ffff (0x3ad9 bytes) 000316 1788190451.768011 mapread 0x02185f thru 0x02af1c (0x96be bytes) 000317 1788190451.770456 trunc from 0x040000 to 0x03bed5 (0x412a bytes) 000318 1788190451.792046 trunc from 0x03bed6 to 0x018bb6 (0x2331f bytes) 000319 1788190451.815052 mapwrite 0x027a12 thru 0x030087 (0x8676 bytes) 000320 1788190451.928176 trunc from 0x030088 to 0x03f126 (0xf09f bytes) 000321 1788190451.978094 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f127): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f127): Mirror 3: 0xd8bd2b4e (stale) Mirror 4: 0xd8bd2b4e (stale) CRC-32 checksum value for chunk [0, 0x3f127): Mirror 1: 0xe667e7f8 Mirror 5: 0xe667e7f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f127: skipped 000322 1788190453.282395 trunc from 0x022a00 to 0x031bc0 (0xf1c1 bytes) 000323 1788190453.312099 read 0x01fe17 thru 0x029b1e (0x9d08 bytes) 000324 1788190453.324762 read 0x02fefb thru 0x03c1e4 (0xc2ea bytes) 000325 1788190453.335915 trunc from 0x03f127 to 0x013307 (0x2be1f bytes) 000326 1788190453.362569 trunc from 0x00e13f to 0x013307 (0x51c9 bytes) 000327 1788190453.371926 mapread 0x00c87f thru 0x013307 (0x6a89 bytes) 000328 1788190453.399665 mapread 0x004b8e thru 0x010f43 (0xc3b6 bytes) 000329 1788190453.401223 fallocate 0x011b49 thru 0x02181f (0xfcd7 bytes) 000330 1788190453.408357 trunc from 0x00a473 to 0x013307 (0x8e95 bytes) 000331 1788190453.414443 trunc from 0x013308 to 0x039fbb (0x26cb4 bytes) 000332 1788190453.432183 punch from 0x02e4ab to 0x039fbb (0xbb11 bytes) 000333 1788190453.437690 write 0x002fb8 thru 0x00a77a (0x77c3 bytes) 000334 1788190453.444373 fallocate 0x02cd67 thru 0x03824c (0xb4e6 bytes) 000335 1788190453.449463 read 0x031394 thru 0x039fbb (0x8c28 bytes) 000336 1788190453.459113 fallocate 0x01024a thru 0x015b84 (0x593b bytes) 000337 1788190453.463843 mapread 0x029e10 thru 0x02d6e5 (0x38d6 bytes) 000338 1788190453.470686 MIRROR_RESYNC 4 mirrors 000339 1788190454.209549 write 0x03d4e9 thru 0x03ffff (0x2b17 bytes) 000340 1788190454.243808 trunc from 0x040000 to 0x03bce5 (0x431a bytes) 000341 1788190454.289153 mapread 0x02bf58 thru 0x033f30 (0x7fd9 bytes) 000342 1788190454.306835 fallocate 0x020699 thru 0x02b37a (0xace2 bytes) 000343 1788190454.319703 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bce6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bce6): Mirror 3: 0xa57eb4b0 (stale) Mirror 4: 0xa57eb4b0 (stale) Mirror 5: 0xa57eb4b0 (stale) CRC-32 checksum value for chunk [0, 0x3bce6): Mirror 1: 0xa57eb4b0 Mirror 6: 0xa57eb4b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bce6: skipped 000344 1788190455.137499 punch from 0x00a84c to 0x00fa9c (0x5251 bytes) 000345 1788190455.177500 write 0x013806 thru 0x01f891 (0xc08c bytes) 000346 1788190455.190893 mapread 0x039e6c thru 0x03bce5 (0x1e7a bytes) 000347 1788190455.212379 mapread 0x00ce81 thru 0x01a62e (0xd7ae bytes) 000348 1788190455.222803 punch from 0x0099d3 to 0x00cb7f (0x31ad bytes) 000349 1788190455.228066 trunc from 0x012a99 to 0x0209fe (0xdf66 bytes) 000350 1788190455.239482 mapwrite 0x002ddb thru 0x007b75 (0x4d9b bytes) 000351 1788190455.265460 trunc from 0x039ff2 to 0x03bce5 (0x1cf4 bytes) 000352 1788190455.269489 read 0x01bb42 thru 0x02262b (0x6aea bytes) 000353 1788190455.278356 write 0x036d5f thru 0x037f4b (0x11ed bytes) 000354 1788190455.283746 MIRROR_RESYNC 5 mirrors 000355 1788190456.267845 mapwrite 0x002b86 thru 0x009f2e (0x73a9 bytes) 000356 1788190456.378015 mapread 0x00425b thru 0x012e5a (0xec00 bytes) 000357 1788190456.389572 trunc from 0x03bce6 to 0x021ffe (0x19ce7 bytes) 000358 1788190456.447204 MIRROR_RESYNC 5 mirrors 000359 1788190457.331408 read 0x00f249 thru 0x01782f (0x85e7 bytes) 000360 1788190457.340323 punch from 0x004883 to 0x013e2c (0xf5aa bytes) 000361 1788190457.383166 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21fff): Mirror 3: 0x75314d7d (stale) Mirror 4: 0x75314d7d (stale) Mirror 6: 0x75314d7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fff): Mirror 3: 0x75314d7d (stale) Mirror 4: 0x75314d7d (stale) Mirror 6: 0x75314d7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fff: skipped 000362 1788190457.783059 mapread 0x0049a5 thru 0x008015 (0x3671 bytes) 000363 1788190457.805302 trunc from 0x01017f to 0x01095e (0x7e0 bytes) 000364 1788190457.813979 MIRROR_RESYNC 4 mirrors 000365 1788190458.652171 MIRROR_EXTEND to 5 mirrors 000366 1788190459.475252 MIRROR_EXTEND to 6 mirrors 000367 1788190460.285079 write 0x01faed thru 0x022f7f (0x3493 bytes) 000368 1788190460.319381 read 0x01a512 thru 0x01b908 (0x13f7 bytes) 000369 1788190460.324802 trunc from 0x022f80 to 0x007274 (0x1bd0b bytes) 000370 1788190460.341122 mapwrite 0x016178 thru 0x016cc5 (0xb4e bytes) 000371 1788190460.382398 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 3: 0x3beec728 (stale) Mirror 6: 0x3beec728 (stale) Mirror 7: 0x3beec728 (stale) Mirror 8: 0x3beec728 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 3: 0x3beec728 (stale) Mirror 6: 0x3beec728 (stale) Mirror 7: 0x3beec728 (stale) Mirror 8: 0x3beec728 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cc6: skipped 000372 1788190461.041904 mapwrite 0x011bb2 thru 0x012b08 (0xf57 bytes) 000373 1788190461.075596 MIRROR_RESYNC 5 mirrors 000374 1788190462.389786 trunc from 0x0064f7 to 0x006b68 (0x672 bytes) 000375 1788190462.463454 mapread 0x005764 thru 0x00c5fc (0x6e99 bytes) 000376 1788190462.512833 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 3: 0x72e4933d (stale) Mirror 6: 0x72e4933d (stale) Mirror 7: 0x72e4933d (stale) Mirror 8: 0x72e4933d (stale) CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 1: 0x72e4933d Mirror 9: 0x72e4933d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cc6: skipped 000377 1788190463.809097 fallocate 0x005832 thru 0x00ab00 (0x52cf bytes) 000378 1788190463.862857 trunc from 0x010e23 to 0x01653f (0x571d bytes) 000379 1788190463.868271 trunc from 0x016cc6 to 0x0248ab (0xdbe6 bytes) 000380 1788190463.930166 write 0x015e82 thru 0x01f390 (0x950f bytes) 000382 1788190463.967173 trunc from 0x02267b to 0x0248ab (0x2231 bytes) 000383 1788190463.982995 mapwrite 0x03dc1c thru 0x03e4e7 (0x8cc bytes) 000385 1788190464.065067 mapwrite 0x003a3e thru 0x010f41 (0xd504 bytes) 000386 1788190464.192143 mapwrite 0x02f43d thru 0x034877 (0x543b bytes) 000387 1788190464.229904 trunc from 0x03e4e8 to 0x0040df (0x3a408 bytes) 000388 1788190464.258911 fallocate 0x01bf03 thru 0x01f285 (0x3383 bytes) 000389 1788190464.266302 read 0x000160 thru 0x002c0a (0x2aab bytes) 000390 1788190464.276676 fallocate 0x0274ce thru 0x031284 (0x9db7 bytes) 000391 1788190464.283285 mapwrite 0x00f87f thru 0x01a96a (0xb0ec bytes) 000392 1788190464.352599 trunc from 0x0026db to 0x0028e6 (0x20c bytes) 000393 1788190464.373380 trunc from 0x0117e9 to 0x017355 (0x5b6d bytes) 000394 1788190464.400114 MIRROR_RESYNC 6 mirrors 000395 1788190465.849046 trunc from 0x003496 to 0x007cf1 (0x485c bytes) 000396 1788190465.879980 punch from 0x01a6d4 to 0x01a96a (0x297 bytes) 000397 1788190465.888407 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a96b): Mirror 3: 0xf21bf5a4 (stale) Mirror 6: 0xf21bf5a4 (stale) Mirror 7: 0xf21bf5a4 (stale) Mirror 9: 0xf21bf5a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a96b): Mirror 3: 0xf21bf5a4 (stale) Mirror 6: 0xf21bf5a4 (stale) Mirror 7: 0xf21bf5a4 (stale) Mirror 9: 0xf21bf5a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a96b: skipped 000398 1788190466.403204 read 0x00556b thru 0x014085 (0xeb1b bytes) 000399 1788190466.424321 read 0x008d54 thru 0x015968 (0xcc15 bytes) 000400 1788190466.448824 punch from 0x00ffc1 to 0x017207 (0x7247 bytes) 000401 1788190466.455311 trunc from 0x010dcb to 0x01173a (0x970 bytes) 000402 1788190466.470503 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1a96b): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a96b): Mirror 3: 0xf21bf5a4 (stale) Mirror 6: 0xf21bf5a4 (stale) Mirror 7: 0xf21bf5a4 (stale) Mirror 9: 0xf21bf5a4 (stale) CRC-32 checksum value for chunk [0, 0x1a96b): Mirror 1: 0xaf011a4e Mirror 10: 0xaf011a4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a96b: skipped 000403 1788190467.507792 mapread 0x00b0c3 thru 0x00bb02 (0xa40 bytes) 000404 1788190467.516345 mapread 0x006c4f thru 0x012ef3 (0xc2a5 bytes) 000405 1788190467.556348 punch from 0x013982 to 0x018bc4 (0x5243 bytes) 000406 1788190467.607373 trunc from 0x01a96b to 0x0348ab (0x19f41 bytes) 000407 1788190467.646896 mapread 0x009c85 thru 0x00f1a6 (0x5522 bytes) 000408 1788190467.668097 punch from 0x0289b5 to 0x02aba3 (0x21ef bytes) 000409 1788190467.685894 trunc from 0x024949 to 0x028d6e (0x4426 bytes) 000410 1788190467.694427 trunc from 0x0348ac to 0x01d24d (0x1765e bytes) 000411 1788190467.726204 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1d24e): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d24e): Mirror 3: 0xe9908e1d (stale) Mirror 6: 0xe9908e1d (stale) Mirror 7: 0xe9908e1d (stale) Mirror 9: 0xe9908e1d (stale) Mirror 10: 0x55df1aec (stale) CRC-32 checksum value for chunk [0, 0x1d24e): Mirror 1: 0xe88c3acf Mirror 11: 0xe88c3acf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d24e: skipped 000412 1788190469.149825 write 0x022195 thru 0x02b61b (0x9487 bytes) 000413 1788190469.199289 write 0x01991d thru 0x024f08 (0xb5ec bytes) 000414 1788190469.206128 fallocate 0x026951 thru 0x02ac6c (0x431c bytes) 000415 1788190469.211473 read 0x01efdf thru 0x024577 (0x5599 bytes) 000416 1788190469.212305 read 0x00e6b2 thru 0x018657 (0x9fa6 bytes) 000417 1788190469.230560 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b61c): Mirror 3: 0x1c38c4df (stale) Mirror 7: 0x1c38c4df (stale) Mirror 9: 0x1c38c4df (stale) Mirror 10: 0xe95e1270 (stale) Mirror 11: 0x73169a73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b61c): Mirror 3: 0x1c38c4df (stale) Mirror 7: 0x1c38c4df (stale) Mirror 9: 0x1c38c4df (stale) Mirror 10: 0xe95e1270 (stale) Mirror 11: 0x73169a73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b61c: skipped 000418 1788190469.655805 mapread 0x01b738 thru 0x01f5ab (0x3e74 bytes) 000419 1788190469.683469 trunc from 0x006282 to 0x0109da (0xa759 bytes) 000420 1788190469.691268 mapwrite 0x02c85a thru 0x02e362 (0x1b09 bytes) 000421 1788190469.739330 punch from 0x018cc6 to 0x0228cf (0x9c0a bytes) 000422 1788190469.758655 mapwrite 0x03dc2d thru 0x03ffff (0x23d3 bytes) 000423 1788190469.804123 trunc from 0x019c05 to 0x01bce0 (0x20dc bytes) 000424 1788190469.808971 read 0x0255f5 thru 0x02a5e9 (0x4ff5 bytes) 000425 1788190469.809885 mapread 0x039bae thru 0x03ffff (0x6452 bytes) 000426 1788190469.811501 mapread 0x00493e thru 0x0147f8 (0xfebb bytes) 000427 1788190469.831341 read 0x01ab4f thru 0x02605b (0xb50d bytes) 000428 1788190469.847264 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88231e3d (stale) Mirror 9: 0x88231e3d (stale) Mirror 10: 0x9910553d (stale) Mirror 11: 0x66e6eebd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88231e3d (stale) Mirror 9: 0x88231e3d (stale) Mirror 10: 0x9910553d (stale) Mirror 11: 0x66e6eebd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1788190470.569180 punch from 0x01e2fb to 0x0273ba (0x90c0 bytes) 000431 1788190470.585581 trunc from 0x040000 to 0x018620 (0x279df bytes) 000432 1788190470.676703 punch from 0x006738 to 0x00c7fd (0x60c6 bytes) 000433 1788190470.703271 mapwrite 0x01c44e thru 0x0208e1 (0x4494 bytes) 000435 1788190470.880059 trunc from 0x006e61 to 0x00a767 (0x3907 bytes) 000438 1788190470.924973 read 0x009005 thru 0x00dca9 (0x4ca5 bytes) 000439 1788190470.973793 read 0x015477 thru 0x0208e1 (0xb46b bytes) 000440 1788190471.015789 punch from 0x0190af to 0x01c4fc (0x344e bytes) 000441 1788190471.041259 read 0x004bb7 thru 0x00c71c (0x7b66 bytes) 000442 1788190471.047173 mapwrite 0x0310ad thru 0x037fd2 (0x6f26 bytes) 000443 1788190471.210249 mapread 0x01abd8 thru 0x01f763 (0x4b8c bytes) 000444 1788190471.211633 mapwrite 0x03b6d6 thru 0x03ffff (0x492a bytes) 000445 1788190471.262211 mapwrite 0x000c0e thru 0x002094 (0x1487 bytes) 000446 1788190471.281481 trunc from 0x00a3d0 to 0x01553d (0xb16e bytes) 000448 1788190471.289314 mapwrite 0x00caaa thru 0x018725 (0xbc7c bytes) 000450 1788190471.367131 trunc from 0x040000 to 0x02fa85 (0x1057a bytes) 000451 1788190471.389149 punch from 0x0026f6 to 0x007654 (0x4f5f bytes) 000452 1788190471.396069 trunc from 0x0231c5 to 0x023d06 (0xb42 bytes) 000453 1788190471.408466 fallocate 0x012427 thru 0x01bb18 (0x96f2 bytes) 000454 1788190471.424233 read 0x008308 thru 0x00e58c (0x6285 bytes) 000456 1788190471.453395 trunc from 0x007291 to 0x013834 (0xc5a4 bytes) 000457 1788190471.465728 write 0x016513 thru 0x01ed35 (0x8823 bytes) 000458 1788190471.474107 read 0x02065e thru 0x02bfce (0xb971 bytes) 000459 1788190471.481265 mapread 0x00c202 thru 0x012f9f (0x6d9e bytes) 000460 1788190471.517315 write 0x03d6aa thru 0x03ffff (0x2956 bytes) 000462 1788190471.520666 mapwrite 0x01fcc5 thru 0x02bcf0 (0xc02c bytes) 000463 1788190471.571390 fallocate 0x0193c6 thru 0x01a8fd (0x1538 bytes) 000464 1788190471.578277 trunc from 0x040000 to 0x03405d (0xbfa2 bytes) 000465 1788190471.604235 trunc from 0x027e11 to 0x029414 (0x1604 bytes) 000466 1788190471.613147 mapread 0x007c31 thru 0x010563 (0x8933 bytes) 000467 1788190471.617363 write 0x009517 thru 0x010f62 (0x7a4c bytes) 000469 1788190471.627722 fallocate 0x02ea2d thru 0x03b8ef (0xcec3 bytes) 000470 1788190471.635187 punch from 0x008819 to 0x0185c2 (0xfdaa bytes) 000473 1788190471.669693 mapread 0x02af78 thru 0x0301d3 (0x525c bytes) 000474 1788190471.687222 mapread 0x0077f8 thru 0x00b004 (0x380d bytes) 000475 1788190471.760553 mapwrite 0x01ede1 thru 0x0233ab (0x45cb bytes) 000476 1788190471.803825 mapread 0x007b34 thru 0x015365 (0xd832 bytes) 000477 1788190472.006111 trunc from 0x03b8f0 to 0x00de7f (0x2da70 bytes) 000478 1788190472.036961 read 0x000086 thru 0x00cd91 (0xcd0c bytes) 000479 1788190472.038550 mapread 0x00d01b thru 0x00de7f (0xe65 bytes) 000480 1788190472.043219 trunc from 0x009481 to 0x00c594 (0x3114 bytes) 000481 1788190472.049566 read 0x0045ff thru 0x005f38 (0x193a bytes) 000482 1788190472.051269 trunc from 0x00de80 to 0x01b856 (0xd9d7 bytes) 000483 1788190472.076985 mapread 0x00c2f2 thru 0x017774 (0xb483 bytes) 000484 1788190472.148119 fallocate 0x00f182 thru 0x017ab3 (0x8932 bytes) 000485 1788190472.187273 mapwrite 0x0332cb thru 0x034a6a (0x17a0 bytes) 000486 1788190472.359661 trunc from 0x01d59d to 0x0206d0 (0x3134 bytes) 000487 1788190472.379393 write 0x010292 thru 0x019aa2 (0x9811 bytes) 000488 1788190472.380849 trunc from 0x026fd6 to 0x030e57 (0x9e82 bytes) 000489 1788190472.387955 mapwrite 0x018bb1 thru 0x01f0cd (0x651d bytes) 000490 1788190472.535164 read 0x003018 thru 0x005348 (0x2331 bytes) 000491 1788190472.549153 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 3: 0xe18fe2a3 (stale) Mirror 9: 0xe18fe2a3 (stale) Mirror 10: 0xed86020f (stale) Mirror 11: 0x329cd85a (stale) CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 1: 0x71e2bd0e Mirror 12: 0x71e2bd0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a6b: skipped 000492 1788190474.001282 trunc from 0x034a6b to 0x01abc3 (0x19ea7 bytes) 000493 1788190474.095301 trunc from 0x00aad8 to 0x00b359 (0x882 bytes) 000494 1788190474.100275 punch from 0x0019aa to 0x004856 (0x2ead bytes) 000495 1788190474.107547 fallocate 0x037fdc thru 0x03c4a9 (0x44ce bytes) 000496 1788190474.112725 read 0x01cf18 thru 0x02034e (0x3437 bytes) 000497 1788190474.123404 trunc from 0x03c4aa to 0x038451 (0x4058 bytes) 000498 1788190474.158019 fallocate 0x01170e thru 0x01a51f (0x8e12 bytes) 000499 1788190474.169334 punch from 0x037393 to 0x038451 (0x10bf bytes) 000500 1788190474.216416 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38452): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38452): Mirror 3: 0x840fcc39 (stale) Mirror 9: 0x840fcc39 (stale) Mirror 10: 0xcc658a3f (stale) Mirror 11: 0x2b30825f (stale) Mirror 12: 0x2fa1ffca (stale) CRC-32 checksum value for chunk [0, 0x38452): Mirror 1: 0xcd6e30d5 Mirror 13: 0xcd6e30d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38452: skipped 000501 1788190475.496637 mapwrite 0x001bc6 thru 0x003525 (0x1960 bytes) 000502 1788190475.625095 fallocate 0x03da86 thru 0x03ffff (0x257a bytes) 000503 1788190475.641080 fallocate 0x01d6a0 thru 0x02b220 (0xdb81 bytes) 000504 1788190475.656029 write 0x020044 thru 0x02bcc6 (0xbc83 bytes) 000505 1788190475.686406 trunc from 0x040000 to 0x033aae (0xc551 bytes) 000506 1788190475.768878 trunc from 0x033aaf to 0x00e9bc (0x250f2 bytes) 000507 1788190475.794439 write 0x010533 thru 0x01f9dd (0xf4ab bytes) 000508 1788190475.797998 mapread 0x010e65 thru 0x01688a (0x5a26 bytes) 000509 1788190475.799273 fallocate 0x002c1b thru 0x0124a0 (0xf886 bytes) 000511 1788190475.809344 read 0x001ee9 thru 0x00d987 (0xba9f bytes) 000512 1788190475.818832 trunc from 0x002edd to 0x00485f (0x1983 bytes) 000513 1788190475.837610 punch from 0x0196f6 to 0x01dc33 (0x453e bytes) 000514 1788190475.894746 trunc from 0x01f9de to 0x0179cc (0x8011 bytes) 000515 1788190475.960558 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x179cd): Mirror 9: 0xc0a1af27 (stale) Mirror 10: 0xd4d52f9f (stale) Mirror 11: 0xcc7a7af2 (stale) Mirror 12: 0xf8cd8da6 (stale) Mirror 13: 0x82f09620 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179cd): Mirror 9: 0xc0a1af27 (stale) Mirror 10: 0xd4d52f9f (stale) Mirror 11: 0xcc7a7af2 (stale) Mirror 12: 0xf8cd8da6 (stale) Mirror 13: 0x82f09620 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179cd: skipped 000516 1788190476.639585 read 0x0151b6 thru 0x0179cc (0x2817 bytes) 000517 1788190476.657150 write 0x011e9d thru 0x016ffe (0x5162 bytes) 000518 1788190476.677976 trunc from 0x00670a to 0x009170 (0x2a67 bytes) 000519 1788190476.697194 fallocate 0x032a8a thru 0x03622d (0x37a4 bytes) 000520 1788190476.707548 trunc from 0x022d28 to 0x02bec7 (0x91a0 bytes) 000521 1788190476.732629 fallocate 0x0226c3 thru 0x02cc52 (0xa590 bytes) 000523 1788190476.743582 trunc from 0x01dde5 to 0x029344 (0xb560 bytes) 000524 1788190476.757341 read 0x02f456 thru 0x033e60 (0x4a0b bytes) 000525 1788190476.782351 trunc from 0x0034a4 to 0x00bf37 (0x8a94 bytes) 000526 1788190476.789787 punch from 0x015022 to 0x022738 (0xd717 bytes) 000527 1788190476.813129 mapread 0x021541 thru 0x0242f1 (0x2db1 bytes) 000528 1788190476.850777 read 0x01badd thru 0x01f22d (0x3751 bytes) 000529 1788190476.886337 trunc from 0x01385e to 0x01aaf9 (0x729c bytes) 000530 1788190476.899775 punch from 0x010c2e to 0x01366d (0x2a40 bytes) 000531 1788190476.909462 trunc from 0x013ca1 to 0x017b99 (0x3ef9 bytes) 000532 1788190476.932074 trunc from 0x030463 to 0x03622d (0x5dcb bytes) 000533 1788190476.946931 trunc from 0x024a3a to 0x02f1b2 (0xa779 bytes) 000535 1788190476.974038 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3622e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3622e): Mirror 9: 0x6bfc2aaf (stale) Mirror 10: 0x5e68c919 (stale) Mirror 11: 0x3feff138 (stale) Mirror 12: 0x2a8db98a (stale) Mirror 13: 0xf65cfa8 (stale) CRC-32 checksum value for chunk [0, 0x3622e): Mirror 1: 0xf385bcc5 Mirror 14: 0xf385bcc5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3622e: skipped 000536 1788190479.106197 punch from 0x023793 to 0x02397f (0x1ed bytes) 000537 1788190479.199346 write 0x00e8fd thru 0x019b08 (0xb20c bytes) 000538 1788190479.245602 read 0x01acee thru 0x028802 (0xdb15 bytes) 000539 1788190479.267022 trunc from 0x01ae42 to 0x024b87 (0x9d46 bytes) 000540 1788190479.280204 write 0x004ebf thru 0x009fce (0x5110 bytes) 000541 1788190479.316139 punch from 0x01dd16 to 0x01dddc (0xc7 bytes) 000543 1788190479.333988 trunc from 0x03622e to 0x02e71c (0x7b11 bytes) 000544 1788190479.402596 trunc from 0x02e71d to 0x01ac68 (0x13ab4 bytes) 000545 1788190479.435749 trunc from 0x01ac69 to 0x01ae1a (0x1b2 bytes) 000546 1788190479.500877 write 0x02681f thru 0x02d3ec (0x6bce bytes) 000547 1788190479.502254 mapwrite 0x01c822 thru 0x02286d (0x604c bytes) 000548 1788190479.629728 write 0x00caf7 thru 0x0179d5 (0xaedf bytes) 000549 1788190479.663653 mapread 0x01c07e thru 0x021d1c (0x5c9f bytes) 000551 1788190479.681832 mapwrite 0x031ce2 thru 0x03edec (0xd10b bytes) 000552 1788190479.825581 trunc from 0x03eded to 0x03da3b (0x13b1 bytes) 000554 1788190479.857181 read 0x0228e7 thru 0x02a449 (0x7b63 bytes) 000557 1788190479.862057 read 0x0185f0 thru 0x02306a (0xaa7b bytes) 000558 1788190479.863408 mapwrite 0x00965c thru 0x01127c (0x7c21 bytes) 000559 1788190479.944138 punch from 0x01c020 to 0x0287b9 (0xc79a bytes) 000560 1788190479.959191 write 0x02a167 thru 0x038fd7 (0xee71 bytes) 000561 1788190479.970502 write 0x005fe4 thru 0x00aa99 (0x4ab6 bytes) 000562 1788190479.971609 trunc from 0x03da3c to 0x02b37c (0x126bf bytes) 000563 1788190480.051250 punch from 0x006fe3 to 0x00f67c (0x869a bytes) 000564 1788190480.103880 mapwrite 0x000cef thru 0x007afd (0x6e0f bytes) 000565 1788190480.241522 mapwrite 0x02780b thru 0x02b0f8 (0x38ee bytes) 000566 1788190480.316188 write 0x02ee58 thru 0x030525 (0x16ce bytes) 000567 1788190480.319137 trunc from 0x030526 to 0x000606 (0x2ff1f bytes) 000568 1788190480.348889 mapwrite 0x02d8e2 thru 0x035a38 (0x8157 bytes) 000569 1788190480.506966 trunc from 0x035a39 to 0x038dc1 (0x3389 bytes) 000570 1788190480.531472 mapwrite 0x012c4f thru 0x019542 (0x68f4 bytes) 000571 1788190480.656101 mapwrite 0x03c99c thru 0x03ffff (0x3664 bytes) 000572 1788190480.779815 MIRROR_RESYNC 7 mirrors 000573 1788190483.899332 trunc from 0x040000 to 0x00c5df (0x33a20 bytes) 000574 1788190483.992149 trunc from 0x00c5e0 to 0x0142e4 (0x7d05 bytes) 000575 1788190484.048217 read 0x013637 thru 0x0142e4 (0xcae bytes) 000576 1788190484.084290 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x142e5): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142e5): Mirror 9: 0xcd406144 (stale) Mirror 10: 0xcd406144 (stale) Mirror 11: 0xcd406144 (stale) Mirror 12: 0xcd406144 (stale) Mirror 13: 0xcd406144 (stale) Mirror 14: 0xcd406144 (stale) CRC-32 checksum value for chunk [0, 0x142e5): Mirror 1: 0xc3dd3f73 Mirror 15: 0xc3dd3f73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142e5: skipped 000578 1788190486.235675 mapread 0x001dbc thru 0x002f71 (0x11b6 bytes) 000579 1788190486.262377 write 0x0248ed thru 0x0345f1 (0xfd05 bytes) 000580 1788190486.369543 fallocate 0x0320c1 thru 0x03ffff (0xdf3f bytes) 000581 1788190486.413301 punch from 0x00422e to 0x00b072 (0x6e45 bytes) 000582 1788190486.426112 punch from 0x03327b to 0x0345f1 (0x1377 bytes) 000583 1788190486.489029 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x345f2): Mirror 9: 0x9583ff36 (stale) Mirror 10: 0x9583ff36 (stale) Mirror 11: 0x9583ff36 (stale) Mirror 13: 0x9583ff36 (stale) Mirror 14: 0x9583ff36 (stale) Mirror 15: 0xc831e3ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345f2): Mirror 9: 0x9583ff36 (stale) Mirror 10: 0x9583ff36 (stale) Mirror 11: 0x9583ff36 (stale) Mirror 13: 0x9583ff36 (stale) Mirror 14: 0x9583ff36 (stale) Mirror 15: 0xc831e3ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345f2: skipped 000584 1788190487.414611 mapwrite 0x01fbad thru 0x023cb2 (0x4106 bytes) 000585 1788190487.472644 write 0x031d8c thru 0x0372f5 (0x556a bytes) 000586 1788190487.499526 write 0x00b9fe thru 0x00dfab (0x25ae bytes) 000587 1788190487.529958 fallocate 0x00307d thru 0x0084aa (0x542e bytes) 000588 1788190487.544104 trunc from 0x001556 to 0x00c54f (0xaffa bytes) 000589 1788190487.580694 read 0x005ffe thru 0x0109a0 (0xa9a3 bytes) 000590 1788190487.597811 mapwrite 0x027306 thru 0x02e7be (0x74b9 bytes) 000591 1788190487.662573 trunc from 0x00d2bd to 0x0161ae (0x8ef2 bytes) 000593 1788190487.674147 trunc from 0x0372f6 to 0x02a7e1 (0xcb14 bytes) 000594 1788190487.739030 write 0x03fd26 thru 0x03ffff (0x2da bytes) 000596 1788190487.740821 read 0x0373ba thru 0x03cf50 (0x5b97 bytes) 000597 1788190487.777430 MIRROR_RESYNC 7 mirrors 000598 1788190489.886160 fallocate 0x00d908 thru 0x018eed (0xb5e6 bytes) 000599 1788190489.962126 punch from 0x01c979 to 0x01d35b (0x9e3 bytes) 000600 1788190489.969983 fallocate 0x02558a thru 0x032e2c (0xd8a3 bytes) 000601 1788190489.978787 mapwrite 0x002605 thru 0x00f059 (0xca55 bytes) 000602 1788190490.114747 trunc from 0x040000 to 0x016c74 (0x2938b bytes) 000603 1788190490.219390 punch from 0x006d0f to 0x015803 (0xeaf5 bytes) 000604 1788190490.266663 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x16c75): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16c75): Mirror 9: 0x98cebf02 (stale) Mirror 10: 0x98cebf02 (stale) Mirror 11: 0x98cebf02 (stale) Mirror 13: 0x98cebf02 (stale) Mirror 14: 0x98cebf02 (stale) Mirror 15: 0x98cebf02 (stale) CRC-32 checksum value for chunk [0, 0x16c75): Mirror 1: 0x4bf1f307 Mirror 16: 0x4bf1f307 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16c75: skipped 000605 1788190492.319079 trunc from 0x007b72 to 0x00e6cd (0x6b5c bytes) 000606 1788190492.367004 mapwrite 0x019254 thru 0x01b53f (0x22ec bytes) 000607 1788190492.437561 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1b540): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b540): Mirror 9: 0xcec79b6d (stale) Mirror 10: 0xcec79b6d (stale) Mirror 11: 0xcec79b6d (stale) Mirror 13: 0xcec79b6d (stale) Mirror 14: 0xcec79b6d (stale) Mirror 15: 0xcec79b6d (stale) Mirror 16: 0xb39dbf30 (stale) CRC-32 checksum value for chunk [0, 0x1b540): Mirror 1: 0xafa69da8 Mirror 17: 0xafa69da8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b540: skipped 000608 1788190493.877825 trunc from 0x01b540 to 0x033e98 (0x18959 bytes) 000610 1788190493.972315 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33e99): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e99): Mirror 9: 0x81726e96 (stale) Mirror 10: 0x81726e96 (stale) Mirror 11: 0x81726e96 (stale) Mirror 13: 0x81726e96 (stale) Mirror 14: 0x81726e96 (stale) Mirror 15: 0x81726e96 (stale) Mirror 16: 0x8c22d0ef (stale) Mirror 17: 0x4fe3356a (stale) CRC-32 checksum value for chunk [0, 0x33e99): Mirror 1: 0x4fe3356a Mirror 18: 0x4fe3356a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e99: skipped 000611 1788190495.710168 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33e99): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e99): Mirror 9: 0x81726e96 (stale) Mirror 10: 0x81726e96 (stale) Mirror 11: 0x81726e96 (stale) Mirror 13: 0x81726e96 (stale) Mirror 14: 0x81726e96 (stale) Mirror 15: 0x81726e96 (stale) Mirror 16: 0x8c22d0ef (stale) Mirror 17: 0x4fe3356a (stale) CRC-32 checksum value for chunk [0, 0x33e99): Mirror 1: 0x4fe3356a Mirror 18: 0x4fe3356a Mirror 19: 0x4fe3356a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e99: skipped 000612 1788190497.472339 trunc from 0x033e99 to 0x03a918 (0x6a80 bytes) 000614 1788190497.556307 write 0x029a90 thru 0x0395a7 (0xfb18 bytes) 000615 1788190497.574809 mapread 0x014f56 thru 0x023f3e (0xefe9 bytes) 000616 1788190497.643657 trunc from 0x01edde to 0x0207c6 (0x19e9 bytes) 000617 1788190497.654832 mapread 0x004b6d thru 0x00aaf9 (0x5f8d bytes) 000618 1788190497.673180 mapread 0x019d10 thru 0x01e121 (0x4412 bytes) 000620 1788190497.683222 trunc from 0x019a2c to 0x022a2a (0x8fff bytes) 000621 1788190497.689760 punch from 0x03389c to 0x03a918 (0x707d bytes) 000622 1788190497.727724 mapread 0x032afb thru 0x03a918 (0x7e1e bytes) 000623 1788190497.748319 write 0x00ddaf thru 0x00eaf2 (0xd44 bytes) 000624 1788190497.759926 trunc from 0x03a919 to 0x007b12 (0x32e06 bytes) 000625 1788190497.808801 fallocate 0x00752e thru 0x00a3e9 (0x2ebc bytes) 000626 1788190497.837175 trunc from 0x007b13 to 0x02da8c (0x25f7a bytes) 000627 1788190497.877173 mapread 0x019dd9 thru 0x020158 (0x6380 bytes) 000628 1788190497.924227 read 0x01debc thru 0x020172 (0x22b7 bytes) 000629 1788190497.931431 mapread 0x02b0fb thru 0x02da8c (0x2992 bytes) 000630 1788190497.947297 fallocate 0x02c2e8 thru 0x03bc96 (0xf9af bytes) 000631 1788190497.960225 mapread 0x03035d thru 0x03bc96 (0xb93a bytes) 000632 1788190497.995294 fallocate 0x00f906 thru 0x0170a3 (0x779e bytes) 000633 1788190498.006609 trunc from 0x03bc97 to 0x03bcb0 (0x1a bytes) 000634 1788190498.049929 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 9: 0xd392792e (stale) Mirror 10: 0xd392792e (stale) Mirror 11: 0xd392792e (stale) Mirror 13: 0xd392792e (stale) Mirror 14: 0xd392792e (stale) Mirror 15: 0xd392792e (stale) Mirror 16: 0x4af94ace (stale) Mirror 17: 0x1d6c56ad (stale) Mirror 18: 0x1d6c56ad (stale) Mirror 19: 0x1d6c56ad (stale) CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 1: 0x4af94ace Mirror 20: 0x4af94ace lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bcb1: skipped 000635 1788190499.634836 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 9: 0xd392792e (stale) Mirror 10: 0xd392792e (stale) Mirror 11: 0xd392792e (stale) Mirror 13: 0xd392792e (stale) Mirror 14: 0xd392792e (stale) Mirror 15: 0xd392792e (stale) Mirror 17: 0x1d6c56ad (stale) Mirror 18: 0x1d6c56ad (stale) Mirror 19: 0x1d6c56ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 9: 0xd392792e (stale) Mirror 10: 0xd392792e (stale) Mirror 11: 0xd392792e (stale) Mirror 13: 0xd392792e (stale) Mirror 14: 0xd392792e (stale) Mirror 15: 0xd392792e (stale) Mirror 17: 0x1d6c56ad (stale) Mirror 18: 0x1d6c56ad (stale) Mirror 19: 0x1d6c56ad (stale) CRC-32 checksum value for chunk [0, 0x3bcb1): Mirror 1: 0x4af94ace Mirror 20: 0x4af94ace lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bcb1: skipped 000636 1788190501.472095 trunc from 0x03bcb1 to 0x013c2b (0x28085 bytes) 000637 1788190501.650569 write 0x00b3d9 thru 0x00ba7b (0x6a3 bytes) 000638 1788190501.661146 punch from 0x00ea01 to 0x013c2b (0x522b bytes) 000639 1788190501.675127 trunc from 0x013580 to 0x013c2b (0x6ac bytes) 000640 1788190501.696701 mapwrite 0x014364 thru 0x020a9d (0xc73a bytes) 000641 1788190501.807275 write 0x010143 thru 0x0135dd (0x349b bytes) 000642 1788190501.812399 trunc from 0x020a9e to 0x0131e3 (0xd8ba bytes) 000643 1788190501.835283 write 0x03cb10 thru 0x03ffff (0x34f0 bytes) 000645 1788190501.838680 punch from 0x004543 to 0x00ec96 (0xa754 bytes) 000646 1788190501.859299 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3e79b355 (stale) Mirror 10: 0x3e79b355 (stale) Mirror 11: 0x3e79b355 (stale) Mirror 13: 0x3e79b355 (stale) Mirror 14: 0x3e79b355 (stale) Mirror 15: 0x3e79b355 (stale) Mirror 17: 0x463f7fa (stale) Mirror 18: 0x463f7fa (stale) Mirror 19: 0x463f7fa (stale) Mirror 20: 0x1d2f56e4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf4c63f62 Mirror 21: 0xf4c63f62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1788190503.171330 fallocate 0x01bab8 thru 0x028ddc (0xd325 bytes) 000648 1788190503.231940 mapread 0x034435 thru 0x038413 (0x3fdf bytes) 000649 1788190503.255021 mapread 0x00c71d thru 0x00fbbc (0x34a0 bytes) 000650 1788190503.271932 mapread 0x0325e9 thru 0x039cf6 (0x770e bytes) 000651 1788190503.276795 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3e79b355 (stale) Mirror 11: 0x3e79b355 (stale) Mirror 13: 0x3e79b355 (stale) Mirror 14: 0x3e79b355 (stale) Mirror 15: 0x3e79b355 (stale) Mirror 17: 0x463f7fa (stale) Mirror 18: 0x463f7fa (stale) Mirror 19: 0x463f7fa (stale) Mirror 20: 0x1d2f56e4 (stale) Mirror 21: 0xf4c63f62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3e79b355 (stale) Mirror 11: 0x3e79b355 (stale) Mirror 13: 0x3e79b355 (stale) Mirror 14: 0x3e79b355 (stale) Mirror 15: 0x3e79b355 (stale) Mirror 17: 0x463f7fa (stale) Mirror 18: 0x463f7fa (stale) Mirror 19: 0x463f7fa (stale) Mirror 20: 0x1d2f56e4 (stale) Mirror 21: 0xf4c63f62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000652 1788190503.819753 trunc from 0x040000 to 0x031dd0 (0xe22f bytes) 000653 1788190503.863681 mapread 0x023395 thru 0x02c3d2 (0x903e bytes) 000654 1788190503.900197 MIRROR_RESYNC 11 mirrors 000655 1788190508.314573 write 0x03f38f thru 0x03ffff (0xc71 bytes) 000656 1788190508.379399 fallocate 0x019ced thru 0x0258a0 (0xbbb4 bytes) 000657 1788190508.389085 write 0x038661 thru 0x03969a (0x103a bytes) 000658 1788190508.415943 MIRROR_RESYNC 11 mirrors 000659 1788190512.169180 fallocate 0x03a584 thru 0x03ffff (0x5a7c bytes) 000661 1788190512.204880 mapwrite 0x00d455 thru 0x0128d4 (0x5480 bytes) 000662 1788190512.237627 punch from 0x001429 to 0x00e04f (0xcc27 bytes) 000663 1788190512.242630 fallocate 0x003ecf thru 0x00d5a9 (0x96db bytes) 000664 1788190512.247130 trunc from 0x0253d4 to 0x029be7 (0x4814 bytes) 000665 1788190512.270949 mapread 0x026fd9 thru 0x028b61 (0x1b89 bytes) 000666 1788190512.289342 mapread 0x03d223 thru 0x03d98c (0x76a bytes) 000667 1788190512.320333 trunc from 0x040000 to 0x02596c (0x1a693 bytes) 000668 1788190512.365162 punch from 0x00a323 to 0x00e0e0 (0x3dbe bytes) 000669 1788190512.371859 trunc from 0x00fd86 to 0x01604c (0x62c7 bytes) 000670 1788190512.383534 mapread 0x00b9b8 thru 0x010505 (0x4b4e bytes) 000671 1788190512.405047 trunc from 0x00723c to 0x016751 (0xf516 bytes) 000672 1788190512.417877 mapwrite 0x033f49 thru 0x03ffff (0xc0b7 bytes) 000673 1788190512.632729 punch from 0x02a1fd to 0x02f86a (0x566e bytes) 000675 1788190512.653701 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2d2908b1 (stale) Mirror 13: 0x2d2908b1 (stale) Mirror 14: 0x2d2908b1 (stale) Mirror 15: 0x2d2908b1 (stale) Mirror 17: 0x2d2908b1 (stale) Mirror 18: 0x2d2908b1 (stale) Mirror 19: 0x2d2908b1 (stale) Mirror 20: 0x2d2908b1 (stale) Mirror 21: 0x2d2908b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2d2908b1 (stale) Mirror 13: 0x2d2908b1 (stale) Mirror 14: 0x2d2908b1 (stale) Mirror 15: 0x2d2908b1 (stale) Mirror 17: 0x2d2908b1 (stale) Mirror 18: 0x2d2908b1 (stale) Mirror 19: 0x2d2908b1 (stale) Mirror 20: 0x2d2908b1 (stale) Mirror 21: 0x2d2908b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1788190513.260884 punch from 0x03942d to 0x03ffff (0x6bd3 bytes) 000678 1788190513.271333 mapread 0x0382c4 thru 0x03ffff (0x7d3c bytes) 000679 1788190513.294081 trunc from 0x001b2d to 0x001fc9 (0x49d bytes) 000680 1788190513.305799 mapwrite 0x00cd98 thru 0x015220 (0x8489 bytes) 000682 1788190513.342755 punch from 0x011692 to 0x01d85e (0xc1cd bytes) 000683 1788190513.346839 mapwrite 0x023298 thru 0x02552a (0x2293 bytes) 000684 1788190513.360339 mapread 0x012d21 thru 0x018f4e (0x622e bytes) 000685 1788190513.369967 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2d2908b1 (stale) Mirror 13: 0x2d2908b1 (stale) Mirror 14: 0x2d2908b1 (stale) Mirror 15: 0x2d2908b1 (stale) Mirror 17: 0x2d2908b1 (stale) Mirror 19: 0x2d2908b1 (stale) Mirror 20: 0x2d2908b1 (stale) Mirror 21: 0x2d2908b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2d2908b1 (stale) Mirror 13: 0x2d2908b1 (stale) Mirror 14: 0x2d2908b1 (stale) Mirror 15: 0x2d2908b1 (stale) Mirror 17: 0x2d2908b1 (stale) Mirror 19: 0x2d2908b1 (stale) Mirror 20: 0x2d2908b1 (stale) Mirror 21: 0x2d2908b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000686 1788190514.006903 MIRROR_RESYNC 9 mirrors 000687 1788190516.945092 mapread 0x02de53 thru 0x03aae8 (0xcc96 bytes) 000688 1788190517.006622 mapread 0x0089b6 thru 0x008d97 (0x3e2 bytes) 000689 1788190517.048467 MIRROR_EXTEND to 10 mirrors 000690 1788190518.501845 mapread 0x03fe92 thru 0x03ffff (0x16e bytes) 000691 1788190518.520746 write 0x0253da thru 0x02fb91 (0xa7b8 bytes) 000692 1788190518.630625 mapread 0x01f21a thru 0x01fa27 (0x80e bytes) 000693 1788190518.651957 mapread 0x02ba6f thru 0x038967 (0xcef9 bytes) 000694 1788190518.693731 read 0x035b9d thru 0x03ffff (0xa463 bytes) 000695 1788190518.735034 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x69241ce0 (stale) Mirror 13: 0x69241ce0 (stale) Mirror 14: 0x69241ce0 (stale) Mirror 15: 0x69241ce0 (stale) Mirror 17: 0x69241ce0 (stale) Mirror 19: 0x69241ce0 (stale) Mirror 20: 0x69241ce0 (stale) Mirror 21: 0x69241ce0 (stale) Mirror 22: 0x69241ce0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x919413b8 Mirror 23: 0x919413b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000696 1788190520.300402 mapwrite 0x013fb5 thru 0x01bd67 (0x7db3 bytes) 000697 1788190520.378988 mapread 0x03dcd4 thru 0x03ffff (0x232c bytes) 000698 1788190520.387364 mapwrite 0x01a758 thru 0x020433 (0x5cdc bytes) 000699 1788190520.410676 trunc from 0x040000 to 0x0138e9 (0x2c716 bytes) 000700 1788190520.432266 punch from 0x0014b8 to 0x00b269 (0x9db2 bytes) 000701 1788190520.442087 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x138ea): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x138ea): Mirror 9: 0x44a5065e (stale) Mirror 13: 0x44a5065e (stale) Mirror 14: 0x44a5065e (stale) Mirror 15: 0x44a5065e (stale) Mirror 17: 0x44a5065e (stale) Mirror 19: 0x44a5065e (stale) Mirror 20: 0x44a5065e (stale) Mirror 21: 0x44a5065e (stale) Mirror 22: 0x44a5065e (stale) Mirror 23: 0x44a5065e (stale) CRC-32 checksum value for chunk [0, 0x138ea): Mirror 1: 0x44a5065e Mirror 24: 0x44a5065e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x138ea: skipped 000702 1788190521.777856 trunc from 0x0076ef to 0x00ab82 (0x3494 bytes) 000703 1788190521.867976 trunc from 0x0138ea to 0x02a604 (0x16d1b bytes) 000704 1788190521.915869 trunc from 0x016efa to 0x01ab7b (0x3c82 bytes) 000705 1788190521.933816 fallocate 0x03fbb2 thru 0x03ffff (0x44e bytes) 000708 1788190521.943910 mapwrite 0x002a6e thru 0x010498 (0xda2b bytes) 000709 1788190522.022338 write 0x00ca1f thru 0x01634a (0x992c bytes) 000710 1788190522.027429 mapwrite 0x02c4ca thru 0x038d48 (0xc87f bytes) 000711 1788190522.146189 read 0x014281 thru 0x0190fa (0x4e7a bytes) 000712 1788190522.177552 fallocate 0x002da6 thru 0x004e3e (0x2099 bytes) 000713 1788190522.192469 mapread 0x03bc8a thru 0x03ffff (0x4376 bytes) 000714 1788190522.199272 write 0x00431a thru 0x005f4a (0x1c31 bytes) 000715 1788190522.210543 write 0x03a7bf thru 0x03ffff (0x5841 bytes) 000716 1788190522.220044 trunc from 0x03e299 to 0x03ffff (0x1d67 bytes) 000717 1788190522.263815 mapwrite 0x02a679 thru 0x02b4b1 (0xe39 bytes) 000719 1788190522.292180 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x69241ce0 (stale) Mirror 13: 0x69241ce0 (stale) Mirror 14: 0x69241ce0 (stale) Mirror 15: 0x69241ce0 (stale) Mirror 17: 0x69241ce0 (stale) Mirror 19: 0x69241ce0 (stale) Mirror 20: 0x69241ce0 (stale) Mirror 21: 0x69241ce0 (stale) Mirror 22: 0x69241ce0 (stale) Mirror 23: 0x919413b8 (stale) Mirror 24: 0x707d801f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b9b3b27 Mirror 25: 0x3b9b3b27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000720 1788190523.552167 read 0x01ead7 thru 0x027a19 (0x8f43 bytes) 000721 1788190523.576387 trunc from 0x040000 to 0x015730 (0x2a8cf bytes) 000722 1788190523.686494 punch from 0x00d7dc to 0x015730 (0x7f55 bytes) 000723 1788190523.691176 trunc from 0x00c613 to 0x013ecc (0x78ba bytes) 000724 1788190523.696378 trunc from 0x00ea07 to 0x012931 (0x3f2b bytes) 000725 1788190523.700460 write 0x005482 thru 0x00b715 (0x6294 bytes) 000726 1788190523.725159 punch from 0x007fa4 to 0x013b55 (0xbbb2 bytes) 000727 1788190523.744394 trunc from 0x00b993 to 0x011a10 (0x607e bytes) 000729 1788190523.750704 read 0x0074dc thru 0x00caa2 (0x55c7 bytes) 000730 1788190523.765951 mapread 0x00e87c thru 0x015730 (0x6eb5 bytes) 000731 1788190523.769726 punch from 0x00285c to 0x00624f (0x39f4 bytes) 000732 1788190523.775964 punch from 0x007cad to 0x015730 (0xda84 bytes) 000733 1788190523.782783 fallocate 0x02df4d thru 0x030fc3 (0x3077 bytes) 000734 1788190523.798396 trunc from 0x015731 to 0x00a10a (0xb626 bytes) 000735 1788190523.837672 punch from 0x0022ee to 0x00a10a (0x7e1d bytes) 000736 1788190523.855141 trunc from 0x00a10b to 0x0225ab (0x184a1 bytes) 000737 1788190523.888290 trunc from 0x0225ac to 0x013ec7 (0xe6e4 bytes) 000738 1788190523.919338 read 0x00e958 thru 0x013ec7 (0x5570 bytes) 000739 1788190523.959753 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ec8): Mirror 9: 0x93ea7716 (stale) Mirror 13: 0x93ea7716 (stale) Mirror 14: 0x93ea7716 (stale) Mirror 15: 0x93ea7716 (stale) Mirror 17: 0x93ea7716 (stale) Mirror 19: 0x93ea7716 (stale) Mirror 20: 0x93ea7716 (stale) Mirror 22: 0x93ea7716 (stale) Mirror 23: 0x93ea7716 (stale) Mirror 24: 0x93ea7716 (stale) Mirror 25: 0x94dc9808 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ec8): Mirror 9: 0x93ea7716 (stale) Mirror 13: 0x93ea7716 (stale) Mirror 14: 0x93ea7716 (stale) Mirror 15: 0x93ea7716 (stale) Mirror 17: 0x93ea7716 (stale) Mirror 19: 0x93ea7716 (stale) Mirror 20: 0x93ea7716 (stale) Mirror 22: 0x93ea7716 (stale) Mirror 23: 0x93ea7716 (stale) Mirror 24: 0x93ea7716 (stale) Mirror 25: 0x94dc9808 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ec8: skipped 000740 1788190524.706293 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ec8): Mirror 9: 0x93ea7716 (stale) Mirror 13: 0x93ea7716 (stale) Mirror 14: 0x93ea7716 (stale) Mirror 15: 0x93ea7716 (stale) Mirror 17: 0x93ea7716 (stale) Mirror 19: 0x93ea7716 (stale) Mirror 20: 0x93ea7716 (stale) Mirror 22: 0x93ea7716 (stale) Mirror 24: 0x93ea7716 (stale) Mirror 25: 0x94dc9808 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ec8): Mirror 9: 0x93ea7716 (stale) Mirror 13: 0x93ea7716 (stale) Mirror 14: 0x93ea7716 (stale) Mirror 15: 0x93ea7716 (stale) Mirror 17: 0x93ea7716 (stale) Mirror 19: 0x93ea7716 (stale) Mirror 20: 0x93ea7716 (stale) Mirror 22: 0x93ea7716 (stale) Mirror 24: 0x93ea7716 (stale) Mirror 25: 0x94dc9808 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ec8: skipped 000741 1788190525.165224 trunc from 0x0136ab to 0x013ec7 (0x81d bytes) 000742 1788190525.168668 fallocate 0x031503 thru 0x031e74 (0x972 bytes) 000743 1788190525.172202 read 0x020301 thru 0x022731 (0x2431 bytes) 000746 1788190525.178939 read 0x01ac09 thru 0x021752 (0x6b4a bytes) 000747 1788190525.189466 MIRROR_RESYNC 11 mirrors 000748 1788190526.720445 trunc from 0x0309c0 to 0x031e74 (0x14b5 bytes) 000749 1788190526.759045 fallocate 0x02dcff thru 0x02ed3e (0x1040 bytes) 000750 1788190526.767549 trunc from 0x00ca6d to 0x01a65f (0xdbf3 bytes) 000751 1788190526.779333 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e75): Mirror 9: 0x5bad38c (stale) Mirror 13: 0x5bad38c (stale) Mirror 14: 0x5bad38c (stale) Mirror 15: 0x5bad38c (stale) Mirror 19: 0x5bad38c (stale) Mirror 20: 0x5bad38c (stale) Mirror 22: 0x5bad38c (stale) Mirror 24: 0x5bad38c (stale) Mirror 25: 0x5bad38c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e75): Mirror 9: 0x5bad38c (stale) Mirror 13: 0x5bad38c (stale) Mirror 14: 0x5bad38c (stale) Mirror 15: 0x5bad38c (stale) Mirror 19: 0x5bad38c (stale) Mirror 20: 0x5bad38c (stale) Mirror 22: 0x5bad38c (stale) Mirror 24: 0x5bad38c (stale) Mirror 25: 0x5bad38c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e75: skipped 000752 1788190527.684132 punch from 0x00c97c to 0x00cd88 (0x40d bytes) 000753 1788190527.689459 write 0x02bb1d thru 0x036448 (0xa92c bytes) 000754 1788190527.713282 mapwrite 0x03e906 thru 0x03f307 (0xa02 bytes) 000755 1788190527.780375 fallocate 0x0096f8 thru 0x00f0e9 (0x59f2 bytes) 000756 1788190527.792274 trunc from 0x03f308 to 0x01d34d (0x21fba bytes) 000757 1788190527.812291 trunc from 0x00541a to 0x007c20 (0x2807 bytes) 000758 1788190527.819384 trunc from 0x010846 to 0x018e71 (0x862c bytes) 000760 1788190527.825895 write 0x0398f8 thru 0x03ace3 (0x13ec bytes) 000761 1788190527.827353 punch from 0x01085a to 0x01d567 (0xcd0e bytes) 000762 1788190527.832420 trunc from 0x03764b to 0x03ace3 (0x3699 bytes) 000763 1788190527.842054 fallocate 0x033305 thru 0x037016 (0x3d12 bytes) 000764 1788190527.847428 trunc from 0x03ace4 to 0x0076da (0x33609 bytes) 000765 1788190527.871463 fallocate 0x0312bb thru 0x03faff (0xe845 bytes) 000766 1788190527.877830 mapread 0x0023ea thru 0x0076da (0x52f1 bytes) 000769 1788190527.890464 trunc from 0x0076db to 0x0369a3 (0x2f2c9 bytes) 000770 1788190527.915952 trunc from 0x0369a4 to 0x0123be (0x245e5 bytes) 000771 1788190527.947093 mapwrite 0x0394b8 thru 0x03ffff (0x6b48 bytes) 000772 1788190528.009625 punch from 0x0115bf to 0x011903 (0x345 bytes) 000773 1788190528.019555 trunc from 0x040000 to 0x011085 (0x2ef7a bytes) 000774 1788190528.046675 mapread 0x00f282 thru 0x011085 (0x1e04 bytes) 000776 1788190528.063137 fallocate 0x000e9c thru 0x003532 (0x2697 bytes) 000777 1788190528.069682 read 0x00485d thru 0x00b9ab (0x714f bytes) 000778 1788190528.077831 MIRROR_RESYNC 10 mirrors 000779 1788190530.137664 fallocate 0x00f34d thru 0x017e8e (0x8b42 bytes) 000780 1788190530.180972 mapwrite 0x023079 thru 0x024b44 (0x1acc bytes) 000781 1788190530.224590 fallocate 0x02446a thru 0x02e97b (0xa512 bytes) 000782 1788190530.235663 punch from 0x00f38d to 0x0140eb (0x4d5f bytes) 000783 1788190530.240753 punch from 0x00fcb2 to 0x01b3d8 (0xb727 bytes) 000784 1788190530.246244 mapwrite 0x0265ca thru 0x0288f8 (0x232f bytes) 000786 1788190530.290349 write 0x033bdc thru 0x0362dc (0x2701 bytes) 000787 1788190530.292829 write 0x009488 thru 0x01184e (0x83c7 bytes) 000788 1788190530.305209 mapwrite 0x02ae9b thru 0x03a2dc (0xf442 bytes) 000789 1788190530.369600 trunc from 0x02a71d to 0x02b3c4 (0xca8 bytes) 000790 1788190530.376448 trunc from 0x037905 to 0x03a2dc (0x29d8 bytes) 000791 1788190530.382228 read 0x020e18 thru 0x02b069 (0xa252 bytes) 000792 1788190530.389355 read 0x02fbc4 thru 0x0346fa (0x4b37 bytes) 000793 1788190530.395287 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a2dd): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2dd): Mirror 9: 0x5095976d (stale) Mirror 13: 0x5095976d (stale) Mirror 14: 0x5095976d (stale) Mirror 15: 0x5095976d (stale) Mirror 19: 0x5095976d (stale) Mirror 20: 0x5095976d (stale) Mirror 22: 0x5095976d (stale) Mirror 24: 0x5095976d (stale) Mirror 25: 0x5095976d (stale) CRC-32 checksum value for chunk [0, 0x3a2dd): Mirror 1: 0xfb61dab6 Mirror 26: 0xfb61dab6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2dd: skipped 000795 1788190531.583008 trunc from 0x03a2dd to 0x02611d (0x141bf bytes) 000796 1788190531.715165 mapwrite 0x025c26 thru 0x028556 (0x2931 bytes) 000797 1788190531.786650 punch from 0x011b32 to 0x01eb51 (0xd020 bytes) 000798 1788190531.793299 fallocate 0x023582 thru 0x03201f (0xea9e bytes) 000799 1788190531.804430 mapwrite 0x01709c thru 0x02143b (0xa3a0 bytes) 000800 1788190531.857309 write 0x004ef9 thru 0x011b88 (0xcc90 bytes) 000801 1788190531.878356 MIRROR_RESYNC 11 mirrors 000802 1788190536.236709 read 0x01dfeb thru 0x01e944 (0x95a bytes) 000803 1788190536.242154 fallocate 0x02bd73 thru 0x02f471 (0x36ff bytes) 000804 1788190536.283613 punch from 0x01a9d3 to 0x01b561 (0xb8f bytes) 000805 1788190536.289785 punch from 0x01c544 to 0x026193 (0x9c50 bytes) 000806 1788190536.295810 mapread 0x0146bc thru 0x023653 (0xef98 bytes) 000807 1788190536.351113 trunc from 0x032020 to 0x03789e (0x587f bytes) 000808 1788190536.381503 read 0x018b85 thru 0x02459a (0xba16 bytes) 000809 1788190536.382856 mapread 0x01c5bb thru 0x025b38 (0x957e bytes) 000810 1788190536.385124 fallocate 0x008bae thru 0x00c9f9 (0x3e4c bytes) 000811 1788190536.399610 trunc from 0x01ee38 to 0x028ddf (0x9fa8 bytes) 000812 1788190536.406369 mapwrite 0x03ab33 thru 0x03ffff (0x54cd bytes) 000813 1788190536.447345 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x663e9916 (stale) Mirror 14: 0x663e9916 (stale) Mirror 15: 0x663e9916 (stale) Mirror 19: 0x663e9916 (stale) Mirror 20: 0x663e9916 (stale) Mirror 22: 0x663e9916 (stale) Mirror 24: 0x663e9916 (stale) Mirror 25: 0x663e9916 (stale) Mirror 26: 0x663e9916 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x663e9916 (stale) Mirror 14: 0x663e9916 (stale) Mirror 15: 0x663e9916 (stale) Mirror 19: 0x663e9916 (stale) Mirror 20: 0x663e9916 (stale) Mirror 22: 0x663e9916 (stale) Mirror 24: 0x663e9916 (stale) Mirror 25: 0x663e9916 (stale) Mirror 26: 0x663e9916 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1788190537.252356 mapwrite 0x007aa5 thru 0x010401 (0x895d bytes) 000815 1788190537.297790 mapread 0x00eaa5 thru 0x01b932 (0xce8e bytes) 000816 1788190537.339010 MIRROR_RESYNC 10 mirrors 000817 1788190540.779901 read 0x00fc3e thru 0x011a91 (0x1e54 bytes) 000818 1788190540.784049 trunc from 0x040000 to 0x00b3c8 (0x34c37 bytes) 000819 1788190540.839310 trunc from 0x00b3c9 to 0x00198d (0x9a3b bytes) 000820 1788190540.861414 trunc from 0x00198e to 0x006c33 (0x52a6 bytes) 000821 1788190540.878256 trunc from 0x006c34 to 0x024178 (0x1d545 bytes) 000822 1788190540.897815 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x24179): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24179): Mirror 9: 0x7c746f81 (stale) Mirror 14: 0x7c746f81 (stale) Mirror 15: 0x7c746f81 (stale) Mirror 19: 0x7c746f81 (stale) Mirror 20: 0x7c746f81 (stale) Mirror 22: 0x7c746f81 (stale) Mirror 24: 0x7c746f81 (stale) Mirror 25: 0x7c746f81 (stale) Mirror 26: 0x7c746f81 (stale) CRC-32 checksum value for chunk [0, 0x24179): Mirror 1: 0x87bca080 Mirror 27: 0x87bca080 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24179: skipped 000823 1788190542.253683 read 0x0046eb thru 0x010a5d (0xc373 bytes) 000824 1788190542.273342 trunc from 0x01965e to 0x01f19e (0x5b41 bytes) 000825 1788190542.330478 trunc from 0x024179 to 0x021c25 (0x2553 bytes) 000826 1788190542.359079 read 0x007524 thru 0x00f5da (0x80b7 bytes) 000828 1788190542.376041 MIRROR_RESYNC 11 mirrors 000830 1788190544.547455 mapwrite 0x0161ee thru 0x020ae2 (0xa8f5 bytes) 000831 1788190544.730075 mapread 0x004fe7 thru 0x00ff9c (0xafb6 bytes) 000832 1788190544.741876 read 0x005f11 thru 0x007c7e (0x1d6e bytes) 000833 1788190544.742837 read 0x00b2ab thru 0x00d7af (0x2505 bytes) 000834 1788190544.749789 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21c26): Mirror 9: 0x93c70a07 (stale) Mirror 14: 0x93c70a07 (stale) Mirror 15: 0x93c70a07 (stale) Mirror 19: 0x93c70a07 (stale) Mirror 20: 0x93c70a07 (stale) Mirror 22: 0x93c70a07 (stale) Mirror 24: 0x93c70a07 (stale) Mirror 25: 0x93c70a07 (stale) Mirror 27: 0x93c70a07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c26): Mirror 9: 0x93c70a07 (stale) Mirror 14: 0x93c70a07 (stale) Mirror 15: 0x93c70a07 (stale) Mirror 19: 0x93c70a07 (stale) Mirror 20: 0x93c70a07 (stale) Mirror 22: 0x93c70a07 (stale) Mirror 24: 0x93c70a07 (stale) Mirror 25: 0x93c70a07 (stale) Mirror 27: 0x93c70a07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c26: skipped 000835 1788190545.471099 write 0x01a492 thru 0x023f3d (0x9aac bytes) 000836 1788190545.493375 trunc from 0x023f3e to 0x0312df (0xd3a2 bytes) 000838 1788190545.525072 write 0x01d44a thru 0x027d4a (0xa901 bytes) 000840 1788190545.535300 punch from 0x0231e6 to 0x0262be (0x30d9 bytes) 000841 1788190545.565229 mapread 0x00660a thru 0x016282 (0xfc79 bytes) 000842 1788190545.600813 fallocate 0x036800 thru 0x03ffff (0x9800 bytes) 000843 1788190545.609708 MIRROR_RESYNC 10 mirrors 000844 1788190548.096778 MIRROR_EXTEND to 11 mirrors 000846 1788190549.204212 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000847 1788190549.812596 fallocate 0x0215d9 thru 0x026adb (0x5503 bytes) 000848 1788190549.910070 write 0x008959 thru 0x00db61 (0x5209 bytes) 000849 1788190549.930628 trunc from 0x040000 to 0x02e57b (0x11a84 bytes) 000850 1788190549.974603 fallocate 0x008d61 thru 0x013f11 (0xb1b1 bytes) 000851 1788190549.990051 MIRROR_RESYNC 10 mirrors 000852 1788190553.146843 trunc from 0x003da9 to 0x00e091 (0xa2e9 bytes) 000853 1788190553.212947 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e57c): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e57c): Mirror 9: 0xdce3580b (stale) Mirror 14: 0xdce3580b (stale) Mirror 15: 0xdce3580b (stale) Mirror 19: 0xdce3580b (stale) Mirror 20: 0xdce3580b (stale) Mirror 22: 0xdce3580b (stale) Mirror 24: 0xdce3580b (stale) Mirror 27: 0xdce3580b (stale) Mirror 28: 0xdce3580b (stale) CRC-32 checksum value for chunk [0, 0x2e57c): Mirror 1: 0xe17b6a5a Mirror 29: 0xe17b6a5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e57c: skipped 000854 1788190554.903099 trunc from 0x02e57c to 0x016162 (0x18419 bytes) 000855 1788190554.992787 mapread 0x005203 thru 0x0064db (0x12d9 bytes) 000856 1788190555.013185 trunc from 0x011a7a to 0x016162 (0x46e9 bytes) 000857 1788190555.029722 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x16163): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16163): Mirror 9: 0x4a84f54e (stale) Mirror 14: 0x4a84f54e (stale) Mirror 15: 0x4a84f54e (stale) Mirror 19: 0x4a84f54e (stale) Mirror 20: 0x4a84f54e (stale) Mirror 22: 0x4a84f54e (stale) Mirror 24: 0x4a84f54e (stale) Mirror 27: 0x4a84f54e (stale) Mirror 28: 0x4a84f54e (stale) Mirror 29: 0x1d3e55ad (stale) CRC-32 checksum value for chunk [0, 0x16163): Mirror 1: 0x1d3e55ad Mirror 30: 0x1d3e55ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16163: skipped 000858 1788190556.265410 read 0x0094d0 thru 0x016162 (0xcc93 bytes) 000860 1788190556.292005 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16163): Mirror 9: 0x4a84f54e (stale) Mirror 14: 0x4a84f54e (stale) Mirror 15: 0x4a84f54e (stale) Mirror 19: 0x4a84f54e (stale) Mirror 20: 0x4a84f54e (stale) Mirror 22: 0x4a84f54e (stale) Mirror 24: 0x4a84f54e (stale) Mirror 27: 0x4a84f54e (stale) Mirror 28: 0x4a84f54e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16163): Mirror 9: 0x4a84f54e (stale) Mirror 14: 0x4a84f54e (stale) Mirror 15: 0x4a84f54e (stale) Mirror 19: 0x4a84f54e (stale) Mirror 20: 0x4a84f54e (stale) Mirror 22: 0x4a84f54e (stale) Mirror 24: 0x4a84f54e (stale) Mirror 27: 0x4a84f54e (stale) Mirror 28: 0x4a84f54e (stale) CRC-32 checksum value for chunk [0, 0x16163): Mirror 1: 0x1d3e55ad Mirror 30: 0x1d3e55ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16163: skipped 000861 1788190557.418373 punch from 0x00b460 to 0x012186 (0x6d27 bytes) 000862 1788190557.514748 read 0x003124 thru 0x0121e3 (0xf0c0 bytes) 000863 1788190557.552264 write 0x01ab1a thru 0x01e559 (0x3a40 bytes) 000864 1788190557.562389 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1e55a): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e55a): Mirror 9: 0x88c6a06 (stale) Mirror 14: 0x88c6a06 (stale) Mirror 15: 0x88c6a06 (stale) Mirror 19: 0x88c6a06 (stale) Mirror 20: 0x88c6a06 (stale) Mirror 22: 0x88c6a06 (stale) Mirror 24: 0x88c6a06 (stale) Mirror 27: 0x88c6a06 (stale) Mirror 28: 0x88c6a06 (stale) Mirror 30: 0xfcb84459 (stale) CRC-32 checksum value for chunk [0, 0x1e55a): Mirror 1: 0xaa75997a Mirror 31: 0xaa75997a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e55a: skipped 000865 1788190558.780135 mapread 0x00e56d thru 0x00fe24 (0x18b8 bytes) 000866 1788190558.796155 trunc from 0x01e55a to 0x01ea8a (0x531 bytes) 000867 1788190558.896856 punch from 0x005924 to 0x00afe7 (0x56c4 bytes) 000868 1788190558.901908 write 0x028434 thru 0x0341f9 (0xbdc6 bytes) 000869 1788190558.910479 read 0x00d6e8 thru 0x019107 (0xba20 bytes) 000871 1788190558.957926 punch from 0x000b51 to 0x00d8d3 (0xcd83 bytes) 000874 1788190558.975708 fallocate 0x002887 thru 0x005167 (0x28e1 bytes) 000875 1788190558.984644 read 0x030fb9 thru 0x0341f9 (0x3241 bytes) 000876 1788190558.995361 write 0x024a59 thru 0x02ad4c (0x62f4 bytes) 000878 1788190559.051181 fallocate 0x02522f thru 0x033053 (0xde25 bytes) 000879 1788190559.080881 punch from 0x0268c7 to 0x033a5c (0xd196 bytes) 000880 1788190559.124846 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x341fa): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341fa): Mirror 9: 0xee5f6dfa (stale) Mirror 14: 0xee5f6dfa (stale) Mirror 15: 0xee5f6dfa (stale) Mirror 19: 0xee5f6dfa (stale) Mirror 20: 0xee5f6dfa (stale) Mirror 22: 0xee5f6dfa (stale) Mirror 24: 0xee5f6dfa (stale) Mirror 27: 0xee5f6dfa (stale) Mirror 28: 0xee5f6dfa (stale) Mirror 30: 0xe005ce80 (stale) Mirror 31: 0x7a8594c3 (stale) CRC-32 checksum value for chunk [0, 0x341fa): Mirror 1: 0x31866d11 Mirror 32: 0x31866d11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341fa: skipped 000881 1788190561.545097 read 0x02c32f thru 0x02ee54 (0x2b26 bytes) 000882 1788190561.563230 fallocate 0x00fc2c thru 0x019cdc (0xa0b1 bytes) 000883 1788190561.715773 trunc from 0x0341fa to 0x02fdbd (0x443c bytes) 000884 1788190561.794477 punch from 0x01c482 to 0x02b4d8 (0xf057 bytes) 000885 1788190561.822996 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2fdbe): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdbe): Mirror 9: 0x9ffefd6d (stale) Mirror 14: 0x9ffefd6d (stale) Mirror 15: 0x9ffefd6d (stale) Mirror 19: 0x9ffefd6d (stale) Mirror 20: 0x9ffefd6d (stale) Mirror 22: 0x9ffefd6d (stale) Mirror 24: 0x9ffefd6d (stale) Mirror 27: 0x9ffefd6d (stale) Mirror 28: 0x9ffefd6d (stale) Mirror 30: 0x2a33efa7 (stale) Mirror 31: 0x85c05204 (stale) Mirror 32: 0xee6832b (stale) CRC-32 checksum value for chunk [0, 0x2fdbe): Mirror 1: 0xaf2cac14 Mirror 33: 0xaf2cac14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdbe: skipped 000886 1788190563.261049 punch from 0x02a1f2 to 0x02da36 (0x3845 bytes) 000888 1788190563.327093 write 0x00e5b1 thru 0x01808c (0x9adc bytes) 000889 1788190563.350739 trunc from 0x00d3ba to 0x015bc6 (0x880d bytes) 000890 1788190563.377648 mapread 0x0235ed thru 0x027056 (0x3a6a bytes) 000891 1788190563.401451 trunc from 0x02fdbe to 0x01ca32 (0x1338b bytes) 000892 1788190563.444675 read 0x01afed thru 0x01ca32 (0x1a46 bytes) 000894 1788190563.452912 trunc from 0x015415 to 0x01b6be (0x62aa bytes) 000895 1788190563.458227 mapread 0x0077b5 thru 0x01225d (0xaaa9 bytes) 000896 1788190563.476076 trunc from 0x019c67 to 0x01c811 (0x2bab bytes) 000897 1788190563.485647 trunc from 0x009638 to 0x017587 (0xdf50 bytes) 000898 1788190563.498111 punch from 0x00a64f to 0x00b694 (0x1046 bytes) 000899 1788190563.514312 trunc from 0x00741c to 0x010454 (0x9039 bytes) 000900 1788190563.534698 MIRROR_RESYNC 14 mirrors 000902 1788190566.262865 fallocate 0x026464 thru 0x036430 (0xffcd bytes) 000903 1788190566.299219 mapwrite 0x01a4f7 thru 0x022b6e (0x8678 bytes) 000904 1788190566.341928 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36431): Mirror 14: 0x24cf5349 (stale) Mirror 15: 0x24cf5349 (stale) Mirror 19: 0x24cf5349 (stale) Mirror 20: 0x24cf5349 (stale) Mirror 22: 0x24cf5349 (stale) Mirror 24: 0x24cf5349 (stale) Mirror 27: 0x24cf5349 (stale) Mirror 28: 0x24cf5349 (stale) Mirror 30: 0x24cf5349 (stale) Mirror 31: 0x24cf5349 (stale) Mirror 32: 0x24cf5349 (stale) Mirror 33: 0x24cf5349 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36431): Mirror 14: 0x24cf5349 (stale) Mirror 15: 0x24cf5349 (stale) Mirror 19: 0x24cf5349 (stale) Mirror 20: 0x24cf5349 (stale) Mirror 22: 0x24cf5349 (stale) Mirror 24: 0x24cf5349 (stale) Mirror 27: 0x24cf5349 (stale) Mirror 28: 0x24cf5349 (stale) Mirror 30: 0x24cf5349 (stale) Mirror 31: 0x24cf5349 (stale) Mirror 32: 0x24cf5349 (stale) Mirror 33: 0x24cf5349 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36431: skipped 000905 1788190567.190241 mapwrite 0x030e9d thru 0x03b5cf (0xa733 bytes) 000906 1788190567.255378 trunc from 0x03b5d0 to 0x03e009 (0x2a3a bytes) 000907 1788190567.275265 mapread 0x039753 thru 0x03e009 (0x48b7 bytes) 000908 1788190567.282608 trunc from 0x038636 to 0x03de8b (0x5856 bytes) 000909 1788190567.287899 trunc from 0x03e00a to 0x03fbc5 (0x1bbc bytes) 000910 1788190567.307074 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbc6): Mirror 14: 0xb4d52632 (stale) Mirror 15: 0xb4d52632 (stale) Mirror 19: 0xb4d52632 (stale) Mirror 20: 0xb4d52632 (stale) Mirror 22: 0xb4d52632 (stale) Mirror 24: 0xb4d52632 (stale) Mirror 27: 0xb4d52632 (stale) Mirror 28: 0xb4d52632 (stale) Mirror 30: 0xb4d52632 (stale) Mirror 31: 0xb4d52632 (stale) Mirror 32: 0xb4d52632 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbc6): Mirror 14: 0xb4d52632 (stale) Mirror 15: 0xb4d52632 (stale) Mirror 19: 0xb4d52632 (stale) Mirror 20: 0xb4d52632 (stale) Mirror 22: 0xb4d52632 (stale) Mirror 24: 0xb4d52632 (stale) Mirror 27: 0xb4d52632 (stale) Mirror 28: 0xb4d52632 (stale) Mirror 30: 0xb4d52632 (stale) Mirror 31: 0xb4d52632 (stale) Mirror 32: 0xb4d52632 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbc6: skipped 000911 1788190567.840058 write 0x0141ad thru 0x01bc89 (0x7add bytes) 000912 1788190567.854047 punch from 0x0300bd to 0x032c95 (0x2bd9 bytes) 000914 1788190567.860730 mapread 0x02d479 thru 0x032eb5 (0x5a3d bytes) 000915 1788190567.883890 fallocate 0x032c78 thru 0x03afb9 (0x8342 bytes) 000916 1788190567.893629 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbc6): Mirror 14: 0xb4d52632 (stale) Mirror 15: 0xb4d52632 (stale) Mirror 19: 0xb4d52632 (stale) Mirror 20: 0xb4d52632 (stale) Mirror 22: 0xb4d52632 (stale) Mirror 24: 0xb4d52632 (stale) Mirror 27: 0xb4d52632 (stale) Mirror 30: 0xb4d52632 (stale) Mirror 31: 0xb4d52632 (stale) Mirror 32: 0xb4d52632 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbc6): Mirror 14: 0xb4d52632 (stale) Mirror 15: 0xb4d52632 (stale) Mirror 19: 0xb4d52632 (stale) Mirror 20: 0xb4d52632 (stale) Mirror 22: 0xb4d52632 (stale) Mirror 24: 0xb4d52632 (stale) Mirror 27: 0xb4d52632 (stale) Mirror 30: 0xb4d52632 (stale) Mirror 31: 0xb4d52632 (stale) Mirror 32: 0xb4d52632 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbc6: skipped 000917 1788190568.577302 trunc from 0x00359d to 0x00e87a (0xb2de bytes) 000918 1788190568.583108 trunc from 0x03fbc6 to 0x03f1c6 (0x9ff bytes) 000919 1788190568.608665 trunc from 0x02f16e to 0x037a71 (0x8904 bytes) 000920 1788190568.619062 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f1c7): Mirror 14: 0x827f5baa (stale) Mirror 15: 0x827f5baa (stale) Mirror 19: 0x827f5baa (stale) Mirror 20: 0x827f5baa (stale) Mirror 22: 0x827f5baa (stale) Mirror 24: 0x827f5baa (stale) Mirror 27: 0x827f5baa (stale) Mirror 31: 0x827f5baa (stale) Mirror 32: 0x827f5baa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1c7): Mirror 14: 0x827f5baa (stale) Mirror 15: 0x827f5baa (stale) Mirror 19: 0x827f5baa (stale) Mirror 20: 0x827f5baa (stale) Mirror 22: 0x827f5baa (stale) Mirror 24: 0x827f5baa (stale) Mirror 27: 0x827f5baa (stale) Mirror 31: 0x827f5baa (stale) Mirror 32: 0x827f5baa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1c7: skipped 000922 1788190569.324389 write 0x020d65 thru 0x0278b3 (0x6b4f bytes) 000923 1788190569.341423 punch from 0x02e967 to 0x036d2c (0x83c6 bytes) 000924 1788190569.346708 read 0x016364 thru 0x016fcb (0xc68 bytes) 000926 1788190569.353384 mapwrite 0x01121d thru 0x01b978 (0xa75c bytes) 000928 1788190569.388327 trunc from 0x02a8d8 to 0x02c072 (0x179b bytes) 000929 1788190569.392727 punch from 0x0071ab to 0x009765 (0x25bb bytes) 000930 1788190569.399477 mapread 0x022e59 thru 0x0315f1 (0xe799 bytes) 000931 1788190569.417008 mapwrite 0x001dfe thru 0x00c159 (0xa35c bytes) 000932 1788190569.451390 read 0x005418 thru 0x009e26 (0x4a0f bytes) 000933 1788190569.452731 write 0x0357c7 thru 0x03ffff (0xa839 bytes) 000935 1788190569.460971 punch from 0x02b895 to 0x03872f (0xce9b bytes) 000936 1788190569.480949 trunc from 0x032cb6 to 0x034d96 (0x20e1 bytes) 000938 1788190569.488228 trunc from 0x040000 to 0x00c64e (0x339b1 bytes) 000939 1788190569.529786 trunc from 0x00c64f to 0x01c128 (0xfada bytes) 000940 1788190569.563296 mapwrite 0x01655d thru 0x01721c (0xcc0 bytes) 000941 1788190569.593267 write 0x006309 thru 0x00f6f6 (0x93ee bytes) 000942 1788190569.596170 write 0x0370eb thru 0x03e1a0 (0x70b6 bytes) 000943 1788190569.598063 mapread 0x02ed9b thru 0x02f1ae (0x414 bytes) 000944 1788190569.605654 mapwrite 0x00d043 thru 0x016f06 (0x9ec4 bytes) 000945 1788190569.629494 read 0x01af5e thru 0x025b60 (0xac03 bytes) 000947 1788190569.631709 write 0x014811 thru 0x02192e (0xd11e bytes) 000948 1788190569.634018 mapwrite 0x015aea thru 0x0229d8 (0xceef bytes) 000950 1788190569.653328 trunc from 0x033be0 to 0x03a055 (0x6476 bytes) 000951 1788190569.658700 fallocate 0x0391c7 thru 0x03ffff (0x6e39 bytes) 000952 1788190569.663055 fallocate 0x015fd0 thru 0x020062 (0xa093 bytes) 000953 1788190569.668845 punch from 0x00eb7b to 0x00ffd1 (0x1457 bytes) 000954 1788190569.674477 mapwrite 0x015070 thru 0x01e519 (0x94aa bytes) 000955 1788190569.693326 write 0x018aa3 thru 0x01e101 (0x565f bytes) 000956 1788190569.694665 punch from 0x037469 to 0x03d88d (0x6425 bytes) 000957 1788190569.700461 write 0x02832d thru 0x0377c1 (0xf495 bytes) 000958 1788190569.709971 trunc from 0x013d3f to 0x02163b (0xd8fd bytes) 000959 1788190569.724938 write 0x038103 thru 0x03f9ac (0x78aa bytes) 000960 1788190569.734214 write 0x014856 thru 0x017051 (0x27fc bytes) 000961 1788190569.745750 mapwrite 0x01800f thru 0x018b04 (0xaf6 bytes) 000962 1788190569.790369 fallocate 0x038696 thru 0x039f11 (0x187c bytes) 000963 1788190569.820039 trunc from 0x02e86f to 0x030214 (0x19a6 bytes) 000964 1788190569.829874 mapwrite 0x01642b thru 0x023e83 (0xda59 bytes) 000965 1788190569.990833 trunc from 0x00cfe9 to 0x018aa8 (0xbac0 bytes) 000966 1788190569.996458 trunc from 0x03f9ad to 0x00f63d (0x3036f bytes) 000967 1788190570.019847 mapread 0x00a545 thru 0x00f63d (0x50f9 bytes) 000968 1788190570.061123 punch from 0x009a5e to 0x00b241 (0x17e4 bytes) 000969 1788190570.071582 read 0x00756f thru 0x00f63d (0x80cf bytes) 000970 1788190570.079036 write 0x03fc2f thru 0x03ffff (0x3d1 bytes) 000971 1788190570.081456 mapread 0x0255d7 thru 0x02e95e (0x9388 bytes) 000972 1788190570.101994 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 31: 0xe20eea22 (stale) Mirror 32: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 31: 0xe20eea22 (stale) Mirror 32: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000973 1788190570.607020 fallocate 0x00042d thru 0x00eda1 (0xe975 bytes) 000975 1788190570.615049 write 0x020182 thru 0x02096c (0x7eb bytes) 000976 1788190570.625450 trunc from 0x02e1dd to 0x034eac (0x6cd0 bytes) 000977 1788190570.630681 write 0x0204e6 thru 0x021975 (0x1490 bytes) 000978 1788190570.637437 read 0x0030c3 thru 0x007717 (0x4655 bytes) 000980 1788190570.649796 read 0x002212 thru 0x007a96 (0x5885 bytes) 000981 1788190570.671288 MIRROR_RESYNC 9 mirrors 000982 1788190573.384109 mapwrite 0x0042be thru 0x0121e7 (0xdf2a bytes) 000983 1788190573.506277 write 0x02819a thru 0x028b88 (0x9ef bytes) 000984 1788190573.515293 write 0x015bdc thru 0x02173f (0xbb64 bytes) 000985 1788190573.522501 mapread 0x0065fb thru 0x015af2 (0xf4f8 bytes) 000986 1788190573.525276 trunc from 0x03d0d4 to 0x03ffff (0x2f2c bytes) 000987 1788190573.531110 read 0x00ab38 thru 0x019d0f (0xf1d8 bytes) 000988 1788190573.532433 punch from 0x03b637 to 0x03ffff (0x49c9 bytes) 000989 1788190573.538574 trunc from 0x040000 to 0x015e3c (0x2a1c3 bytes) 000990 1788190573.570100 punch from 0x00bb3b to 0x0111da (0x56a0 bytes) 000991 1788190573.575021 fallocate 0x01f3ca thru 0x0204a2 (0x10d9 bytes) 000992 1788190573.583578 trunc from 0x0104bb to 0x015e3c (0x5982 bytes) 000993 1788190573.607008 fallocate 0x020836 thru 0x02d463 (0xcc2e bytes) 000994 1788190573.621036 fallocate 0x0247dc thru 0x0320cd (0xd8f2 bytes) 000996 1788190573.632652 punch from 0x0027ba to 0x011dbc (0xf603 bytes) 000997 1788190573.641965 trunc from 0x012d16 to 0x015e3c (0x3127 bytes) 000998 1788190573.652431 trunc from 0x015e3d to 0x01adde (0x4fa2 bytes) 000999 1788190573.672092 mapwrite 0x039211 thru 0x03ffff (0x6def bytes) 001000 1788190573.757632 MIRROR_RESYNC 9 mirrors All operations completed A-OK! PASS 70a (149s) == sanity-flr test 71: check mirror extend parent fid ==== 11:36:21 (1788190581) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x17e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72a: check append on FLR file with setstripe ========================================================== 11:36:32 (1788190592) striped dir -i0 -c2 -H all_char /mnt/lustre/d72a.sanity-flr layout after create /mnt/lustre/d72a.sanity-flr/f72a.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154309 s, 13.6 MB/s layout after 2M append+resync /mnt/lustre/d72a.sanity-flr/f72a.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258677 s, 4.1 MB/s /mnt/lustre/d72a.sanity-flr/f72a.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 layout after 1M append+resync /mnt/lustre/d72a.sanity-flr/f72a.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72a (10s) == sanity-flr test 72b: check append on FLR file with open(O_APPEND) ========================================================== 11:36:42 (1788190602) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d72b.sanity-flr layout after create /mnt/lustre/d72b.sanity-flr/f72b.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192148 s, 10.9 MB/s lfs mirror mirror: fail to ftruncate mirror 2 to 2097152: Input/output error (5) layout after 2M append+resync /mnt/lustre/d72b.sanity-flr/f72b.sanity-flr lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11199 s, 9.4 MB/s /mnt/lustre/d72b.sanity-flr/f72b.sanity-flr lcm_layout_gen: 14 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 layout after 1M append+resync /mnt/lustre/d72b.sanity-flr/f72b.sanity-flr lcm_layout_gen: 16 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72b (9s) == sanity-flr test 200a: stress test ===================== 11:36:51 (1788190611) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8188128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7463424 .. done Extending file size to 9243872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9109344 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62062 62063 62065 62067 62068 failed 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:38:15 (1788190695) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2346624 .. Extending file size to 1076128 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 1356192 .. Extending file size to 5445280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5001216 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7582784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7417536 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8695360 .. Extending file size to 8936448 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8971168 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9114272 .. Extending file size to 8986016 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9201344 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9363200 .. Waiting 63704 63705 63706 63708 63709 done 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:39:37 (1788190777) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 28152 3578868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22056 3584964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50208 7163832 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 28152 3578868 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22056 3584964 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 50208 7163832 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150878 s, 13.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197412 s, 10.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x25f27187 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:39:58 (1788190798) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130317 s, 16.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:40:05 (1788190805) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:40:12 (1788190812) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0673033 s, 31.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x205:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:40:20 (1788190820) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0726178 s, 28.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:40:27 (1788190827) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0503038 s, 41.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:41:00 (1788190860) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839562 s, 12.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x207:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x206:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x207:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x206:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:41:53 (1788190913) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247265 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:42:44 (1788190964) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg127-server: Pool lustre.test_204f created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0952007 s, 22.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg127-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:44:08 (1788191048) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:44:17 (1788191057) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:44:27 (1788191067) oleg127-server: Pool lustre.test_206 created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg127-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:44:57 (1788191097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143308 s, 7.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339063 s, 30.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x48c4fe24 Mirror 3: 0x48c4fe24 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:45:11 (1788191111) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:45:15 (1788191115) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:45:20 (1788191120) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0726359 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2240000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1db0000 Read this (4 bytes): BBBB 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,noappend_layout,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:46:04 (1788191164) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0165502 s, 495 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:46:12 (1788191172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834143 s, 12.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611333 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 11:46:14.000000000 -0400 Modify: 2026-08-31 11:46:14.000000000 -0400 Change: 2026-08-31 11:46:15.000000000 -0400 Birth: 2026-08-31 11:46:14.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:46:22 (1788191182) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021787 s, 48.1 MB/s oleg127-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg127-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg127-server: O/280000401/d1/161: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:46:43 (1788191203) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.265796 s, 154 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177728 s, 230 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 11:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xca:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 11:46 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:46:51 (1788191211) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:47:00 (1788191220) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1988 sec =========== 11:47:10 (1788191230) === sanity-flr: start cleanup 11:47:12 (1788191232) === === sanity-flr: finish cleanup 11:47:21 (1788191241) ===