-----============= acceptance-small: sanity-flr ============----- Sat Aug 8 01:05:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:05:26 (1786165526) === oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8df005236800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8df005236800.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:06:10 (1786165570) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8260] == sanity-flr test 0a: lfs mirror create with -N option == 01:06:12 (1786165572) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:06:33 (1786165593) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:06:38 (1786165598) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:06:42 (1786165602) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:07:10 (1786165630) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:07:14 (1786165634) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:07:18 (1786165638) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8df005236800.stats=clear osc.lustre-OST0001-osc-ffff8df005236800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:07:28 (1786165648) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:07:41 (1786165661) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:07:50 (1786165670) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:07:55 (1786165675) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:08:03 (1786165683) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:08:13 (1786165693) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:08:21 (1786165701) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:08:29 (1786165709) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:08:37 (1786165717) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:08:48 (1786165728) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:08:56 (1786165736) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.409229 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105174 s, 10.0 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:09:04 (1786165744) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123826 s, 16.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:09:13 (1786165753) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:09:26 (1786165766) Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:10:49 (1786165849) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.361056 s, 17.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:11:38 (1786165898) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:13:06 (1786165986) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.73765 s, 19.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 8 01:13 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.114s user 0m0.000s sys 0m0.200s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.628s user 0m0.001s sys 0m0.221s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.134s user 0m0.001s sys 0m0.158s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df005236800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:14:47 (1786166087) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:14:51 (1786166091) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:14:54 (1786166094) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:14:57 (1786166097) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 01:15:05 (1786166105) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.630189 s, 20.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:15:18 (1786166118) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215491 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597583 s, 17.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:15:25 (1786166125) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:15:35 (1786166135) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101371 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00140246 s, 730 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00247828 s, 413 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00426347 s, 240 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 01:15:47 (1786166147) create test files with size 15758760 .. 765bfe6f8ca9d0ce8cc49e1105bac114 - 2fa04b85f5b9399fd80822dda163d4ae - 4274e404ac8ecb3ebd35847cfecc2cff - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 01:16:12 (1786166172) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:16:30 (1786166190) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:16:37 (1786166197) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103715 s, 10.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491824 s, 21.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 01:16:47 (1786166207) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0872092 s, 24.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.457273 s, 9.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139565 s, 15.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 01:17:04 (1786166224) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 01:17:09 (1786166229) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:17:13 (1786166233) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275263 s, 38.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 01:17:20 (1786166240) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177908 s, 17.7 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:17:32 (1786166252) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:17:44 (1786166264) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281471 s, 37.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:17:59 (1786166279) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8df00ee95800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.459991 s, 22.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8df025100800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:18:17 (1786166297) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:18:58 (1786166338) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:19:36 (1786166376) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:19:39 (1786166379) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223816 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.102976 s, 30.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xafbcc4bd (nosync,stale) Mirror 2: 0xafbcc4bd (nosync,stale) ** verify mirror 2 doesn't change original checksum: adbc9409af34c12d751c3a8fe10bb1b1 - mirror 1 checksum: 87ec23dfe350656e35c78746bd7822b4 - mirror 2 checksum: adbc9409af34c12d751c3a8fe10bb1b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T01:19:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T01:19:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: adbc9409af34c12d751c3a8fe10bb1b1 - mirror 1 checksum: 87ec23dfe350656e35c78746bd7822b4 - mirror 2 checksum: 87ec23dfe350656e35c78746bd7822b4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T01:19:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T01:19:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:19:50 (1786166390) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030257 s, 34.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:19:58 (1786166398) Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:20:13 (1786166413) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:44 (1786166444) targets are mounted 01:20:44 (1786166444) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556105 s, 18.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:21:03 (1786166463) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0204375 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00952785 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:21:16 (1786166476) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.100225 s, 204 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0155663 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:21:26 (1786166486) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00878225 s, 1.9 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:21:37 (1786166497) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.072367 s, 283 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:21:47 (1786166507) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00668778 s, 612 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0264329 s, 155 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:21:56 (1786166516) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:22:25 (1786166545) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:22:56 (1786166576) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:23:19 (1786166599) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:23:45 (1786166625) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:23:54 (1786166634) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57625 mirror split pid 57626 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57625 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57626 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:25:03 (1786166703) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6706 fallocating to largest ever: 0x39aa4 000001 1786166706.706854 fallocate 0x0373f8 thru 0x039aa3 (0x26ac bytes) 000002 1786166706.725209 fallocate 0x01f848 thru 0x02ae29 (0xb5e2 bytes) 000003 1786166706.731822 mapread 0x024952 thru 0x02fe05 (0xb4b4 bytes) 000004 1786166706.782777 trunc from 0x0011c6 to 0x002e97 (0x1cd2 bytes) 000007 1786166706.794347 fallocate 0x020de1 thru 0x022c09 (0x1e29 bytes) 000009 1786166706.804808 fallocate 0x027587 thru 0x02824c (0xcc6 bytes) 000010 1786166706.815151 mapwrite 0x0029bf thru 0x00e451 (0xba93 bytes) 000011 1786166706.909839 punch from 0x032cea to 0x039aa3 (0x6dba bytes) 000012 1786166706.916400 read 0x008fae thru 0x00b548 (0x259b bytes) 000013 1786166706.918269 trunc from 0x02e5df to 0x030769 (0x218b bytes) 000016 1786166706.963595 trunc from 0x039aa4 to 0x031d06 (0x7d9d bytes) 000017 1786166707.001807 read 0x023694 thru 0x027a6c (0x43d9 bytes) 000018 1786166707.003689 mapread 0x029826 thru 0x030905 (0x70e0 bytes) 000019 1786166707.019109 mapwrite 0x025daa thru 0x032656 (0xc8ad bytes) 000020 1786166707.107383 trunc from 0x032657 to 0x01e4fc (0x1415a bytes) 000021 1786166707.135325 mapread 0x01640c thru 0x01e4fc (0x80f1 bytes) 000022 1786166707.137097 trunc from 0x01e4fd to 0x011cd2 (0xc82a bytes) 000023 1786166707.165591 mapwrite 0x00a720 thru 0x00a747 (0x28 bytes) 000024 1786166707.189650 MIRROR_EXTEND to 2 mirrors 000025 1786166707.976753 read 0x00bf29 thru 0x011cd2 (0x5daa bytes) 000026 1786166707.986240 trunc from 0x002473 to 0x00c125 (0x9cb3 bytes) 000027 1786166708.028355 read 0x00ef36 thru 0x011cd2 (0x2d9d bytes) 000028 1786166708.040160 mapwrite 0x00bfd9 thru 0x00f1f6 (0x321e bytes) 000029 1786166708.058198 punch from 0x00c320 to 0x011cd2 (0x59b3 bytes) 000030 1786166708.063457 trunc from 0x001e76 to 0x00970e (0x7899 bytes) 000031 1786166708.068643 trunc from 0x011cd3 to 0x016b8b (0x4eb9 bytes) 000033 1786166708.119796 trunc from 0x001611 to 0x0018dd (0x2cd bytes) 000034 1786166708.124461 mapwrite 0x014d4d thru 0x0228fb (0xdbaf bytes) 000035 1786166708.182220 mapwrite 0x032670 thru 0x03803e (0x59cf bytes) 000038 1786166708.231108 write 0x0220b7 thru 0x02234b (0x295 bytes) 000039 1786166708.232378 punch from 0x0295fa to 0x02e496 (0x4e9d bytes) 000040 1786166708.238370 mapwrite 0x015431 thru 0x016165 (0xd35 bytes) 000041 1786166708.253311 mapread 0x03397a thru 0x03803e (0x46c5 bytes) 000042 1786166708.255505 mapwrite 0x00c6c0 thru 0x00f2af (0x2bf0 bytes) 000043 1786166708.293704 mapwrite 0x004a56 thru 0x00cc18 (0x81c3 bytes) 000044 1786166708.353785 mapwrite 0x02f4c1 thru 0x035336 (0x5e76 bytes) 000045 1786166708.372456 punch from 0x00b40b to 0x00bfa6 (0xb9c bytes) 000046 1786166708.377828 mapread 0x01334d thru 0x01368f (0x343 bytes) 000047 1786166708.378935 mapread 0x025799 thru 0x02592b (0x193 bytes) 000048 1786166708.381336 trunc from 0x021ca1 to 0x02aa27 (0x8d87 bytes) 000049 1786166708.386606 read 0x02b93c thru 0x0375b6 (0xbc7b bytes) 000051 1786166708.397296 fallocate 0x00fe86 thru 0x01fbcf (0xfd4a bytes) 000052 1786166708.402385 write 0x01e397 thru 0x02784e (0x94b8 bytes) 000054 1786166708.404313 mapwrite 0x010e82 thru 0x014fd2 (0x4151 bytes) 000055 1786166708.427396 trunc from 0x02cb06 to 0x02fefe (0x33f9 bytes) 000056 1786166708.435255 trunc from 0x03444f to 0x03803e (0x3bf0 bytes) 000057 1786166708.441308 fallocate 0x013cef thru 0x01c76d (0x8a7f bytes) 000058 1786166708.447757 mapwrite 0x009771 thru 0x01807f (0xe90f bytes) 000059 1786166708.484458 read 0x011377 thru 0x01ba26 (0xa6b0 bytes) 000060 1786166708.485451 mapread 0x0105c4 thru 0x016ef9 (0x6936 bytes) 000061 1786166708.487774 trunc from 0x03803f to 0x02350c (0x14b32 bytes) 000062 1786166708.509295 punch from 0x0096ea to 0x01611d (0xca34 bytes) 000063 1786166708.517741 trunc from 0x02350d to 0x02b047 (0x7b3b bytes) 000064 1786166708.537312 mapread 0x016880 thru 0x01c657 (0x5dd8 bytes) 000065 1786166708.554355 write 0x03821d thru 0x03ffff (0x7de3 bytes) 000066 1786166708.555809 punch from 0x01e0d0 to 0x02541d (0x734e bytes) 000067 1786166708.561518 punch from 0x035a60 to 0x03ffff (0xa5a0 bytes) 000068 1786166708.578646 read 0x023c63 thru 0x029422 (0x57c0 bytes) 000069 1786166708.589216 read 0x033963 thru 0x03ed00 (0xb39e bytes) 000070 1786166708.589935 read 0x016d0b thru 0x01dacd (0x6dc3 bytes) 000071 1786166708.594511 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1636f7a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26025ad Mirror 3: 0x26025ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000072 1786166709.368323 punch from 0x024cca to 0x02ea52 (0x9d89 bytes) 000073 1786166709.411877 mapread 0x02d178 thru 0x02f2e9 (0x2172 bytes) 000074 1786166709.438101 mapwrite 0x0231d3 thru 0x029289 (0x60b7 bytes) 000075 1786166709.469470 punch from 0x0093d5 to 0x011622 (0x824e bytes) 000076 1786166709.477471 trunc from 0x040000 to 0x00a7d7 (0x35828 bytes) 000077 1786166709.534991 mapread 0x00a4d0 thru 0x00a7d7 (0x308 bytes) 000078 1786166709.550829 write 0x0393a2 thru 0x03a5a6 (0x1205 bytes) 000079 1786166709.552388 trunc from 0x020642 to 0x02daea (0xd4a9 bytes) 000080 1786166709.558313 trunc from 0x03a5a7 to 0x00a393 (0x30213 bytes) 000081 1786166709.580021 trunc from 0x0052e3 to 0x00a393 (0x50b1 bytes) 000082 1786166709.586504 trunc from 0x00a394 to 0x02fa8c (0x256f9 bytes) 000083 1786166709.601255 mapread 0x00e614 thru 0x00eef1 (0x8de bytes) 000084 1786166709.612217 trunc from 0x012bed to 0x020a7a (0xde8e bytes) 000085 1786166709.617619 trunc from 0x009d31 to 0x0195fb (0xf8cb bytes) 000086 1786166709.623409 trunc from 0x019117 to 0x0275d8 (0xe4c2 bytes) 000088 1786166709.629746 trunc from 0x023682 to 0x02e49b (0xae1a bytes) 000089 1786166709.635415 punch from 0x016b36 to 0x0230f4 (0xc5bf bytes) 000090 1786166709.640384 mapwrite 0x027cb1 thru 0x035b80 (0xded0 bytes) 000091 1786166709.699198 read 0x027b88 thru 0x031c62 (0xa0db bytes) 000092 1786166709.700120 read 0x027048 thru 0x02dd07 (0x6cc0 bytes) 000093 1786166709.701430 trunc from 0x035b81 to 0x02d00d (0x8b73 bytes) 000094 1786166709.716038 mapwrite 0x025a5c thru 0x0320ed (0xc692 bytes) 000095 1786166709.773176 write 0x0316a9 thru 0x037fb7 (0x690f bytes) 000096 1786166709.774578 mapwrite 0x02971d thru 0x035dd9 (0xc6bd bytes) 000097 1786166709.800909 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 3: 0xbcd11a52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 3: 0xbcd11a52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb8: skipped 000099 1786166710.227224 write 0x009010 thru 0x012651 (0x9642 bytes) 000100 1786166710.259386 read 0x0223dc thru 0x029972 (0x7597 bytes) 000101 1786166710.283661 read 0x02e321 thru 0x032adf (0x47bf bytes) 000102 1786166710.297332 mapwrite 0x02809d thru 0x033d6c (0xbcd0 bytes) 000104 1786166710.365971 write 0x03aefe thru 0x03ea9e (0x3ba1 bytes) 000105 1786166710.367336 fallocate 0x02467a thru 0x03359e (0xef25 bytes) fallocating to largest ever: 0x40000 000107 1786166710.380802 fallocate 0x03b295 thru 0x03ffff (0x4d6b bytes) 000108 1786166710.399492 mapwrite 0x00300d thru 0x0118cf (0xe8c3 bytes) 000109 1786166710.449700 read 0x01695d thru 0x0212ff (0xa9a3 bytes) 000110 1786166710.450569 punch from 0x002a38 to 0x00d158 (0xa721 bytes) 000111 1786166710.458737 fallocate 0x01aeb8 thru 0x01e207 (0x3350 bytes) 000114 1786166710.472106 read 0x003e0b thru 0x004f84 (0x117a bytes) 000115 1786166710.488693 mapread 0x011989 thru 0x01a324 (0x899c bytes) 000116 1786166710.490209 mapwrite 0x02d87e thru 0x034b3a (0x72bd bytes) 000117 1786166710.551034 mapread 0x03aced thru 0x03d30f (0x2623 bytes) 000118 1786166710.558616 fallocate 0x02ba51 thru 0x032438 (0x69e8 bytes) 000119 1786166710.570897 punch from 0x0393a7 to 0x03ffff (0x6c59 bytes) 000120 1786166710.588490 punch from 0x0105bf to 0x01841d (0x7e5f bytes) 000121 1786166710.597389 read 0x000f04 thru 0x0104b1 (0xf5ae bytes) 000122 1786166710.624688 trunc from 0x01a7b5 to 0x020f62 (0x67ae bytes) 000123 1786166710.631456 fallocate 0x02dc84 thru 0x0319df (0x3d5c bytes) 000124 1786166710.638374 fallocate 0x027ef3 thru 0x031d63 (0x9e71 bytes) 000125 1786166710.657386 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000126 1786166710.870247 mapread 0x02ecb8 thru 0x036f2f (0x8278 bytes) 000127 1786166710.905837 fallocate 0x01e851 thru 0x01f6d8 (0xe88 bytes) 000128 1786166710.914354 read 0x0397da thru 0x03e888 (0x50af bytes) 000129 1786166710.926802 trunc from 0x040000 to 0x02ff4c (0x100b3 bytes) 000130 1786166710.964364 write 0x0177be thru 0x0211bb (0x99fe bytes) 000131 1786166710.978490 punch from 0x010943 to 0x011b73 (0x1231 bytes) 000132 1786166710.984677 mapread 0x0032f0 thru 0x0091bd (0x5ece bytes) 000134 1786166711.000581 trunc from 0x007104 to 0x008e8d (0x1d8a bytes) 000135 1786166711.007318 mapwrite 0x0354b0 thru 0x03ffff (0xab50 bytes) 000136 1786166711.140305 read 0x002737 thru 0x0124a5 (0xfd6f bytes) 000137 1786166711.156370 fallocate 0x02d763 thru 0x03c6ed (0xef8b bytes) 000138 1786166711.162729 punch from 0x009989 to 0x00bd26 (0x239e bytes) 000139 1786166711.177214 punch from 0x0079eb to 0x00876d (0xd83 bytes) 000141 1786166711.197103 write 0x03c6a6 thru 0x03ffff (0x395a bytes) 000142 1786166711.204958 mapread 0x01c155 thru 0x02b001 (0xeead bytes) 000143 1786166711.207454 mapread 0x02e894 thru 0x02eabe (0x22b bytes) 000144 1786166711.208622 mapwrite 0x00fe42 thru 0x0115fc (0x17bb bytes) 000145 1786166711.262429 MIRROR_EXTEND to 2 mirrors 000146 1786166712.046172 MIRROR_RESYNC 2 mirrors 000147 1786166712.364628 mapwrite 0x026e3b thru 0x028909 (0x1acf bytes) 000148 1786166712.486020 write 0x02b481 thru 0x036df6 (0xb976 bytes) 000149 1786166712.515111 read 0x0270a3 thru 0x03349c (0xc3fa bytes) 000150 1786166712.517115 trunc from 0x040000 to 0x033218 (0xcde7 bytes) 000151 1786166712.598245 fallocate 0x011d83 thru 0x01e9fa (0xcc78 bytes) 000154 1786166712.623337 read 0x00335a thru 0x006dab (0x3a52 bytes) 000155 1786166712.639166 mapread 0x01c62b thru 0x029da0 (0xd776 bytes) 000156 1786166712.644841 mapread 0x027c81 thru 0x0310b8 (0x9438 bytes) 000157 1786166712.654805 mapread 0x031017 thru 0x032ff3 (0x1fdd bytes) 000158 1786166712.663673 trunc from 0x007317 to 0x00c58b (0x5275 bytes) 000159 1786166712.673139 mapwrite 0x010c73 thru 0x016791 (0x5b1f bytes) 000160 1786166712.713009 write 0x027707 thru 0x02797d (0x277 bytes) 000161 1786166712.720781 write 0x0218b2 thru 0x02cd01 (0xb450 bytes) 000162 1786166712.730308 trunc from 0x033219 to 0x0174d5 (0x1bd43 bytes) 000163 1786166712.775401 fallocate 0x00ef35 thru 0x010297 (0x1363 bytes) 000164 1786166712.795098 read 0x00f72c thru 0x016b3b (0x7410 bytes) 000165 1786166712.813487 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000166 1786166713.019367 punch from 0x011217 to 0x0174d5 (0x62bf bytes) 000167 1786166713.025778 mapread 0x011ee0 thru 0x012c29 (0xd4a bytes) 000168 1786166713.037475 read 0x00d471 thru 0x0111ec (0x3d7c bytes) 000169 1786166713.044622 read 0x016981 thru 0x0174d5 (0xb55 bytes) 000170 1786166713.055321 trunc from 0x003f2c to 0x013bc2 (0xfc97 bytes) 000171 1786166713.064182 mapread 0x012d19 thru 0x0174d5 (0x47bd bytes) 000173 1786166713.078694 trunc from 0x007327 to 0x0087ad (0x1487 bytes) 000174 1786166713.085705 write 0x01011f thru 0x0172c3 (0x71a5 bytes) 000175 1786166713.098773 trunc from 0x001582 to 0x002b81 (0x1600 bytes) 000176 1786166713.104765 trunc from 0x0174d6 to 0x004367 (0x1316e bytes) 000178 1786166713.124144 fallocate 0x010e9a thru 0x020276 (0xf3dd bytes) 000179 1786166713.130021 trunc from 0x0009ee to 0x004367 (0x397a bytes) 000180 1786166713.135734 mapwrite 0x02d3e5 thru 0x033787 (0x63a3 bytes) 000182 1786166713.181343 trunc from 0x033788 to 0x02d975 (0x5e12 bytes) 000183 1786166713.202815 read 0x016517 thru 0x025f3d (0xfa27 bytes) 000184 1786166713.213293 trunc from 0x003135 to 0x00fc93 (0xcb5f bytes) 000185 1786166713.218993 write 0x00dc7e thru 0x019c39 (0xbfbc bytes) 000186 1786166713.225431 read 0x029a96 thru 0x02d975 (0x3ee0 bytes) 000187 1786166713.226165 trunc from 0x02a986 to 0x02d975 (0x2ff0 bytes) 000188 1786166713.233268 punch from 0x0247d6 to 0x02b750 (0x6f7b bytes) 000189 1786166713.239557 trunc from 0x02c438 to 0x02d56e (0x1137 bytes) 000190 1786166713.244917 mapwrite 0x03a3c0 thru 0x03ffff (0x5c40 bytes) 000191 1786166713.359573 mapwrite 0x002a21 thru 0x00a38c (0x796c bytes) 000192 1786166713.385789 trunc from 0x040000 to 0x02e301 (0x11cfe bytes) 000193 1786166713.413992 trunc from 0x02e302 to 0x01738c (0x16f75 bytes) 000194 1786166713.445696 read 0x00415b thru 0x00f25a (0xb100 bytes) 000195 1786166713.447238 fallocate 0x03a6ce thru 0x03aaf2 (0x425 bytes) 000197 1786166713.461320 write 0x000abe thru 0x0089f1 (0x7f34 bytes) 000198 1786166713.465015 mapwrite 0x03f28c thru 0x03ff5e (0xcd3 bytes) 000199 1786166713.537101 write 0x02ea5a thru 0x034abc (0x6063 bytes) 000200 1786166713.555089 MIRROR_EXTEND to 2 mirrors 000201 1786166714.628171 mapwrite 0x034803 thru 0x0364b1 (0x1caf bytes) 000203 1786166714.804316 mapwrite 0x0032a3 thru 0x0053a7 (0x2105 bytes) 000205 1786166714.897749 MIRROR_RESYNC 2 mirrors 000206 1786166715.837713 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000207 1786166716.058028 mapwrite 0x00a498 thru 0x013ba9 (0x9712 bytes) 000208 1786166716.153702 trunc from 0x03ff5f to 0x036d4a (0x9214 bytes) 000209 1786166716.211771 mapwrite 0x035932 thru 0x036fb2 (0x1681 bytes) 000210 1786166716.325543 punch from 0x01439e to 0x01edbc (0xaa1f bytes) 000211 1786166716.359760 mapread 0x0015bc thru 0x003586 (0x1fcb bytes) 000212 1786166716.426418 mapwrite 0x01c8a8 thru 0x02c5d6 (0xfd2f bytes) 000213 1786166716.562404 trunc from 0x036fb3 to 0x03892a (0x1978 bytes) 000214 1786166716.649666 write 0x016ca8 thru 0x01f1cc (0x8525 bytes) 000215 1786166716.652349 punch from 0x030359 to 0x0386d0 (0x8378 bytes) 000217 1786166716.671240 mapread 0x0092c1 thru 0x0109dc (0x771c bytes) 000218 1786166716.673101 mapread 0x02ef21 thru 0x03892a (0x9a0a bytes) 000219 1786166716.732739 read 0x0195ed thru 0x01b17d (0x1b91 bytes) 000220 1786166716.736126 trunc from 0x018ff1 to 0x0218f4 (0x8904 bytes) 000221 1786166716.777813 mapwrite 0x00be77 thru 0x014e7e (0x9008 bytes) 000222 1786166716.839259 write 0x037102 thru 0x03e418 (0x7317 bytes) 000223 1786166716.842657 fallocate 0x03b72e thru 0x03ea53 (0x3326 bytes) 000224 1786166716.848573 punch from 0x006e66 to 0x0074af (0x64a bytes) 000225 1786166716.854578 write 0x012e2c thru 0x015198 (0x236d bytes) 000226 1786166716.856343 mapread 0x0152e7 thru 0x01e1ce (0x8ee8 bytes) 000227 1786166716.867409 mapwrite 0x025b5f thru 0x03260b (0xcaad bytes) 000229 1786166716.893755 fallocate 0x03722f thru 0x03ffff (0x8dd1 bytes) 000230 1786166716.899167 trunc from 0x040000 to 0x025971 (0x1a68e bytes) 000231 1786166716.925710 punch from 0x0190fc to 0x02256f (0x9474 bytes) 000232 1786166716.932775 mapread 0x01b2bd thru 0x01c952 (0x1696 bytes) 000235 1786166716.946069 mapwrite 0x017b34 thru 0x023906 (0xbdd3 bytes) 000236 1786166716.963072 trunc from 0x016ca5 to 0x01efde (0x833a bytes) 000237 1786166716.967463 punch from 0x003a18 to 0x00b0d9 (0x76c2 bytes) 000239 1786166716.972357 read 0x00b379 thru 0x00ecb5 (0x393d bytes) 000240 1786166716.978689 mapread 0x01ab37 thru 0x0229d2 (0x7e9c bytes) 000241 1786166716.979692 punch from 0x00a0fa to 0x0140c6 (0x9fcd bytes) 000242 1786166716.983778 trunc from 0x017ba1 to 0x0199bc (0x1e1c bytes) 000244 1786166717.006858 mapread 0x00f480 thru 0x01a336 (0xaeb7 bytes) 000245 1786166717.082557 mapread 0x0237a8 thru 0x025971 (0x21ca bytes) 000247 1786166717.085744 mapwrite 0x00c38c thru 0x01c374 (0xffe9 bytes) 000248 1786166717.166113 write 0x00ba40 thru 0x01225b (0x681c bytes) 000249 1786166717.182614 fallocate 0x020119 thru 0x023f28 (0x3e10 bytes) 000250 1786166717.191206 fallocate 0x00eefb thru 0x014cd2 (0x5dd8 bytes) 000252 1786166717.203324 read 0x006684 thru 0x00f1ba (0x8b37 bytes) 000253 1786166717.215570 trunc from 0x015b35 to 0x01766b (0x1b37 bytes) 000254 1786166717.226343 punch from 0x014ef7 to 0x023f59 (0xf063 bytes) 000255 1786166717.232727 trunc from 0x00515c to 0x00cb4a (0x79ef bytes) 000257 1786166717.259725 punch from 0x0106e3 to 0x01d830 (0xd14e bytes) 000258 1786166717.270227 trunc from 0x018b57 to 0x025971 (0xce1b bytes) 000261 1786166717.279262 read 0x00c15d thru 0x011139 (0x4fdd bytes) 000263 1786166717.309418 trunc from 0x025972 to 0x01061f (0x15352 bytes) 000264 1786166717.337899 mapwrite 0x0119f1 thru 0x0122a9 (0x8b9 bytes) 000265 1786166717.428178 mapwrite 0x02a1cb thru 0x036104 (0xbf3a bytes) 000266 1786166717.555858 punch from 0x000c15 to 0x007501 (0x68ed bytes) 000268 1786166717.562822 fallocate 0x009120 thru 0x00d522 (0x4403 bytes) 000270 1786166717.570053 trunc from 0x036105 to 0x01b841 (0x1a8c3 bytes) 000272 1786166717.592918 punch from 0x00c312 to 0x01a7e6 (0xe4d5 bytes) 000273 1786166717.599356 fallocate 0x029d46 thru 0x033653 (0x990e bytes) 000274 1786166717.607249 punch from 0x002b77 to 0x004b65 (0x1fef bytes) 000275 1786166717.613120 trunc from 0x01160b to 0x012b62 (0x1558 bytes) 000276 1786166717.620515 trunc from 0x00f240 to 0x01a296 (0xb057 bytes) 000277 1786166717.626841 mapwrite 0x02d6d5 thru 0x0370a2 (0x99ce bytes) 000281 1786166717.741128 write 0x039594 thru 0x03fa72 (0x64df bytes) 000283 1786166717.745028 mapwrite 0x026361 thru 0x03397d (0xd61d bytes) 000284 1786166717.816036 fallocate 0x02de6c thru 0x03ab4a (0xccdf bytes) 000285 1786166717.824992 trunc from 0x03fa73 to 0x02b70c (0x14366 bytes) 000286 1786166717.868229 mapread 0x00fef3 thru 0x010b7a (0xc88 bytes) 000287 1786166717.885196 trunc from 0x02b70d to 0x030233 (0x4b27 bytes) 000288 1786166717.908361 read 0x0091c8 thru 0x009428 (0x261 bytes) 000289 1786166717.913574 trunc from 0x0168e9 to 0x0201e8 (0x9900 bytes) 000290 1786166717.932041 mapwrite 0x02a9a9 thru 0x02faec (0x5144 bytes) 000291 1786166717.999643 read 0x00b0e3 thru 0x0112c1 (0x61df bytes) 000292 1786166718.000868 write 0x00ba99 thru 0x01a4ba (0xea22 bytes) 000293 1786166718.006661 punch from 0x01e23c to 0x026006 (0x7dcb bytes) 000295 1786166718.025718 trunc from 0x02236f to 0x029c18 (0x78aa bytes) 000296 1786166718.046308 mapread 0x0061ca thru 0x00e481 (0x82b8 bytes) 000297 1786166718.047586 read 0x01578b thru 0x023690 (0xdf06 bytes) 000298 1786166718.069694 write 0x03e373 thru 0x03ffff (0x1c8d bytes) 000299 1786166718.071044 trunc from 0x040000 to 0x00b586 (0x34a79 bytes) 000300 1786166718.099915 write 0x00e6ca thru 0x01b4c3 (0xcdfa bytes) 000301 1786166718.102339 punch from 0x009f0e to 0x014bad (0xaca0 bytes) 000302 1786166718.140829 mapread 0x00f026 thru 0x011383 (0x235e bytes) 000304 1786166718.163299 write 0x014a95 thru 0x01ed14 (0xa280 bytes) 000305 1786166718.172932 trunc from 0x003f13 to 0x00c480 (0x856e bytes) 000306 1786166718.183088 mapwrite 0x00ad25 thru 0x01a95b (0xfc37 bytes) 000307 1786166718.247712 read 0x0027fc thru 0x003074 (0x879 bytes) 000308 1786166718.262190 trunc from 0x0041dc to 0x006503 (0x2328 bytes) 000309 1786166718.269707 mapread 0x00240d thru 0x0123e0 (0xffd4 bytes) 000310 1786166718.297383 trunc from 0x01e4ed to 0x01ed14 (0x828 bytes) 000311 1786166718.308409 punch from 0x011d69 to 0x01ed14 (0xcfac bytes) 000312 1786166718.316843 write 0x01c923 thru 0x0220f6 (0x57d4 bytes) 000313 1786166718.330107 mapread 0x012164 thru 0x016c2f (0x4acc bytes) 000315 1786166718.398784 write 0x001b8d thru 0x005d57 (0x41cb bytes) 000316 1786166718.405404 write 0x00eb15 thru 0x0105e0 (0x1acc bytes) 000317 1786166718.413280 trunc from 0x00a06f to 0x00eaed (0x4a7f bytes) 000318 1786166718.443452 mapread 0x0106a4 thru 0x012b90 (0x24ed bytes) 000319 1786166718.460785 trunc from 0x0142a7 to 0x021f3b (0xdc95 bytes) 000320 1786166718.469389 punch from 0x011b82 to 0x012b71 (0xff0 bytes) 000321 1786166718.480419 write 0x03d84d thru 0x03daa3 (0x257 bytes) 000323 1786166718.484292 trunc from 0x03daa4 to 0x01619d (0x27906 bytes) 000324 1786166718.507051 mapwrite 0x010907 thru 0x019cfe (0x93f8 bytes) 000325 1786166718.644934 trunc from 0x00e625 to 0x019cfe (0xb6da bytes) 000327 1786166718.662731 read 0x0060ce thru 0x00aa5b (0x498e bytes) 000328 1786166718.679734 read 0x018b62 thru 0x019cfe (0x119d bytes) 000329 1786166718.680758 trunc from 0x019cff to 0x03148f (0x17791 bytes) 000330 1786166718.716613 mapwrite 0x00909a thru 0x00d851 (0x47b8 bytes) 000331 1786166718.741589 trunc from 0x031490 to 0x003976 (0x2db19 bytes) 000332 1786166718.779575 mapwrite 0x03f020 thru 0x03ffff (0xfe0 bytes) 000333 1786166718.939168 read 0x0166d5 thru 0x01998b (0x32b7 bytes) 000334 1786166718.944339 mapwrite 0x0283c6 thru 0x031740 (0x937b bytes) 000335 1786166719.073918 trunc from 0x040000 to 0x030409 (0xfbf6 bytes) 000336 1786166719.127708 write 0x00e30b thru 0x015663 (0x7359 bytes) 000337 1786166719.130880 fallocate 0x02c314 thru 0x02e1bd (0x1eaa bytes) 000338 1786166719.154039 trunc from 0x03040a to 0x0311c8 (0xdbf bytes) 000339 1786166719.203980 fallocate 0x02c083 thru 0x02f9ea (0x3968 bytes) 000340 1786166719.221287 trunc from 0x0311c9 to 0x03290b (0x1743 bytes) 000341 1786166719.265724 mapwrite 0x006187 thru 0x01288a (0xc704 bytes) 000342 1786166719.292316 mapread 0x019596 thru 0x019c96 (0x701 bytes) 000343 1786166719.293469 mapread 0x00b589 thru 0x015cb0 (0xa728 bytes) 000344 1786166719.295213 trunc from 0x03290c to 0x01936d (0x1959e bytes) 000345 1786166719.310909 mapread 0x003a8a thru 0x0111ea (0xd761 bytes) 000349 1786166719.317030 fallocate 0x018ab8 thru 0x01bf95 (0x34de bytes) 000350 1786166719.324695 mapwrite 0x00ebdb thru 0x016748 (0x7b6e bytes) 000352 1786166719.347609 mapwrite 0x01e8d1 thru 0x02996e (0xb09e bytes) 000353 1786166719.395660 read 0x013b97 thru 0x014fb2 (0x141c bytes) 000355 1786166719.398529 trunc from 0x02996f to 0x026478 (0x34f6 bytes) 000357 1786166719.415615 read 0x01568a thru 0x0199ab (0x4322 bytes) 000360 1786166719.418715 trunc from 0x0259a9 to 0x026478 (0xad0 bytes) 000361 1786166719.424571 fallocate 0x026802 thru 0x02bc41 (0x5440 bytes) 000362 1786166719.432291 trunc from 0x02bc42 to 0x0006cb (0x2b576 bytes) 000364 1786166719.454215 mapread 0x0001a5 thru 0x0006cb (0x527 bytes) 000366 1786166719.457317 mapwrite 0x028b18 thru 0x0305c4 (0x7aad bytes) 000368 1786166719.512559 mapread 0x025633 thru 0x02d7d6 (0x81a4 bytes) 000369 1786166719.514180 punch from 0x01f077 to 0x02ceb5 (0xde3f bytes) 000370 1786166719.521261 mapwrite 0x0395ad thru 0x03ffff (0x6a53 bytes) 000371 1786166719.581503 trunc from 0x03b709 to 0x03ffff (0x48f7 bytes) 000372 1786166719.588221 punch from 0x037ed5 to 0x03ffff (0x812b bytes) 000373 1786166719.594571 trunc from 0x040000 to 0x023f10 (0x1c0ef bytes) 000374 1786166719.612232 trunc from 0x023f11 to 0x02c347 (0x8437 bytes) 000375 1786166719.626383 fallocate 0x01612b thru 0x022522 (0xc3f8 bytes) 000378 1786166719.641760 read 0x00a91b thru 0x0144d7 (0x9bbd bytes) 000379 1786166719.643157 trunc from 0x02c348 to 0x0216b0 (0xac97 bytes) 000380 1786166719.660520 fallocate 0x016b9d thru 0x022757 (0xbbbb bytes) 000381 1786166719.671393 fallocate 0x01fa26 thru 0x023dfd (0x43d8 bytes) 000382 1786166719.676536 mapwrite 0x01ca76 thru 0x028cc4 (0xc24f bytes) 000383 1786166719.769738 write 0x0066a3 thru 0x00f01d (0x897b bytes) 000384 1786166719.772693 read 0x021f8c thru 0x028cc4 (0x6d39 bytes) 000385 1786166719.773633 write 0x011a99 thru 0x017af1 (0x6059 bytes) 000386 1786166719.776382 write 0x02fdd4 thru 0x0372aa (0x74d7 bytes) 000388 1786166719.779880 trunc from 0x01d02a to 0x01fbf6 (0x2bcd bytes) 000390 1786166719.786484 read 0x02d6ce thru 0x0372aa (0x9bdd bytes) 000391 1786166719.795113 punch from 0x01aeb3 to 0x01ef86 (0x40d4 bytes) 000392 1786166719.801254 fallocate 0x01b542 thru 0x01d4ff (0x1fbe bytes) 000393 1786166719.812403 mapwrite 0x016b11 thru 0x020ead (0xa39d bytes) 000394 1786166719.868737 fallocate 0x023bac thru 0x032808 (0xec5d bytes) 000395 1786166719.874021 trunc from 0x0372ab to 0x0344f7 (0x2db3 bytes) 000396 1786166719.892110 punch from 0x001072 to 0x001fe1 (0xf70 bytes) 000399 1786166719.901051 write 0x033d0e thru 0x03ffff (0xc2f2 bytes) 000400 1786166719.903428 trunc from 0x00730d to 0x0142e1 (0xcfd5 bytes) 000401 1786166719.910874 fallocate 0x028815 thru 0x02dc65 (0x5451 bytes) 000402 1786166719.924538 mapwrite 0x01f8af thru 0x022b16 (0x3268 bytes) 000403 1786166719.943287 mapread 0x0002ed thru 0x00290a (0x261e bytes) 000404 1786166719.956405 MIRROR_EXTEND to 2 mirrors 000405 1786166720.844928 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000406 1786166721.122169 fallocate 0x002995 thru 0x004a43 (0x20af bytes) 000407 1786166721.155892 mapwrite 0x019336 thru 0x01e6f7 (0x53c2 bytes) 000408 1786166721.240837 trunc from 0x02ddcc to 0x0313a7 (0x35dc bytes) 000409 1786166721.258803 read 0x030e41 thru 0x037883 (0x6a43 bytes) 000412 1786166721.285752 write 0x03539f thru 0x03b759 (0x63bb bytes) 000413 1786166721.304709 trunc from 0x040000 to 0x01237b (0x2dc84 bytes) 000414 1786166721.332162 write 0x02d28a thru 0x036f14 (0x9c8b bytes) 000415 1786166721.337716 read 0x024710 thru 0x032e35 (0xe726 bytes) 000416 1786166721.352575 mapwrite 0x01b04f thru 0x025fa5 (0xaf57 bytes) 000418 1786166721.388525 read 0x0230fe thru 0x0328f9 (0xf7fc bytes) 000419 1786166721.389639 read 0x03241a thru 0x036f14 (0x4afb bytes) 000420 1786166721.390667 mapread 0x00e95e thru 0x01d339 (0xe9dc bytes) 000421 1786166721.393193 read 0x024a7d thru 0x03458a (0xfb0e bytes) 000422 1786166721.394094 trunc from 0x012285 to 0x013cc0 (0x1a3c bytes) 000423 1786166721.406808 trunc from 0x034515 to 0x036f14 (0x2a00 bytes) 000424 1786166721.420004 write 0x008cd7 thru 0x00f69e (0x69c8 bytes) 000425 1786166721.421425 read 0x01b78e thru 0x027702 (0xbf75 bytes) 000427 1786166721.432158 read 0x00414f thru 0x00ad9c (0x6c4e bytes) 000428 1786166721.432992 fallocate 0x01563f thru 0x02302d (0xd9ef bytes) 000430 1786166721.448588 fallocate 0x007a43 thru 0x00d826 (0x5de4 bytes) 000431 1786166721.454841 fallocate 0x02a7e8 thru 0x02aa53 (0x26c bytes) 000432 1786166721.461985 read 0x03148f thru 0x036f14 (0x5a86 bytes) 000433 1786166721.472924 mapread 0x01b090 thru 0x021e10 (0x6d81 bytes) 000435 1786166721.474758 fallocate 0x01ef58 thru 0x021a84 (0x2b2d bytes) 000436 1786166721.480999 trunc from 0x036f15 to 0x00ffad (0x26f67 bytes) 000438 1786166721.515599 trunc from 0x004971 to 0x00ffad (0xb63d bytes) 000439 1786166721.537929 mapwrite 0x0398dc thru 0x03ffff (0x6724 bytes) 000440 1786166721.589250 trunc from 0x020af8 to 0x028c41 (0x814a bytes) 000441 1786166721.595766 write 0x033ab4 thru 0x03ffff (0xc54c bytes) 000442 1786166721.602755 trunc from 0x00bce1 to 0x01a228 (0xe548 bytes) 000444 1786166721.610280 trunc from 0x00dcd6 to 0x019bfc (0xbf27 bytes) 000445 1786166721.621160 trunc from 0x040000 to 0x0366f3 (0x990c bytes) 000446 1786166721.647277 mapwrite 0x0051d3 thru 0x0105b8 (0xb3e6 bytes) 000447 1786166721.751731 mapwrite 0x03c1aa thru 0x03ffff (0x3e56 bytes) 000448 1786166721.820922 mapwrite 0x002a1a thru 0x010233 (0xd81a bytes) 000449 1786166721.852350 read 0x03aa09 thru 0x03c75c (0x1d54 bytes) 000450 1786166721.869107 trunc from 0x040000 to 0x0323e1 (0xdc1e bytes) 000451 1786166721.888086 fallocate 0x01c253 thru 0x0212e6 (0x5094 bytes) 000452 1786166721.894390 mapread 0x02ff16 thru 0x0323e1 (0x24cc bytes) 000453 1786166721.897036 mapwrite 0x03a7af thru 0x03eaeb (0x433d bytes) 000454 1786166721.990967 read 0x0147ce thru 0x021e02 (0xd635 bytes) 000455 1786166721.992355 fallocate 0x01a771 thru 0x027520 (0xcdb0 bytes) 000456 1786166722.004752 read 0x007b83 thru 0x01028d (0x870b bytes) 000457 1786166722.006617 write 0x03225e thru 0x03e2e5 (0xc088 bytes) 000458 1786166722.011552 fallocate 0x01421f thru 0x01fc2a (0xba0c bytes) 000459 1786166722.019406 punch from 0x02e8c6 to 0x031a06 (0x3141 bytes) 000460 1786166722.025696 trunc from 0x03eaec to 0x00655d (0x3858e bytes) 000461 1786166722.045635 trunc from 0x00655e to 0x032ce0 (0x2c783 bytes) 000462 1786166722.068354 mapread 0x01e964 thru 0x029425 (0xaac2 bytes) 000463 1786166722.086283 mapread 0x017354 thru 0x025762 (0xe40f bytes) 000464 1786166722.087633 punch from 0x025194 to 0x029855 (0x46c2 bytes) 000465 1786166722.098211 trunc from 0x0149b4 to 0x015020 (0x66d bytes) 000466 1786166722.110760 trunc from 0x008a50 to 0x0127cf (0x9d80 bytes) 000467 1786166722.120106 punch from 0x02f1b2 to 0x032ce0 (0x3b2f bytes) 000468 1786166722.133434 mapread 0x015a12 thru 0x024aff (0xf0ee bytes) 000469 1786166722.151634 write 0x030e29 thru 0x03c9c1 (0xbb99 bytes) 000470 1786166722.169297 mapwrite 0x0327ac thru 0x037189 (0x49de bytes) 000472 1786166722.202219 fallocate 0x03add1 thru 0x03ffff (0x522f bytes) 000473 1786166722.209247 mapwrite 0x01814e thru 0x018ec8 (0xd7b bytes) 000474 1786166722.230780 write 0x03b0fa thru 0x03ffff (0x4f06 bytes) 000475 1786166722.236363 trunc from 0x040000 to 0x02b3f6 (0x14c09 bytes) 000476 1786166722.263258 write 0x01d0b0 thru 0x0216c2 (0x4613 bytes) 000477 1786166722.267898 fallocate 0x03b008 thru 0x03ffff (0x4ff8 bytes) 000478 1786166722.274924 punch from 0x01d554 to 0x02b3f6 (0xdea3 bytes) 000479 1786166722.297210 read 0x02126a thru 0x02b3f6 (0xa18d bytes) 000480 1786166722.311167 read 0x024b90 thru 0x02b3f6 (0x6867 bytes) 000481 1786166722.315018 read 0x01c8db thru 0x023fbe (0x76e4 bytes) 000482 1786166722.323957 trunc from 0x02b3f7 to 0x002024 (0x293d2 bytes) 000484 1786166722.354161 trunc from 0x001b70 to 0x002024 (0x4b5 bytes) 000485 1786166722.373470 MIRROR_EXTEND to 2 mirrors 000487 1786166722.989265 punch from 0x00044c to 0x002024 (0x1bd9 bytes) 000488 1786166723.032446 mapwrite 0x01556c thru 0x021298 (0xbd2d bytes) 000489 1786166723.130985 trunc from 0x021299 to 0x01236f (0xef29 bytes) 000490 1786166723.151985 trunc from 0x0109ac to 0x01236f (0x19c4 bytes) 000491 1786166723.159130 punch from 0x00773b to 0x00a4ba (0x2d80 bytes) 000492 1786166723.167134 punch from 0x011a9d to 0x01236f (0x8d3 bytes) 000493 1786166723.172139 read 0x008674 thru 0x0113e8 (0x8d75 bytes) 000494 1786166723.181999 fallocate 0x035cde thru 0x03ffff (0xa322 bytes) 000495 1786166723.187249 trunc from 0x040000 to 0x0182dc (0x27d23 bytes) 000497 1786166723.209086 read 0x0143aa thru 0x0153bd (0x1014 bytes) 000498 1786166723.233050 read 0x00ccca thru 0x0182dc (0xb613 bytes) 000499 1786166723.233889 mapwrite 0x02daf3 thru 0x031e36 (0x4344 bytes) 000500 1786166723.328133 trunc from 0x031e37 to 0x03bf92 (0xa15c bytes) 000501 1786166723.359738 read 0x038b0f thru 0x03bf92 (0x3484 bytes) 000502 1786166723.380076 trunc from 0x03bf93 to 0x00bfa4 (0x2ffee bytes) 000504 1786166723.420765 trunc from 0x00bfa5 to 0x022eae (0x16f0a bytes) 000505 1786166723.452370 mapread 0x0223e8 thru 0x022eae (0xac7 bytes) 000506 1786166723.462211 fallocate 0x01579f thru 0x01ce22 (0x7684 bytes) 000508 1786166723.473555 mapread 0x00c831 thru 0x0113f7 (0x4bc7 bytes) 000510 1786166723.535818 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000511 1786166723.798668 fallocate 0x03a3c9 thru 0x03ffff (0x5c37 bytes) 000514 1786166723.806634 fallocate 0x005252 thru 0x0135d7 (0xe386 bytes) 000516 1786166723.815697 read 0x03e4b7 thru 0x03ffff (0x1b49 bytes) 000517 1786166723.822049 read 0x0031ec thru 0x00852e (0x5343 bytes) 000518 1786166723.836056 fallocate 0x0311d3 thru 0x038092 (0x6ec0 bytes) 000519 1786166723.842187 mapwrite 0x0084ba thru 0x00dc14 (0x575b bytes) 000520 1786166723.859845 trunc from 0x040000 to 0x037f66 (0x8099 bytes) 000521 1786166723.883877 trunc from 0x037f67 to 0x00b85c (0x2c70a bytes) 000523 1786166723.908728 trunc from 0x00b85d to 0x007a86 (0x3dd6 bytes) 000524 1786166723.929676 MIRROR_EXTEND to 2 mirrors 000525 1786166724.577909 trunc from 0x007a87 to 0x018cfb (0x11275 bytes) 000526 1786166724.634175 read 0x003601 thru 0x011834 (0xe234 bytes) 000527 1786166724.644575 fallocate 0x005b4d thru 0x013202 (0xd6b6 bytes) 000528 1786166724.653883 fallocate 0x02146c thru 0x0251a3 (0x3d38 bytes) 000529 1786166724.661284 trunc from 0x0251a4 to 0x032e77 (0xdcd4 bytes) 000530 1786166724.696234 read 0x02eb81 thru 0x032e77 (0x42f7 bytes) 000531 1786166724.708107 trunc from 0x032e78 to 0x02f40f (0x3a68 bytes) 000532 1786166724.724054 trunc from 0x022912 to 0x029545 (0x6c34 bytes) 000533 1786166724.731096 punch from 0x00ca2d to 0x013f2f (0x7503 bytes) 000534 1786166724.735894 mapwrite 0x000b9e thru 0x0024cf (0x1932 bytes) 000535 1786166724.750931 trunc from 0x02f410 to 0x021cc0 (0xd74f bytes) 000536 1786166724.769035 write 0x00679c thru 0x015101 (0xe966 bytes) 000537 1786166724.772232 trunc from 0x00b83d to 0x010c10 (0x53d4 bytes) 000538 1786166724.796764 trunc from 0x00c4cc to 0x0156c8 (0x91fd bytes) 000539 1786166724.802505 trunc from 0x021cc1 to 0x030a0e (0xed4e bytes) 000540 1786166724.818640 read 0x01e8fd thru 0x023e35 (0x5539 bytes) 000541 1786166724.838439 trunc from 0x030a0f to 0x009724 (0x272ea bytes) 000542 1786166724.858705 trunc from 0x008a3c to 0x009724 (0xce9 bytes) 000543 1786166724.864125 punch from 0x0085e2 to 0x009724 (0x1143 bytes) 000544 1786166724.870153 punch from 0x003c3f to 0x006586 (0x2948 bytes) 000545 1786166724.875867 trunc from 0x002843 to 0x00425b (0x1a19 bytes) 000546 1786166724.881902 mapwrite 0x00aa2e thru 0x00e96a (0x3f3d bytes) 000547 1786166724.943668 punch from 0x0087aa to 0x00ca86 (0x42dd bytes) 000548 1786166724.951052 mapread 0x00e8d8 thru 0x00e96a (0x93 bytes) 000549 1786166724.952307 mapread 0x00c296 thru 0x00e96a (0x26d5 bytes) 000550 1786166724.957677 write 0x02e5f8 thru 0x03a6ff (0xc108 bytes) 000551 1786166724.960358 write 0x00f1f6 thru 0x01bfc3 (0xcdce bytes) 000552 1786166724.973880 fallocate 0x022ea2 thru 0x025ff3 (0x3152 bytes) 000553 1786166724.979052 mapwrite 0x03991a thru 0x03ffff (0x66e6 bytes) 000554 1786166725.034123 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d6146a1 Mirror 3: 0x3d6146a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000555 1786166725.873695 trunc from 0x001b68 to 0x00d485 (0xb91e bytes) 000556 1786166725.919285 write 0x00b3a2 thru 0x01442b (0x908a bytes) 000557 1786166725.951285 fallocate 0x02117c thru 0x02a6b3 (0x9538 bytes) 000558 1786166725.965135 trunc from 0x040000 to 0x01cdd6 (0x23229 bytes) 000559 1786166725.999260 mapread 0x00aa16 thru 0x0102f7 (0x58e2 bytes) 000560 1786166726.007313 fallocate 0x000518 thru 0x004e11 (0x48fa bytes) 000561 1786166726.013156 fallocate 0x00bbcb thru 0x011ebd (0x62f3 bytes) 000563 1786166726.027094 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cdd7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cdd7): Mirror 2: 0x9f78e2e6 (stale) Mirror 3: 0xa57de75b (stale) CRC-32 checksum value for chunk [0, 0x1cdd7): Mirror 1: 0xa0f81280 Mirror 4: 0xa0f81280 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cdd7: skipped 000564 1786166726.913093 mapwrite 0x01a60a thru 0x02a0b8 (0xfaaf bytes) 000565 1786166727.023887 trunc from 0x0129b4 to 0x01b8ca (0x8f17 bytes) 000566 1786166727.029881 fallocate 0x03bb5e thru 0x03ffff (0x44a2 bytes) 000567 1786166727.035535 mapwrite 0x02e4a9 thru 0x036e14 (0x896c bytes) 000568 1786166727.094569 write 0x00273c thru 0x00b14a (0x8a0f bytes) 000569 1786166727.113540 mapwrite 0x02cd3c thru 0x031e39 (0x50fe bytes) 000570 1786166727.147831 write 0x01250d thru 0x01dfd8 (0xbacc bytes) 000571 1786166727.154852 read 0x035f06 thru 0x036e14 (0xf0f bytes) 000572 1786166727.155615 write 0x00650d thru 0x007fb0 (0x1aa4 bytes) 000573 1786166727.157174 mapwrite 0x003a72 thru 0x008d98 (0x5327 bytes) 000574 1786166727.176698 mapwrite 0x003288 thru 0x003826 (0x59f bytes) 000575 1786166727.197002 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36e15): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e15): Mirror 2: 0xee17e9d6 (stale) Mirror 3: 0xcb3a29c6 (stale) Mirror 4: 0x69d3979a (stale) CRC-32 checksum value for chunk [0, 0x36e15): Mirror 1: 0x72e1b221 Mirror 5: 0x72e1b221 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e15: skipped 000576 1786166727.990280 trunc from 0x0018a0 to 0x0088b9 (0x701a bytes) 000577 1786166728.029202 fallocate 0x0348d8 thru 0x03d59e (0x8cc7 bytes) 000578 1786166728.041173 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 2: 0xe09797d0 (stale) Mirror 3: 0xd0092159 (stale) Mirror 4: 0x8e6564a2 (stale) Mirror 5: 0x1a5190bc (stale) CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 1: 0x430eb689 Mirror 6: 0x430eb689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d59f: skipped 000580 1786166728.984914 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 2: 0xe09797d0 (stale) Mirror 3: 0xd0092159 (stale) Mirror 5: 0x1a5190bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 2: 0xe09797d0 (stale) Mirror 3: 0xd0092159 (stale) Mirror 5: 0x1a5190bc (stale) CRC-32 checksum value for chunk [0, 0x3d59f): Mirror 1: 0x430eb689 Mirror 6: 0x430eb689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d59f: skipped 000581 1786166729.822217 mapread 0x02a51a thru 0x02b199 (0xc80 bytes) 000582 1786166729.855194 trunc from 0x018d0a to 0x02766d (0xe964 bytes) 000583 1786166729.908002 trunc from 0x03d59f to 0x0366fa (0x6ea4 bytes) 000584 1786166729.938126 mapread 0x002818 thru 0x009cc5 (0x74ae bytes) 000585 1786166729.958272 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366fb): Mirror 3: 0xb5b0eaf1 (stale) Mirror 5: 0x5d191ae9 (stale) Mirror 6: 0x8728c2ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366fb): Mirror 3: 0xb5b0eaf1 (stale) Mirror 5: 0x5d191ae9 (stale) Mirror 6: 0x8728c2ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366fb: skipped 000586 1786166730.354100 trunc from 0x010efa to 0x014d4a (0x3e51 bytes) 000588 1786166730.368648 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x366fb): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366fb): Mirror 3: 0xb5b0eaf1 (stale) Mirror 5: 0x5d191ae9 (stale) Mirror 6: 0x8728c2ac (stale) CRC-32 checksum value for chunk [0, 0x366fb): Mirror 1: 0xf8be3f91 Mirror 7: 0xf8be3f91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366fb: skipped 000589 1786166731.665142 mapread 0x02202c thru 0x02c25d (0xa232 bytes) 000590 1786166731.696607 punch from 0x00458c to 0x007e87 (0x38fc bytes) 000591 1786166731.746042 trunc from 0x0366fb to 0x0023d2 (0x34328 bytes) 000592 1786166731.793267 mapread 0x0008bd thru 0x001590 (0xcd4 bytes) 000593 1786166731.801271 read 0x000063 thru 0x0023d2 (0x2370 bytes) 000594 1786166731.801891 trunc from 0x00103c to 0x0023d2 (0x1397 bytes) 000595 1786166731.808240 trunc from 0x0023d3 to 0x00cada (0xa708 bytes) 000596 1786166731.826878 punch from 0x00602f to 0x00ca18 (0x69ea bytes) 000597 1786166731.831256 read 0x00348d thru 0x009381 (0x5ef5 bytes) 000598 1786166731.848868 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xcadb): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcadb): Mirror 3: 0x2421ab7f (stale) Mirror 5: 0xb54099c0 (stale) Mirror 6: 0xead4fe07 (stale) Mirror 7: 0xead4fe07 (stale) CRC-32 checksum value for chunk [0, 0xcadb): Mirror 1: 0x329767b Mirror 8: 0x329767b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcadb: skipped 000600 1786166733.011344 trunc from 0x008d3f to 0x00cada (0x3d9c bytes) 000601 1786166733.105991 trunc from 0x005ee9 to 0x0076d8 (0x17f0 bytes) 000602 1786166733.115588 mapwrite 0x025fa9 thru 0x035e37 (0xfe8f bytes) 000603 1786166733.241191 read 0x014930 thru 0x01ce82 (0x8553 bytes) 000604 1786166733.253493 mapread 0x029691 thru 0x035e37 (0xc7a7 bytes) 000605 1786166733.254916 fallocate 0x01fc9f thru 0x028260 (0x85c2 bytes) 000606 1786166733.277349 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e38): Mirror 5: 0xb72f463b (stale) Mirror 6: 0xc5764fbd (stale) Mirror 7: 0xe07841b3 (stale) Mirror 8: 0x81bce8a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e38): Mirror 5: 0xb72f463b (stale) Mirror 6: 0xc5764fbd (stale) Mirror 7: 0xe07841b3 (stale) Mirror 8: 0x81bce8a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e38: skipped 000607 1786166733.666552 fallocate 0x00dd42 thru 0x0173b8 (0x9677 bytes) 000610 1786166733.675080 fallocate 0x013a3c thru 0x01850f (0x4ad4 bytes) 000611 1786166733.680909 trunc from 0x013aef to 0x019e14 (0x6326 bytes) 000612 1786166733.686791 punch from 0x017068 to 0x0177c5 (0x75e bytes) 000613 1786166733.695799 mapread 0x000300 thru 0x00f576 (0xf277 bytes) 000614 1786166733.719174 write 0x01ee8f thru 0x021f05 (0x3077 bytes) 000615 1786166733.733994 mapread 0x02d92e thru 0x035e37 (0x850a bytes) 000617 1786166733.744916 read 0x015e74 thru 0x01dd96 (0x7f23 bytes) 000618 1786166733.756602 mapread 0x004f57 thru 0x00f2ad (0xa357 bytes) 000619 1786166733.757978 mapread 0x01feca thru 0x022a03 (0x2b3a bytes) 000620 1786166733.772032 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e38): Mirror 5: 0xb72f463b (stale) Mirror 6: 0xc5764fbd (stale) Mirror 8: 0x81bce8a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e38): Mirror 5: 0xb72f463b (stale) Mirror 6: 0xc5764fbd (stale) Mirror 8: 0x81bce8a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e38: skipped 000621 1786166734.106594 mapwrite 0x013bab thru 0x019bb5 (0x600b bytes) 000622 1786166734.171232 punch from 0x001d98 to 0x011d5c (0xffc5 bytes) 000623 1786166734.179868 punch from 0x013e83 to 0x022c0a (0xed88 bytes) 000624 1786166734.187999 trunc from 0x035e38 to 0x012e2c (0x2300b bytes) 000625 1786166734.211831 trunc from 0x012e2d to 0x00b0f5 (0x7d37 bytes) 000626 1786166734.248604 punch from 0x0043b2 to 0x00b0f5 (0x6d44 bytes) 000627 1786166734.256776 fallocate 0x036039 thru 0x03b6ad (0x5675 bytes) 000628 1786166734.264338 mapwrite 0x03d50e thru 0x03eca4 (0x1797 bytes) 000629 1786166734.297961 trunc from 0x03eca5 to 0x0001d5 (0x3eacf bytes) 000630 1786166734.310051 read 0x000160 thru 0x0001d5 (0x76 bytes) 000631 1786166734.314983 fallocate 0x003132 thru 0x00750d (0x43dc bytes) 000633 1786166734.320556 trunc from 0x00750e to 0x014330 (0xce23 bytes) 000634 1786166734.339761 mapread 0x009865 thru 0x00c240 (0x29dc bytes) 000635 1786166734.359803 trunc from 0x014331 to 0x0048a4 (0xfa8c bytes) 000636 1786166734.377895 fallocate 0x00ce6e thru 0x00f3dd (0x2570 bytes) 000637 1786166734.386096 mapread 0x0045e1 thru 0x0048a4 (0x2c4 bytes) 000638 1786166734.388184 read 0x0005e1 thru 0x0048a4 (0x42c4 bytes) 000639 1786166734.395256 write 0x00825e thru 0x00dd8e (0x5b31 bytes) 000640 1786166734.398567 write 0x02853f thru 0x0376d7 (0xf199 bytes) 000641 1786166734.401672 write 0x019705 thru 0x023a1e (0xa31a bytes) 000642 1786166734.413602 read 0x02f886 thru 0x031bc8 (0x2343 bytes) 000643 1786166734.415393 trunc from 0x0376d8 to 0x02fcf9 (0x79de bytes) 000644 1786166734.438604 write 0x01179f thru 0x017dfb (0x665d bytes) 000645 1786166734.442128 fallocate 0x014687 thru 0x017f62 (0x38dc bytes) 000647 1786166734.454929 read 0x0025d6 thru 0x010975 (0xe3a0 bytes) 000648 1786166734.456333 write 0x025910 thru 0x0340f0 (0xe7e1 bytes) 000650 1786166734.464612 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x340f1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340f1): Mirror 5: 0x208f3bba (stale) Mirror 6: 0x8e98eeeb (stale) Mirror 8: 0x2ff9ec6a (stale) CRC-32 checksum value for chunk [0, 0x340f1): Mirror 1: 0xa07d0ca0 Mirror 9: 0xa07d0ca0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340f1: skipped 000651 1786166735.885172 fallocate 0x02c01a thru 0x034dd0 (0x8db7 bytes) 000653 1786166735.952354 mapwrite 0x03966b thru 0x03ffff (0x6995 bytes) 000654 1786166736.036491 trunc from 0x038cb4 to 0x03ffff (0x734c bytes) 000655 1786166736.046408 MIRROR_RESYNC 5 mirrors 000656 1786166737.820678 trunc from 0x040000 to 0x00eab4 (0x3154b bytes) 000658 1786166737.870127 punch from 0x00c5d3 to 0x00eab4 (0x24e2 bytes) 000659 1786166737.880079 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeab5): Mirror 5: 0x617d1eb4 (stale) Mirror 6: 0x617d1eb4 (stale) Mirror 9: 0x617d1eb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeab5): Mirror 5: 0x617d1eb4 (stale) Mirror 6: 0x617d1eb4 (stale) Mirror 9: 0x617d1eb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeab5: skipped 000660 1786166738.222608 fallocate 0x03156c thru 0x038623 (0x70b8 bytes) 000661 1786166738.228923 punch from 0x003a84 to 0x00b1c2 (0x773f bytes) 000662 1786166738.238313 mapwrite 0x0136be thru 0x01caec (0x942f bytes) 000663 1786166738.301244 trunc from 0x000f7f to 0x0056b1 (0x4733 bytes) 000664 1786166738.308167 mapwrite 0x038c83 thru 0x03f592 (0x6910 bytes) 000665 1786166738.354778 trunc from 0x03f593 to 0x03ea9b (0xaf7 bytes) 000666 1786166738.372834 trunc from 0x03ea9c to 0x031317 (0xd784 bytes) 000668 1786166738.398847 write 0x007499 thru 0x009a45 (0x25ad bytes) 000669 1786166738.411221 mapwrite 0x03025e thru 0x0397bb (0x955e bytes) 000670 1786166738.474144 mapwrite 0x004789 thru 0x0122f9 (0xdb71 bytes) 000671 1786166738.519450 punch from 0x0328f4 to 0x0397bb (0x6ec8 bytes) 000672 1786166738.526383 fallocate 0x02a3d1 thru 0x02bd3e (0x196e bytes) 000673 1786166738.533138 fallocate 0x028132 thru 0x034e8e (0xcd5d bytes) 000674 1786166738.540147 fallocate 0x032663 thru 0x0332af (0xc4d bytes) 000675 1786166738.551818 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x397bc): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397bc): Mirror 5: 0xcc13a20 (stale) Mirror 6: 0xcc13a20 (stale) Mirror 9: 0xcc13a20 (stale) CRC-32 checksum value for chunk [0, 0x397bc): Mirror 1: 0xceecf33 Mirror 10: 0xceecf33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397bc: skipped 000676 1786166739.679244 write 0x01e9dd thru 0x02c201 (0xd825 bytes) 000677 1786166739.756358 mapread 0x02ca6a thru 0x031876 (0x4e0d bytes) 000679 1786166739.798023 write 0x0087e6 thru 0x01147d (0x8c98 bytes) 000680 1786166739.831075 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397bc): Mirror 5: 0xcc13a20 (stale) Mirror 6: 0xcc13a20 (stale) Mirror 10: 0xceecf33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397bc): Mirror 5: 0xcc13a20 (stale) Mirror 6: 0xcc13a20 (stale) Mirror 10: 0xceecf33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397bc: skipped 000681 1786166740.436291 write 0x0269c5 thru 0x035542 (0xeb7e bytes) 000682 1786166740.469067 fallocate 0x03c4e0 thru 0x03c7b3 (0x2d4 bytes) 000683 1786166740.486581 read 0x025738 thru 0x02e155 (0x8a1e bytes) 000684 1786166740.502847 fallocate 0x02150d thru 0x02bc2a (0xa71e bytes) 000685 1786166740.513176 mapwrite 0x000283 thru 0x004cf6 (0x4a74 bytes) 000686 1786166740.576814 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397bc): Mirror 6: 0xcc13a20 (stale) Mirror 10: 0xceecf33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397bc): Mirror 6: 0xcc13a20 (stale) Mirror 10: 0xceecf33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397bc: skipped 000687 1786166740.989119 write 0x0240ec thru 0x031861 (0xd776 bytes) 000688 1786166741.022188 mapwrite 0x005aba thru 0x01058f (0xaad6 bytes) 000689 1786166741.078424 write 0x028a8d thru 0x02c636 (0x3baa bytes) 000691 1786166741.081481 trunc from 0x0397bc to 0x018d65 (0x20a56 bytes) 000692 1786166741.139868 fallocate 0x03c05c thru 0x03ffff (0x3fa4 bytes) 000693 1786166741.146038 mapwrite 0x01ccc3 thru 0x024dbc (0x80fa bytes) 000694 1786166741.197292 write 0x015cb2 thru 0x016082 (0x3d1 bytes) 000695 1786166741.203757 MIRROR_RESYNC 3 mirrors 000696 1786166741.935764 mapread 0x006a4e thru 0x01517c (0xe72f bytes) 000698 1786166742.034458 trunc from 0x024dbd to 0x03047f (0xb6c3 bytes) 000699 1786166742.147124 mapread 0x016b55 thru 0x021092 (0xa53e bytes) 000700 1786166742.181301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30480): Mirror 6: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30480): Mirror 6: 0x173461b5 (stale) Mirror 10: 0x173461b5 (stale) CRC-32 checksum value for chunk [0, 0x30480): Mirror 1: 0x173461b5 Mirror 11: 0x173461b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30480: skipped 000701 1786166743.186028 trunc from 0x030480 to 0x008e59 (0x27626 bytes) 000702 1786166743.269463 punch from 0x0055ff to 0x006b2a (0x152c bytes) 000703 1786166743.275442 trunc from 0x008e5a to 0x0211be (0x18365 bytes) 000704 1786166743.293846 trunc from 0x0211bf to 0x000314 (0x20eaa bytes) 000705 1786166743.310378 trunc from 0x00013a to 0x000314 (0x1db bytes) 000706 1786166743.317072 mapread 0x000026 thru 0x000314 (0x2ef bytes) 000707 1786166743.323126 write 0x01b8b2 thru 0x02035b (0x4aaa bytes) 000708 1786166743.325114 punch from 0x002e2d to 0x005ed6 (0x30aa bytes) 000709 1786166743.329928 read 0x006189 thru 0x006470 (0x2e8 bytes) 000710 1786166743.335687 write 0x028bfc thru 0x02d026 (0x442b bytes) 000711 1786166743.337164 write 0x021419 thru 0x02b318 (0x9f00 bytes) 000712 1786166743.348435 trunc from 0x014afa to 0x017ba4 (0x30ab bytes) 000713 1786166743.353891 write 0x021b8c thru 0x0308e4 (0xed59 bytes) 000714 1786166743.356505 mapread 0x01b4e1 thru 0x02305c (0x7b7c bytes) 000715 1786166743.362426 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x308e5): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308e5): Mirror 6: 0xe5ef2f06 (stale) Mirror 10: 0xe5ef2f06 (stale) Mirror 11: 0xe5ef2f06 (stale) CRC-32 checksum value for chunk [0, 0x308e5): Mirror 1: 0xdb830c6b Mirror 12: 0xdb830c6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308e5: skipped 000716 1786166744.117283 trunc from 0x0308e5 to 0x0055d9 (0x2b30b bytes) 000717 1786166744.185991 trunc from 0x0055da to 0x00f56d (0x9f94 bytes) 000718 1786166744.210841 mapwrite 0x03e401 thru 0x03ffff (0x1bff bytes) 000719 1786166744.258803 trunc from 0x040000 to 0x022503 (0x1dafc bytes) 000720 1786166744.277897 read 0x013cce thru 0x018f28 (0x525b bytes) 000721 1786166744.286600 mapwrite 0x007b86 thru 0x0102cd (0x8748 bytes) 000722 1786166744.317740 mapread 0x01bf07 thru 0x022503 (0x65fd bytes) 000723 1786166744.328231 read 0x00fa97 thru 0x01644b (0x69b5 bytes) 000725 1786166744.337671 mapread 0x00427d thru 0x008b35 (0x48b9 bytes) 000726 1786166744.338910 read 0x013783 thru 0x0213a3 (0xdc21 bytes) 000727 1786166744.340031 write 0x000561 thru 0x001685 (0x1125 bytes) 000728 1786166744.341365 write 0x03e9f3 thru 0x03ffff (0x160d bytes) 000729 1786166744.343088 fallocate 0x00ee18 thru 0x014b14 (0x5cfd bytes) 000730 1786166744.348969 fallocate 0x030602 thru 0x03ff58 (0xf957 bytes) 000731 1786166744.359422 MIRROR_RESYNC 5 mirrors 000732 1786166745.660330 write 0x015f4c thru 0x02297c (0xca31 bytes) 000733 1786166745.696644 punch from 0x003676 to 0x00439a (0xd25 bytes) 000734 1786166745.707491 trunc from 0x040000 to 0x03dbe7 (0x2418 bytes) 000735 1786166745.739296 mapwrite 0x03d192 thru 0x03ffff (0x2e6e bytes) 000736 1786166745.804766 trunc from 0x0120c5 to 0x021811 (0xf74d bytes) 000738 1786166745.809554 read 0x01bd0c thru 0x027fc4 (0xc2b9 bytes) 000739 1786166745.820164 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x54227c38 (stale) Mirror 10: 0x54227c38 (stale) Mirror 11: 0x54227c38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x54227c38 (stale) Mirror 10: 0x54227c38 (stale) Mirror 11: 0x54227c38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1786166746.389377 trunc from 0x040000 to 0x02e8a3 (0x1175c bytes) 000741 1786166746.441125 punch from 0x028fa0 to 0x02e8a3 (0x5904 bytes) 000742 1786166746.457317 write 0x004d3d thru 0x00dd1d (0x8fe1 bytes) 000743 1786166746.480525 write 0x03da0d thru 0x03e8c6 (0xeba bytes) 000744 1786166746.484088 trunc from 0x03e8c7 to 0x00a0c0 (0x34806 bytes) 000745 1786166746.510030 punch from 0x003b6a to 0x006912 (0x2da9 bytes) 000746 1786166746.528768 mapread 0x007f22 thru 0x00a0c0 (0x219f bytes) 000747 1786166746.530277 trunc from 0x008faa to 0x009c69 (0xcc0 bytes) 000748 1786166746.535712 read 0x00837f thru 0x00a0c0 (0x1d42 bytes) 000750 1786166746.545514 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa0c1): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa0c1): Mirror 6: 0xcb955bdb (stale) Mirror 10: 0xcb955bdb (stale) Mirror 11: 0xcb955bdb (stale) CRC-32 checksum value for chunk [0, 0xa0c1): Mirror 1: 0xf1441269 Mirror 12: 0xf1441269 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa0c1: skipped 000751 1786166747.633922 trunc from 0x000273 to 0x007a80 (0x780e bytes) 000752 1786166747.687036 trunc from 0x00a0c1 to 0x0347a1 (0x2a6e1 bytes) 000753 1786166747.739365 trunc from 0x002b4c to 0x007b45 (0x4ffa bytes) 000754 1786166747.750415 MIRROR_RESYNC 5 mirrors 000755 1786166749.176322 punch from 0x01dc2c to 0x023651 (0x5a26 bytes) 000756 1786166749.209278 punch from 0x00e21e to 0x0139c4 (0x57a7 bytes) 000757 1786166749.216623 fallocate 0x01ec60 thru 0x0209d5 (0x1d76 bytes) 000758 1786166749.223268 punch from 0x03205e to 0x0347a1 (0x2744 bytes) 000759 1786166749.228157 read 0x02bfbe thru 0x0347a1 (0x87e4 bytes) 000760 1786166749.239495 punch from 0x01162b to 0x012151 (0xb27 bytes) 000761 1786166749.260129 read 0x02f815 thru 0x0347a1 (0x4f8d bytes) 000762 1786166749.261358 mapwrite 0x027307 thru 0x02dfb0 (0x6caa bytes) 000763 1786166749.313520 fallocate 0x02ec80 thru 0x037508 (0x8889 bytes) 000764 1786166749.324747 mapwrite 0x03d729 thru 0x03ffff (0x28d7 bytes) 000765 1786166749.400720 read 0x009af3 thru 0x00e9c0 (0x4ece bytes) 000766 1786166749.402425 fallocate 0x01733e thru 0x0256af (0xe372 bytes) 000767 1786166749.408821 read 0x0101d0 thru 0x01bd5b (0xbb8c bytes) 000768 1786166749.410288 mapwrite 0x0174b8 thru 0x01b2e0 (0x3e29 bytes) 000769 1786166749.432528 write 0x0082bd thru 0x009b49 (0x188d bytes) 000770 1786166749.434775 read 0x00d39a thru 0x019901 (0xc568 bytes) 000771 1786166749.436085 read 0x02b9f8 thru 0x037fa7 (0xc5b0 bytes) 000772 1786166749.437460 write 0x03d1a1 thru 0x03d3a7 (0x207 bytes) 000773 1786166749.441072 trunc from 0x0243e1 to 0x026b2d (0x274d bytes) 000774 1786166749.453072 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xedad359e (stale) Mirror 10: 0xedad359e (stale) Mirror 11: 0xedad359e (stale) Mirror 12: 0xedad359e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9fd2438f Mirror 13: 0x9fd2438f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1786166750.919687 punch from 0x03b91e to 0x03ffff (0x46e2 bytes) 000776 1786166750.976110 trunc from 0x040000 to 0x030db5 (0xf24a bytes) 000777 1786166751.003404 read 0x00a20d thru 0x00de4d (0x3c41 bytes) 000778 1786166751.012996 punch from 0x0192dc to 0x01c8b0 (0x35d5 bytes) 000779 1786166751.019730 write 0x02d4c1 thru 0x0389da (0xb51a bytes) 000780 1786166751.027477 write 0x01a719 thru 0x01d4e3 (0x2dcb bytes) 000781 1786166751.039195 fallocate 0x03b1d2 thru 0x03ffff (0x4e2e bytes) 000782 1786166751.046457 fallocate 0x021341 thru 0x02db6d (0xc82d bytes) 000783 1786166751.053084 read 0x018666 thru 0x01d7be (0x5159 bytes) 000784 1786166751.066429 fallocate 0x01e359 thru 0x01e519 (0x1c1 bytes) 000785 1786166751.081857 fallocate 0x027e82 thru 0x02e079 (0x61f8 bytes) 000786 1786166751.098933 trunc from 0x0389db to 0x001f52 (0x36a88 bytes) 000787 1786166751.140888 punch from 0x000ff3 to 0x001f52 (0xf60 bytes) 000788 1786166751.155322 mapwrite 0x007987 thru 0x00b28a (0x3904 bytes) 000789 1786166751.356237 mapwrite 0x01bb39 thru 0x01d9d5 (0x1e9d bytes) 000790 1786166751.432073 fallocate 0x000836 thru 0x00d299 (0xca64 bytes) 000792 1786166751.448125 write 0x032292 thru 0x035215 (0x2f84 bytes) 000793 1786166751.459292 punch from 0x016ee6 to 0x01a63f (0x375a bytes) 000794 1786166751.466800 punch from 0x01d889 to 0x01e256 (0x9ce bytes) 000795 1786166751.479244 mapread 0x02aa8e thru 0x02c552 (0x1ac5 bytes) 000796 1786166751.498268 read 0x003dbf thru 0x00ce7f (0x90c1 bytes) 000797 1786166751.513106 trunc from 0x02d747 to 0x035215 (0x7acf bytes) 000798 1786166751.528117 read 0x00746b thru 0x00f96e (0x8504 bytes) 000799 1786166751.529382 trunc from 0x035216 to 0x03fac7 (0xa8b2 bytes) 000800 1786166751.563892 MIRROR_RESYNC 6 mirrors 000801 1786166753.836213 mapwrite 0x013f2b thru 0x01fc3a (0xbd10 bytes) 000802 1786166754.027311 write 0x00af99 thru 0x015405 (0xa46d bytes) 000803 1786166754.040286 write 0x010e35 thru 0x01c4a3 (0xb66f bytes) 000804 1786166754.044593 trunc from 0x03fac8 to 0x02282d (0x1d29a bytes) 000805 1786166754.080914 read 0x020ef7 thru 0x02282d (0x1937 bytes) 000806 1786166754.081649 fallocate 0x0199eb thru 0x024e21 (0xb437 bytes) 000807 1786166754.092890 mapwrite 0x01d62a thru 0x0284f2 (0xaec9 bytes) 000808 1786166754.183246 mapread 0x026a77 thru 0x0284f2 (0x1a7c bytes) 000809 1786166754.188815 mapread 0x00aa68 thru 0x00dfc1 (0x355a bytes) 000810 1786166754.201777 MIRROR_RESYNC 6 mirrors 000813 1786166755.868751 punch from 0x00f453 to 0x01835d (0x8f0b bytes) 000814 1786166755.963251 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x284f3): Mirror 6: 0x517dc43a (stale) Mirror 10: 0x517dc43a (stale) Mirror 11: 0x517dc43a (stale) Mirror 12: 0x517dc43a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284f3): Mirror 6: 0x517dc43a (stale) Mirror 10: 0x517dc43a (stale) Mirror 11: 0x517dc43a (stale) Mirror 12: 0x517dc43a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284f3: skipped 000815 1786166756.710743 mapwrite 0x031c28 thru 0x036d8a (0x5163 bytes) 000816 1786166756.836932 read 0x0150d5 thru 0x01543a (0x366 bytes) 000817 1786166756.848991 fallocate 0x034160 thru 0x036ed7 (0x2d78 bytes) 000818 1786166756.854703 fallocate 0x01c685 thru 0x027578 (0xaef4 bytes) 000819 1786166756.864551 trunc from 0x036d8b to 0x00b0ad (0x2bcdd bytes) 000820 1786166756.896477 mapwrite 0x02be33 thru 0x03bb40 (0xfd0e bytes) 000821 1786166757.118125 mapread 0x00f103 thru 0x01c2fc (0xd1fa bytes) 000822 1786166757.172620 punch from 0x033d9c to 0x034a9b (0xd00 bytes) 000823 1786166757.192868 trunc from 0x0262ad to 0x02d9cd (0x7721 bytes) 000824 1786166757.215677 trunc from 0x01603e to 0x016833 (0x7f6 bytes) 000825 1786166757.228605 mapread 0x026220 thru 0x02dd56 (0x7b37 bytes) 000826 1786166757.243461 trunc from 0x02d5f0 to 0x035c5c (0x866d bytes) 000827 1786166757.249608 mapread 0x007be1 thru 0x00a299 (0x26b9 bytes) 000828 1786166757.263843 read 0x032a63 thru 0x039e20 (0x73be bytes) 000829 1786166757.275592 mapread 0x003750 thru 0x00649a (0x2d4b bytes) 000830 1786166757.277346 mapread 0x00d2e6 thru 0x014fe3 (0x7cfe bytes) 000831 1786166757.280397 punch from 0x028cb5 to 0x036531 (0xd87d bytes) 000832 1786166757.294574 mapwrite 0x0187fc thru 0x01dc23 (0x5428 bytes) 000833 1786166757.329178 punch from 0x02dd83 to 0x02f104 (0x1382 bytes) 000834 1786166757.347884 read 0x03b53f thru 0x03bb40 (0x602 bytes) 000835 1786166757.357442 mapread 0x00a0be thru 0x0103da (0x631d bytes) 000836 1786166757.363677 punch from 0x0229c1 to 0x024faa (0x25ea bytes) 000837 1786166757.378556 read 0x024213 thru 0x031cd3 (0xdac1 bytes) 000838 1786166757.416200 read 0x02fece thru 0x03b771 (0xb8a4 bytes) 000839 1786166757.416930 mapwrite 0x01b1c2 thru 0x02b120 (0xff5f bytes) 000841 1786166757.473615 trunc from 0x012eb0 to 0x01314a (0x29b bytes) 000842 1786166757.481721 trunc from 0x0311e7 to 0x03a5e1 (0x93fb bytes) 000843 1786166757.498571 read 0x01f425 thru 0x0210f3 (0x1ccf bytes) 000844 1786166757.502278 write 0x01914f thru 0x022830 (0x96e2 bytes) 000845 1786166757.508816 punch from 0x036edf to 0x03bb40 (0x4c62 bytes) 000846 1786166757.526580 write 0x001561 thru 0x00bd02 (0xa7a2 bytes) 000847 1786166757.535358 trunc from 0x00c9e9 to 0x013d96 (0x73ae bytes) 000848 1786166757.546475 punch from 0x00c3c0 to 0x01c311 (0xff52 bytes) 000849 1786166757.592799 trunc from 0x0162c2 to 0x021a25 (0xb764 bytes) 000850 1786166757.607782 trunc from 0x00a531 to 0x015b30 (0xb600 bytes) 000851 1786166757.623840 trunc from 0x03bb41 to 0x00b389 (0x307b7 bytes) 000852 1786166757.656383 trunc from 0x00b38a to 0x018863 (0xd4da bytes) 000853 1786166757.718660 mapread 0x008c97 thru 0x017c70 (0xefda bytes) 000854 1786166757.747677 fallocate 0x029536 thru 0x037327 (0xddf2 bytes) 000855 1786166757.764718 read 0x008a82 thru 0x012b30 (0xa0af bytes) 000856 1786166757.765775 fallocate 0x02355b thru 0x025799 (0x223f bytes) 000857 1786166757.792148 mapread 0x000e27 thru 0x00935c (0x8536 bytes) 000858 1786166757.797738 trunc from 0x017176 to 0x0252fa (0xe185 bytes) 000859 1786166757.819912 mapread 0x0153dd thru 0x01ab69 (0x578d bytes) 000860 1786166757.865811 trunc from 0x005c3f to 0x00b600 (0x59c2 bytes) 000861 1786166757.887796 fallocate 0x01d015 thru 0x02681f (0x980b bytes) 000862 1786166757.924200 punch from 0x00c387 to 0x01bf44 (0xfbbe bytes) 000863 1786166757.937782 fallocate 0x008719 thru 0x0091ad (0xa95 bytes) 000865 1786166757.959694 read 0x0048af thru 0x00eb46 (0xa298 bytes) 000866 1786166758.007612 write 0x02c0f2 thru 0x033eb5 (0x7dc4 bytes) 000867 1786166758.019801 write 0x03fb18 thru 0x03ffff (0x4e8 bytes) 000868 1786166758.025470 punch from 0x0016e4 to 0x00339b (0x1cb8 bytes) 000869 1786166758.039390 read 0x0326b6 thru 0x03f7ac (0xd0f7 bytes) 000870 1786166758.059871 punch from 0x00ed12 to 0x01b242 (0xc531 bytes) 000871 1786166758.075599 punch from 0x0264fa to 0x031600 (0xb107 bytes) 000872 1786166758.107529 mapread 0x00bbcb thru 0x00cfaa (0x13e0 bytes) 000873 1786166758.108581 trunc from 0x040000 to 0x02e949 (0x116b6 bytes) 000874 1786166758.177794 read 0x011b83 thru 0x0160b4 (0x4532 bytes) 000875 1786166758.227362 read 0x000bbf thru 0x00f3be (0xe800 bytes) 000876 1786166758.234751 mapwrite 0x0000a3 thru 0x00c022 (0xbf80 bytes) 000877 1786166758.290531 mapread 0x00505b thru 0x009235 (0x41db bytes) 000878 1786166758.291610 trunc from 0x0061d8 to 0x00e873 (0x869c bytes) 000879 1786166758.299839 mapwrite 0x0224e2 thru 0x031d85 (0xf8a4 bytes) 000880 1786166758.440274 fallocate 0x00fa70 thru 0x0108c2 (0xe53 bytes) 000881 1786166758.460704 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: 0x790f5336 (stale) Mirror 10: 0x790f5336 (stale) Mirror 11: 0x790f5336 (stale) Mirror 12: 0x790f5336 (stale) CRC-32 checksum value for chunk [0, 0x31d86): Mirror 1: 0x88a4a04c Mirror 13: 0x88a4a04c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d86: skipped 000882 1786166760.101063 fallocate 0x0394d9 thru 0x039ff2 (0xb1a bytes) 000883 1786166760.170322 mapread 0x00edc2 thru 0x01d1f9 (0xe438 bytes) 000884 1786166760.233730 fallocate 0x00ca85 thru 0x01a55c (0xdad8 bytes) 000885 1786166760.263679 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: 0x790f5336 (stale) Mirror 10: 0x790f5336 (stale) Mirror 11: 0x790f5336 (stale) Mirror 13: 0x88a4a04c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: 0x790f5336 (stale) Mirror 10: 0x790f5336 (stale) Mirror 11: 0x790f5336 (stale) Mirror 13: 0x88a4a04c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d86: skipped 000886 1786166760.963213 trunc from 0x02a371 to 0x02c75f (0x23ef bytes) 000887 1786166760.977269 mapwrite 0x00f9a6 thru 0x01b8ed (0xbf48 bytes) 000888 1786166761.068987 read 0x02a9b4 thru 0x02c39d (0x19ea bytes) 000889 1786166761.083350 write 0x02b06b thru 0x031c11 (0x6ba7 bytes) 000890 1786166761.133906 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: 0x790f5336 (stale) Mirror 10: 0x790f5336 (stale) Mirror 11: 0x790f5336 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d86): Mirror 6: 0x790f5336 (stale) Mirror 10: 0x790f5336 (stale) Mirror 11: 0x790f5336 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d86: skipped 000891 1786166761.817133 trunc from 0x031d86 to 0x028eee (0x8e97 bytes) 000892 1786166761.855781 punch from 0x023989 to 0x028eee (0x5566 bytes) 000893 1786166761.864632 trunc from 0x0202e0 to 0x028eee (0x8c0f bytes) 000894 1786166761.875763 mapread 0x01583c thru 0x01ac01 (0x53c6 bytes) 000895 1786166761.924151 trunc from 0x013502 to 0x01ef6e (0xba6d bytes) 000896 1786166761.934117 mapread 0x019f6e thru 0x028331 (0xe3c4 bytes) 000897 1786166761.992361 MIRROR_RESYNC 4 mirrors 000898 1786166764.016556 punch from 0x028404 to 0x028eee (0xaeb bytes) 000899 1786166764.119192 mapwrite 0x002b23 thru 0x00a30b (0x77e9 bytes) 000900 1786166764.239641 read 0x024339 thru 0x028eee (0x4bb6 bytes) 000901 1786166764.291083 read 0x01f09e thru 0x0279bc (0x891f bytes) 000902 1786166764.328289 write 0x036011 thru 0x03ffff (0x9fef bytes) 000903 1786166764.338667 read 0x034bff thru 0x03ffff (0xb401 bytes) 000905 1786166764.382695 read 0x035d3a thru 0x03ffff (0xa2c6 bytes) 000906 1786166764.398304 MIRROR_RESYNC 4 mirrors 000907 1786166766.680206 MIRROR_EXTEND to 5 mirrors 000908 1786166767.744932 mapwrite 0x021032 thru 0x02d5c7 (0xc596 bytes) 000909 1786166767.885877 fallocate 0x01c93b thru 0x0236e6 (0x6dac bytes) 000910 1786166767.894817 fallocate 0x0180d6 thru 0x0244f3 (0xc41e bytes) 000912 1786166767.901557 mapread 0x01dbce thru 0x0250f9 (0x752c bytes) 000913 1786166767.902995 mapwrite 0x038277 thru 0x03ffff (0x7d89 bytes) 000914 1786166767.929554 punch from 0x00a532 to 0x0100b3 (0x5b82 bytes) 000915 1786166767.934508 fallocate 0x0110f3 thru 0x011d68 (0xc76 bytes) 000917 1786166767.939651 write 0x01d13b thru 0x02d02a (0xfef0 bytes) 000918 1786166767.946188 write 0x03aa49 thru 0x03ffff (0x55b7 bytes) 000919 1786166767.948114 mapread 0x037156 thru 0x03ffff (0x8eaa bytes) 000920 1786166767.950258 punch from 0x02734a to 0x032839 (0xb4f0 bytes) 000921 1786166767.977964 fallocate 0x004cb0 thru 0x00ee1a (0xa16b bytes) 000923 1786166767.987736 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x65bbdfe6 (stale) Mirror 11: 0x65bbdfe6 (stale) Mirror 12: 0x65bbdfe6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x65bbdfe6 (stale) Mirror 11: 0x65bbdfe6 (stale) Mirror 12: 0x65bbdfe6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000924 1786166768.385820 trunc from 0x03fb92 to 0x03ffff (0x46e bytes) 000925 1786166768.391531 trunc from 0x040000 to 0x02619f (0x19e60 bytes) 000926 1786166768.416127 mapread 0x015dd1 thru 0x021551 (0xb781 bytes) 000927 1786166768.444837 fallocate 0x02d85e thru 0x030f2b (0x36ce bytes) 000928 1786166768.450234 fallocate 0x00c15e thru 0x00fc9f (0x3b42 bytes) 000929 1786166768.455643 punch from 0x000176 to 0x00ddc9 (0xdc54 bytes) 000930 1786166768.460883 read 0x0244fe thru 0x02619f (0x1ca2 bytes) 000931 1786166768.462232 trunc from 0x0261a0 to 0x029c00 (0x3a61 bytes) 000932 1786166768.488245 punch from 0x00a84b to 0x00dcde (0x3494 bytes) 000933 1786166768.493127 write 0x03760c thru 0x03c17f (0x4b74 bytes) 000935 1786166768.495992 fallocate 0x000405 thru 0x0079df (0x75db bytes) 000936 1786166768.501548 punch from 0x0189f9 to 0x022f14 (0xa51c bytes) 000937 1786166768.507298 write 0x0103a3 thru 0x013d9e (0x39fc bytes) 000938 1786166768.509248 mapread 0x003f9f thru 0x0102c9 (0xc32b bytes) 000939 1786166768.520463 punch from 0x00cf6e to 0x013460 (0x64f3 bytes) 000940 1786166768.538653 punch from 0x0170c8 to 0x024114 (0xd04d bytes) 000941 1786166768.544299 mapwrite 0x034aa2 thru 0x03db58 (0x90b7 bytes) 000942 1786166768.592862 trunc from 0x03db59 to 0x01fbef (0x1df69 bytes) 000943 1786166768.606891 trunc from 0x01fbf0 to 0x0261fb (0x660c bytes) 000944 1786166768.630149 mapwrite 0x038812 thru 0x03ffff (0x77ee bytes) 000945 1786166768.669800 punch from 0x014497 to 0x0162c9 (0x1e33 bytes) 000946 1786166768.677289 read 0x007470 thru 0x014ce5 (0xd876 bytes) 000948 1786166768.689669 mapread 0x0083ea thru 0x014caf (0xc8c6 bytes) 000949 1786166768.691680 read 0x027223 thru 0x036a6a (0xf848 bytes) 000950 1786166768.692725 fallocate 0x0027cf thru 0x007c7f (0x54b1 bytes) 000951 1786166768.698820 trunc from 0x0114fd to 0x017331 (0x5e35 bytes) 000952 1786166768.703870 trunc from 0x01ec2e to 0x0234ab (0x487e bytes) 000953 1786166768.709603 punch from 0x00bc8a to 0x013411 (0x7788 bytes) 000954 1786166768.715262 mapwrite 0x01beb2 thru 0x021ba0 (0x5cef bytes) 000955 1786166768.742896 trunc from 0x010a83 to 0x01c560 (0xbade bytes) 000956 1786166768.748507 read 0x03c4d1 thru 0x03ffff (0x3b2f bytes) 000957 1786166768.749446 punch from 0x01a489 to 0x01ffd6 (0x5b4e bytes) 000958 1786166768.756172 trunc from 0x01a930 to 0x01ae30 (0x501 bytes) 000959 1786166768.761007 punch from 0x034cb8 to 0x03dd95 (0x90de bytes) 000960 1786166768.767365 trunc from 0x01015b to 0x015df9 (0x5c9f bytes) 000961 1786166768.775241 trunc from 0x040000 to 0x01d50c (0x22af3 bytes) 000962 1786166768.791477 trunc from 0x01d50d to 0x025b13 (0x8607 bytes) 000963 1786166768.810298 trunc from 0x00aa11 to 0x0134ec (0x8adc bytes) 000964 1786166768.820413 read 0x011f52 thru 0x015dc5 (0x3e74 bytes) 000965 1786166768.837851 fallocate 0x03bb74 thru 0x03ffff (0x448c bytes) 000966 1786166768.856735 trunc from 0x011c89 to 0x01c7e2 (0xab5a bytes) 000967 1786166768.872721 trunc from 0x025b14 to 0x029bee (0x40db bytes) 000968 1786166768.924436 punch from 0x008100 to 0x012cf5 (0xabf6 bytes) 000969 1786166768.960392 trunc from 0x029bef to 0x019d72 (0xfe7c bytes) 000970 1786166768.987360 punch from 0x0072fc to 0x00d5d2 (0x62d7 bytes) 000971 1786166769.006230 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19d73): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19d73): Mirror 10: 0x7b84b660 (stale) Mirror 11: 0x7b84b660 (stale) Mirror 12: 0x7b84b660 (stale) CRC-32 checksum value for chunk [0, 0x19d73): Mirror 1: 0x3d94654f Mirror 13: 0x3d94654f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19d73: skipped 000973 1786166770.516918 read 0x0068cf thru 0x012eb0 (0xc5e2 bytes) 000974 1786166770.558824 punch from 0x00aafc to 0x017b74 (0xd079 bytes) 000975 1786166770.633597 trunc from 0x00e082 to 0x0180f1 (0xa070 bytes) 000976 1786166770.646202 trunc from 0x019d73 to 0x025b6c (0xbdfa bytes) 000977 1786166770.719134 trunc from 0x025b6d to 0x0044cc (0x216a0 bytes) 000978 1786166770.748381 write 0x00c795 thru 0x019f9f (0xd80b bytes) 000980 1786166770.758387 read 0x010d7c thru 0x016002 (0x5287 bytes) 000981 1786166770.759292 mapread 0x00fde5 thru 0x019f9f (0xa1bb bytes) 000983 1786166770.773592 read 0x00cde3 thru 0x01779d (0xa9bb bytes) 000984 1786166770.781894 punch from 0x00fac6 to 0x017fcc (0x8507 bytes) 000985 1786166770.820969 trunc from 0x00bca1 to 0x019f9f (0xe2ff bytes) 000986 1786166770.838869 trunc from 0x00d911 to 0x00df91 (0x681 bytes) 000987 1786166770.855555 mapwrite 0x02e8d2 thru 0x03556a (0x6c99 bytes) 000988 1786166770.973076 trunc from 0x005f89 to 0x0071a3 (0x121b bytes) 000989 1786166770.988066 write 0x000951 thru 0x002546 (0x1bf6 bytes) 000990 1786166770.989869 fallocate 0x008a4a thru 0x008b47 (0xfe bytes) 000991 1786166771.003601 write 0x03d101 thru 0x03ffff (0x2eff bytes) 000992 1786166771.004792 mapread 0x00d587 thru 0x0122bb (0x4d35 bytes) 000993 1786166771.005928 write 0x029d53 thru 0x036a2b (0xccd9 bytes) 000994 1786166771.033316 fallocate 0x00406d thru 0x00b21b (0x71af bytes) 000995 1786166771.047726 trunc from 0x040000 to 0x035d37 (0xa2c8 bytes) 000997 1786166771.079774 mapread 0x005165 thru 0x0140ea (0xef86 bytes) 000999 1786166771.113968 punch from 0x0006b7 to 0x00335e (0x2ca8 bytes) 001000 1786166771.141715 punch from 0x02781d to 0x028d29 (0x150d bytes) All operations completed A-OK! PASS 70a (73s) == sanity-flr test 71: check mirror extend parent fid ==== 01:26:17 (1786166777) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x146:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 01:26:29 (1786166789) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133535 s, 15.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114516 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:26:38 (1786166798) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6508384 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6933632 .. Extending file size to 7727008 .. Extending file size to 8329152 .. Extending file size to 7908128 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8327904 .. Extending file size to 8340992 .. Extending file size to 8612288 .. Extending file size to 8446528 .. done Extending file size to 9181216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8702080 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61199 61200 61203 61204 61207 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:27:59 (1786166879) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2277792 .. Extending file size to 3273280 .. failed Extending file size to 7958400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2410592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7498368 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8804576 .. failed Extending file size to 9108096 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9247456 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62563 62564 62566 62568 62570 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:29:22 (1786166962) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24768 3582252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18860 3588160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43628 7170412 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24768 3582252 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18860 3588160 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43628 7170412 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0918889 s, 22.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197355 s, 10.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9d4453d9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:29:41 (1786166981) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114145 s, 18.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 01:29:49 (1786166989) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x146:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x146:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x147:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:29:56 (1786166996) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104693 s, 20.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:30:04 (1786167004) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108893 s, 19.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x148:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x149:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x148:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x149:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x146:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:30:13 (1786167013) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109891 s, 19.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:30:47 (1786167047) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128682 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bd:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:31:37 (1786167097) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197732 s, 10.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:32:31 (1786167151) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg234-server: Pool lustre.test_204f created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180426 s, 11.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg234-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:33:50 (1786167230) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:33:58 (1786167238) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:34:07 (1786167247) oleg234-server: Pool lustre.test_206 created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg234-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:34:36 (1786167276) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783491 s, 13.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054266 s, 19.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd94382aa Mirror 3: 0xd94382aa PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:34:48 (1786167288) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:34:52 (1786167292) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:34:56 (1786167296) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0624525 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x960000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x960000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x960000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1630000 Read this (4 bytes): BBBB 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:35:34 (1786167334) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0147778 s, 554 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:35:41 (1786167341) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773994 s, 13.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb0 lmm_fid: [0x200000405:0xb0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb0 lmm_fid: [0x200000405:0xb0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611312 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-08 01:35:43.000000000 -0400 Modify: 2026-08-08 01:35:43.000000000 -0400 Change: 2026-08-08 01:35:45.000000000 -0400 Birth: 2026-08-08 01:35:43.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:35:51 (1786167351) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364343 s, 28.8 MB/s oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg234-server: O/280000401/d22/156: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:36:18 (1786167378) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.360448 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0326142 s, 126 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 8 01:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 8 01:36 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:36:30 (1786167390) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:36:38 (1786167398) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1900 sec =========== 01:36:48 (1786167408) === sanity-flr: start cleanup 01:36:49 (1786167409) === === sanity-flr: finish cleanup 01:36:58 (1786167418) ===