-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 14:45:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg248-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:45:43 (1786733143) === oleg248-client.virtnet: executing check_config_client /mnt/lustre oleg248-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg248-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8af019070800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8af019070800.idle_timeout=debug disable quota as required oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:46:21 (1786733181) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 14:46:24 (1786733184) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:46:45 (1786733205) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:46:49 (1786733209) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:46:53 (1786733213) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:47:17 (1786733237) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:47:20 (1786733240) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:47:23 (1786733243) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8af019070800.stats=clear osc.lustre-OST0001-osc-ffff8af019070800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:47:32 (1786733252) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:47:43 (1786733263) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:47:51 (1786733271) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:47:54 (1786733274) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:48:03 (1786733283) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 14:48:15 (1786733295) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:48:23 (1786733303) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:48:32 (1786733312) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:48:41 (1786733321) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:48:52 (1786733332) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:48:59 (1786733339) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.12387 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930195 s, 11.3 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:49:08 (1786733348) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.427478 s, 44.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:49:17 (1786733357) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:49:29 (1786733369) Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:50:37 (1786733437) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.330205 s, 22.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:51:21 (1786733481) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:52:46 (1786733566) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.3832 s, 15.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 14:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.951s user 0m0.001s sys 0m0.205s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.133s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.235s user 0m0.001s sys 0m0.168s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8af019070800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:54:23 (1786733663) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:54:27 (1786733667) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:54:31 (1786733671) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:54:35 (1786733675) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 14:54:46 (1786733686) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.796397 s, 15.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:55:00 (1786733700) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405363 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603396 s, 17.4 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:55:09 (1786733709) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:55:18 (1786733718) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00160849 s, 637 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127793 s, 801 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215818 s, 474 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.002036 s, 503 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 14:55:29 (1786733729) create test files with size 15746650 .. 060b360887888b64466d2bf5bd941cfd - 93491ed1f1fc5c5a2e39fcdf4953d123 - 5ee2cdb42c8a3578385244144cfb48ba - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 14:55:53 (1786733753) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:56:08 (1786733768) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:56:15 (1786733775) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615446 s, 17.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715246 s, 14.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 14:56:26 (1786733786) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0963481 s, 21.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.361887 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147041 s, 14.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 14:56:42 (1786733802) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:56:45 (1786733805) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:56:47 (1786733807) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335201 s, 31.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 14:56:54 (1786733814) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185072 s, 17.0 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:57:04 (1786733824) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:57:16 (1786733836) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.458071 s, 22.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:57:30 (1786733850) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8af01937f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.443567 s, 23.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8af003bb6000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:57:47 (1786733867) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:58:30 (1786733910) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:59:08 (1786733948) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:59:11 (1786733951) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.26932 s, 11.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223892 s, 14.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x86cbe5ec (nosync,stale) Mirror 2: 0x86cbe5ec (nosync,stale) ** verify mirror 2 doesn't change original checksum: e86dc6ab64be1467326ff2c05eba4ac3 - mirror 1 checksum: 5f5af569c2e7899809370b38e278350e - mirror 2 checksum: e86dc6ab64be1467326ff2c05eba4ac3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T14:59:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T14:59:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e86dc6ab64be1467326ff2c05eba4ac3 - mirror 1 checksum: 5f5af569c2e7899809370b38e278350e - mirror 2 checksum: 5f5af569c2e7899809370b38e278350e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T14:59:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T14:59:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:59:22 (1786733962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712286 s, 14.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:59:31 (1786733971) Failing mds1 on oleg248-server Stopping /mnt/lustre-mds1 (opts:) on oleg248-server 14:59:43 (1786733983) shut down facet: mds1 facet_host: oleg248-server facet_failover_host: oleg248-server Failover mds1 to oleg248-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 15:00:08 (1786734008) targets are mounted 15:00:08 (1786734008) facet_failover done oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355476 s, 29.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:00:24 (1786734024) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0531674 s, 385 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228631 s, 896 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:00:36 (1786734036) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.109159 s, 188 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0349799 s, 585 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (17s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:00:53 (1786734053) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00717703 s, 2.3 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:01:04 (1786734064) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0399002 s, 513 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:01:12 (1786734072) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.01095 s, 374 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309986 s, 132 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:01:20 (1786734080) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:01:41 (1786734101) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:02:09 (1786734129) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:02:33 (1786734153) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:02:58 (1786734178) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:03:05 (1786734185) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57579 mirror split pid 57581 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6972: 57581 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6972: 57581 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57579 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57581 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 15:04:13 (1786734253) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4256 000001 1786734256.541678 mapwrite 0x0127e5 thru 0x01bf42 (0x975e bytes) 000002 1786734256.631322 read 0x015079 thru 0x01bf42 (0x6eca bytes) zero_range to largest ever: 0x1bf43 000003 1786734256.632491 trunc from 0x00fe1e to 0x01bf42 (0xc125 bytes) 000004 1786734256.644041 read 0x0039f5 thru 0x012196 (0xe7a2 bytes) fallocating to largest ever: 0x33eb5 000005 1786734256.670052 fallocate 0x02cc86 thru 0x033eb4 (0x722f bytes) 000008 1786734256.682548 mapwrite 0x003e47 thru 0x00628a (0x2444 bytes) 000009 1786734256.702014 punch from 0x008199 to 0x00d75e (0x55c6 bytes) 000010 1786734256.709237 trunc from 0x005b02 to 0x013951 (0xde50 bytes) 000011 1786734256.717049 fallocate 0x01d96a thru 0x020c42 (0x32d9 bytes) truncating to largest ever: 0x3ac7a 000012 1786734256.725142 trunc from 0x020c43 to 0x03ac79 (0x1a037 bytes) 000013 1786734256.746198 trunc from 0x01b1be to 0x01d02c (0x1e6f bytes) 000014 1786734256.753450 mapwrite 0x004fcd thru 0x00f4db (0xa50f bytes) 000015 1786734256.964970 read 0x039b0c thru 0x03ac79 (0x116e bytes) 000016 1786734256.966018 fallocate 0x028a13 thru 0x030547 (0x7b35 bytes) 000017 1786734256.978998 read 0x01b727 thru 0x01d016 (0x18f0 bytes) 000018 1786734256.995501 read 0x013b48 thru 0x020a0e (0xcec7 bytes) 000019 1786734256.999582 trunc from 0x03ac7a to 0x032ec8 (0x7db1 bytes) 000020 1786734257.025179 read 0x0164f7 thru 0x01c22f (0x5d39 bytes) 000022 1786734257.026757 trunc from 0x00390a to 0x008fb3 (0x56aa bytes) 000023 1786734257.053233 mapread 0x023b44 thru 0x026108 (0x25c5 bytes) 000024 1786734257.063911 mapwrite 0x01bd58 thru 0x020c7b (0x4f24 bytes) 000025 1786734257.135200 fallocate 0x029eee thru 0x039b9a (0xfcad bytes) 000026 1786734257.148654 write 0x03c88e thru 0x03cd1a (0x48d bytes) 000028 1786734257.159451 mapwrite 0x01f541 thru 0x02de2b (0xe8eb bytes) 000029 1786734257.224320 trunc from 0x03cd1b to 0x00e341 (0x2e9d9 bytes) 000030 1786734257.257373 mapwrite 0x0217b7 thru 0x02a918 (0x9162 bytes) 000031 1786734257.352638 MIRROR_EXTEND to 2 mirrors 000032 1786734258.336188 MIRROR_RESYNC 2 mirrors 000033 1786734258.756915 punch from 0x007d79 to 0x0140cc (0xc354 bytes) 000034 1786734258.809308 mapwrite 0x02778c thru 0x02fcfa (0x856f bytes) 000036 1786734258.922336 trunc from 0x029fa7 to 0x02ea57 (0x4ab1 bytes) 000037 1786734258.928818 trunc from 0x02fcfb to 0x02fbcd (0x12d bytes) 000038 1786734258.949322 mapread 0x01fb08 thru 0x024813 (0x4d0c bytes) 000039 1786734258.964534 read 0x0030ed thru 0x004249 (0x115d bytes) 000040 1786734258.976159 mapwrite 0x00f9e5 thru 0x01aa27 (0xb043 bytes) 000041 1786734259.027627 punch from 0x02675f to 0x02fbcd (0x946f bytes) 000042 1786734259.036155 mapwrite 0x0300f5 thru 0x0337a8 (0x36b4 bytes) 000043 1786734259.090547 mapread 0x02fae5 thru 0x0337a8 (0x3cc4 bytes) 000044 1786734259.094009 punch from 0x017cbd to 0x018f8e (0x12d2 bytes) 000045 1786734259.106136 MIRROR_RESYNC 2 mirrors 000046 1786734259.845009 fallocate 0x025658 thru 0x029ad3 (0x447c bytes) 000048 1786734259.894260 fallocate 0x00277f thru 0x00df27 (0xb7a9 bytes) 000050 1786734259.903285 punch from 0x0289ef to 0x029f96 (0x15a8 bytes) 000051 1786734259.915456 mapread 0x02380e thru 0x02f136 (0xb929 bytes) 000052 1786734259.983241 read 0x009bff thru 0x00f94e (0x5d50 bytes) 000053 1786734259.998530 write 0x026614 thru 0x03168b (0xb078 bytes) 000054 1786734260.002247 write 0x01a2e2 thru 0x029ab0 (0xf7cf bytes) 000055 1786734260.013495 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000056 1786734260.287625 read 0x00b188 thru 0x00d073 (0x1eec bytes) 000057 1786734260.308650 read 0x004cc9 thru 0x00baf4 (0x6e2c bytes) 000058 1786734260.342249 mapwrite 0x0055d0 thru 0x00d172 (0x7ba3 bytes) 000059 1786734260.391848 trunc from 0x01c2e9 to 0x01ff35 (0x3c4d bytes) 000060 1786734260.413470 punch from 0x03270a to 0x0337a8 (0x109f bytes) 000062 1786734260.432616 trunc from 0x0337a9 to 0x001b78 (0x31c30 bytes) 000063 1786734260.492008 punch from 0x000568 to 0x001b78 (0x1611 bytes) 000064 1786734260.497524 write 0x01c2fb thru 0x02356e (0x7274 bytes) 000065 1786734260.500531 write 0x006dd9 thru 0x0079e2 (0xc0a bytes) 000066 1786734260.511310 mapwrite 0x01e314 thru 0x022698 (0x4385 bytes) 000069 1786734260.536152 mapwrite 0x01955d thru 0x025980 (0xc424 bytes) 000071 1786734260.614570 trunc from 0x011879 to 0x016757 (0x4edf bytes) 000072 1786734260.621624 trunc from 0x025981 to 0x03889b (0x12f1b bytes) 000073 1786734260.650582 mapread 0x01fe97 thru 0x02956a (0x96d4 bytes) 000074 1786734260.680632 trunc from 0x0122ba to 0x014b1f (0x2866 bytes) 000075 1786734260.686203 punch from 0x0078e8 to 0x01392c (0xc045 bytes) 000076 1786734260.692547 punch from 0x023db8 to 0x032cc8 (0xef11 bytes) 000077 1786734260.698560 mapwrite 0x006192 thru 0x00bebe (0x5d2d bytes) 000078 1786734260.746291 read 0x033a59 thru 0x03889b (0x4e43 bytes) 000079 1786734260.771619 write 0x0280b2 thru 0x03439d (0xc2ec bytes) 000081 1786734260.777419 trunc from 0x038203 to 0x03889b (0x699 bytes) 000082 1786734260.783074 write 0x02fd42 thru 0x037982 (0x7c41 bytes) 000083 1786734260.784857 mapwrite 0x01a75e thru 0x01b3e1 (0xc84 bytes) 000084 1786734260.808506 punch from 0x020315 to 0x020d00 (0x9ec bytes) 000088 1786734260.821449 mapwrite 0x031802 thru 0x03eac5 (0xd2c4 bytes) 000089 1786734260.871049 trunc from 0x025514 to 0x0256dd (0x1ca bytes) 000091 1786734260.878253 mapwrite 0x02e2f4 thru 0x030149 (0x1e56 bytes) 000093 1786734260.896061 mapread 0x00af12 thru 0x00fa29 (0x4b18 bytes) 000094 1786734260.897572 trunc from 0x020342 to 0x02067a (0x339 bytes) 000095 1786734260.903161 read 0x02b48e thru 0x0360a1 (0xac14 bytes) 000096 1786734260.904550 trunc from 0x037f11 to 0x03ad0b (0x2dfb bytes) 000097 1786734260.910119 mapread 0x01e198 thru 0x02916e (0xafd7 bytes) 000098 1786734260.937054 mapread 0x027e04 thru 0x029f45 (0x2142 bytes) 000099 1786734260.944345 write 0x019c27 thru 0x01a5ee (0x9c8 bytes) 000100 1786734260.946144 trunc from 0x00c3f8 to 0x01426f (0x7e78 bytes) 000101 1786734260.960317 read 0x02f330 thru 0x02fee0 (0xbb1 bytes) 000102 1786734260.961087 fallocate 0x02ee04 thru 0x03212c (0x3329 bytes) 000103 1786734260.969601 trunc from 0x027919 to 0x02a8b3 (0x2f9b bytes) 000104 1786734260.977483 trunc from 0x03eac6 to 0x008e1a (0x35cab bytes) 000105 1786734261.004317 read 0x0014e8 thru 0x008e1a (0x7933 bytes) 000106 1786734261.006016 write 0x03d5b8 thru 0x03ffff (0x2a48 bytes) 000107 1786734261.007930 read 0x03d30d thru 0x03ffff (0x2cf3 bytes) 000108 1786734261.009416 fallocate 0x017e58 thru 0x01f83d (0x79e6 bytes) 000110 1786734261.016430 punch from 0x02e7d0 to 0x02e87a (0xab bytes) 000111 1786734261.022395 punch from 0x0098e5 to 0x014fcf (0xb6eb bytes) 000112 1786734261.028320 mapwrite 0x0392b6 thru 0x03ffff (0x6d4a bytes) 000113 1786734261.062646 mapread 0x01b8a4 thru 0x0259f3 (0xa150 bytes) 000115 1786734261.067542 trunc from 0x040000 to 0x00c0d3 (0x33f2c bytes) 000116 1786734261.096429 read 0x008730 thru 0x00c0d3 (0x39a4 bytes) 000117 1786734261.098528 trunc from 0x00c0d4 to 0x0392c9 (0x2d1f6 bytes) 000119 1786734261.121222 trunc from 0x0392ca to 0x02df6c (0xb35d bytes) 000120 1786734261.141120 punch from 0x00be92 to 0x0111ce (0x533d bytes) 000121 1786734261.148222 mapwrite 0x008239 thru 0x0132bd (0xb085 bytes) 000122 1786734261.199533 mapread 0x027329 thru 0x02df6c (0x6c44 bytes) 000123 1786734261.202519 punch from 0x00e57e to 0x0113ca (0x2e4d bytes) 000124 1786734261.209331 fallocate 0x03516b thru 0x038cb4 (0x3b4a bytes) 000125 1786734261.217611 mapwrite 0x028cc5 thru 0x02d6bb (0x49f7 bytes) 000126 1786734261.237202 read 0x003b73 thru 0x004474 (0x902 bytes) 000127 1786734261.243685 MIRROR_EXTEND to 2 mirrors 000128 1786734262.359633 punch from 0x016ce4 to 0x02645d (0xf77a bytes) 000130 1786734262.407125 mapread 0x000e53 thru 0x00bc72 (0xae20 bytes) 000131 1786734262.425594 trunc from 0x02df6d to 0x012213 (0x1bd59 bytes) 000132 1786734262.492898 trunc from 0x012214 to 0x02bc6f (0x19a5c bytes) 000133 1786734262.522251 trunc from 0x02bc70 to 0x00437a (0x278f5 bytes) 000134 1786734262.543269 trunc from 0x000729 to 0x00437a (0x3c52 bytes) 000135 1786734262.549511 punch from 0x00395a to 0x00437a (0xa21 bytes) 000136 1786734262.555079 fallocate 0x02a785 thru 0x03737c (0xcbf8 bytes) 000137 1786734262.560384 punch from 0x01f8db to 0x021659 (0x1d7f bytes) 000138 1786734262.565487 read 0x024c9f thru 0x032357 (0xd6b9 bytes) 000139 1786734262.574501 mapwrite 0x024703 thru 0x0323f6 (0xdcf4 bytes) 000140 1786734262.607534 mapwrite 0x03aaef thru 0x03ffff (0x5511 bytes) 000141 1786734262.649337 mapread 0x01a454 thru 0x02a18c (0xfd39 bytes) 000142 1786734262.665515 write 0x01d0e3 thru 0x02611c (0x903a bytes) 000143 1786734262.667535 trunc from 0x040000 to 0x02e0a0 (0x11f5f bytes) 000145 1786734262.685234 mapwrite 0x03cd77 thru 0x03ffff (0x3289 bytes) 000146 1786734262.738577 fallocate 0x013a17 thru 0x01c339 (0x8923 bytes) 000147 1786734262.745386 mapwrite 0x005e14 thru 0x0147e4 (0xe9d1 bytes) 000148 1786734262.837361 MIRROR_RESYNC 2 mirrors 000150 1786734263.803392 fallocate 0x01e079 thru 0x028d0a (0xac92 bytes) 000151 1786734263.867495 mapwrite 0x02390b thru 0x02d251 (0x9947 bytes) 000152 1786734263.953501 trunc from 0x040000 to 0x02ad4e (0x152b1 bytes) 000153 1786734263.988578 fallocate 0x02b434 thru 0x0363fc (0xafc9 bytes) 000154 1786734264.000006 trunc from 0x02ad4f to 0x0062db (0x24a73 bytes) 000155 1786734264.022039 trunc from 0x005a45 to 0x0062db (0x897 bytes) 000156 1786734264.041520 read 0x00071e thru 0x0062db (0x5bbe bytes) 000157 1786734264.053164 punch from 0x005790 to 0x0062db (0xb4c bytes) 000158 1786734264.060355 trunc from 0x0037b0 to 0x0062db (0x2b2c bytes) 000159 1786734264.072185 write 0x02e575 thru 0x0306c2 (0x214e bytes) 000160 1786734264.080495 punch from 0x0046fa to 0x013fd7 (0xf8de bytes) 000161 1786734264.089434 trunc from 0x011ee9 to 0x014225 (0x233d bytes) 000162 1786734264.098343 trunc from 0x0008a9 to 0x00e6d3 (0xde2b bytes) 000163 1786734264.108584 trunc from 0x0306c3 to 0x027cc4 (0x89fe bytes) 000164 1786734264.127194 read 0x000bcd thru 0x00823c (0x7670 bytes) 000165 1786734264.159961 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27cc5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cc5): Mirror 2: 0xf1fb0469 (stale) CRC-32 checksum value for chunk [0, 0x27cc5): Mirror 1: 0x8feff470 Mirror 3: 0x8feff470 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cc5: skipped 000166 1786734265.137966 punch from 0x01e826 to 0x01ec49 (0x424 bytes) 000168 1786734265.208910 write 0x0337b7 thru 0x03bb64 (0x83ae bytes) 000169 1786734265.225753 read 0x01429e thru 0x01ad0e (0x6a71 bytes) 000170 1786734265.246511 write 0x034ba6 thru 0x03ffff (0xb45a bytes) 000171 1786734265.251267 write 0x02e873 thru 0x030eac (0x263a bytes) fallocating to largest ever: 0x3b67a 000172 1786734265.287840 fallocate 0x02bede thru 0x03b679 (0xf79c bytes) 000173 1786734265.312592 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000174 1786734265.825202 mapread 0x019d4c thru 0x022e90 (0x9145 bytes) 000175 1786734265.871592 MIRROR_RESYNC 2 mirrors 000176 1786734267.098293 mapwrite 0x03417d thru 0x0367bf (0x2643 bytes) 000177 1786734267.217291 mapread 0x01f369 thru 0x028b12 (0x97aa bytes) 000178 1786734267.256283 fallocate 0x01646f thru 0x01b3dc (0x4f6e bytes) 000179 1786734267.262328 read 0x02989d thru 0x029ce8 (0x44c bytes) 000180 1786734267.263660 fallocate 0x012ed8 thru 0x01c747 (0x9870 bytes) 000181 1786734267.277521 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2781dfb8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa8d0c990 Mirror 4: 0xa8d0c990 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped zero_range to largest ever: 0x40000 000182 1786734268.618096 trunc from 0x038115 to 0x03ffff (0x7eeb bytes) 000183 1786734268.692414 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2781dfb8 (stale) Mirror 4: 0xa8d0c990 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x30e5d378 Mirror 5: 0x30e5d378 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000184 1786734269.811741 punch from 0x00fac2 to 0x011b16 (0x2055 bytes) 000185 1786734269.857824 punch from 0x00bd11 to 0x00c57b (0x86b bytes) 000186 1786734269.871279 trunc from 0x030d58 to 0x0395b7 (0x8860 bytes) 000187 1786734269.877304 read 0x005cb4 thru 0x0140e3 (0xe430 bytes) 000188 1786734269.892117 trunc from 0x028321 to 0x03444a (0xc12a bytes) 000189 1786734269.898307 mapwrite 0x005cdc thru 0x006713 (0xa38 bytes) 000190 1786734269.919257 trunc from 0x01962f to 0x01fcf0 (0x66c2 bytes) 000191 1786734269.925266 write 0x02bc8b thru 0x03bb07 (0xfe7d bytes) 000192 1786734269.944912 punch from 0x003440 to 0x0106f5 (0xd2b6 bytes) 000193 1786734269.950165 trunc from 0x040000 to 0x02f594 (0x10a6b bytes) 000194 1786734269.988154 mapwrite 0x022314 thru 0x031ba5 (0xf892 bytes) 000195 1786734270.106244 write 0x030aae thru 0x03c02a (0xb57d bytes) 000196 1786734270.110117 mapwrite 0x015192 thru 0x01f63d (0xa4ac bytes) 000197 1786734270.189172 read 0x035e06 thru 0x03bbc8 (0x5dc3 bytes) 000198 1786734270.195509 mapread 0x01e199 thru 0x02ac12 (0xca7a bytes) 000199 1786734270.198572 mapwrite 0x03f57b thru 0x03ffff (0xa85 bytes) 000200 1786734270.285742 mapwrite 0x00727c thru 0x00d901 (0x6686 bytes) 000201 1786734270.320320 MIRROR_RESYNC 4 mirrors 000202 1786734271.999203 trunc from 0x040000 to 0x037217 (0x8de8 bytes) 000203 1786734272.034640 mapwrite 0x033341 thru 0x0398d8 (0x6598 bytes) 000204 1786734272.087955 mapread 0x02a8ea thru 0x02d962 (0x3079 bytes) 000205 1786734272.104553 MIRROR_RESYNC 4 mirrors 000206 1786734273.102987 write 0x01a43f thru 0x023c6b (0x982d bytes) 000207 1786734273.190182 read 0x00e2d6 thru 0x01d560 (0xf28b bytes) 000208 1786734273.205983 mapread 0x00ceed thru 0x00f612 (0x2726 bytes) 000209 1786734273.221453 trunc from 0x02c5f0 to 0x037cd3 (0xb6e4 bytes) 000210 1786734273.227984 mapread 0x0209ca thru 0x02ae03 (0xa43a bytes) 000211 1786734273.241484 trunc from 0x029a21 to 0x0359c5 (0xbfa5 bytes) 000212 1786734273.248866 fallocate 0x01f01a thru 0x029498 (0xa47f bytes) 000213 1786734273.255603 punch from 0x022b37 to 0x022e6f (0x339 bytes) 000214 1786734273.287386 write 0x03428c thru 0x034354 (0xc9 bytes) 000215 1786734273.299566 trunc from 0x0398d9 to 0x01e9f0 (0x1aee8 bytes) 000216 1786734273.365582 mapwrite 0x005e07 thru 0x00f0a8 (0x92a2 bytes) 000217 1786734273.403964 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e9f1): Mirror 3: 0xeaaa5c02 (stale) Mirror 5: 0xeaaa5c02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e9f1): Mirror 3: 0xeaaa5c02 (stale) Mirror 5: 0xeaaa5c02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e9f1: skipped 000218 1786734273.695333 fallocate 0x03eede thru 0x03ffff (0x1122 bytes) 000219 1786734273.702444 mapwrite 0x000441 thru 0x0063ea (0x5faa bytes) 000220 1786734273.750510 write 0x03b11a thru 0x03ffff (0x4ee6 bytes) 000221 1786734273.755926 fallocate 0x034c08 thru 0x035e86 (0x127f bytes) 000222 1786734273.760609 punch from 0x0050ed to 0x009dd6 (0x4cea bytes) 000223 1786734273.765590 trunc from 0x03400f to 0x0340f5 (0xe7 bytes) 000224 1786734273.776523 read 0x02d43b thru 0x03877c (0xb342 bytes) 000225 1786734273.811820 trunc from 0x040000 to 0x0172ab (0x28d54 bytes) 000227 1786734273.850260 write 0x036a37 thru 0x037297 (0x861 bytes) 000228 1786734273.867443 mapread 0x006365 thru 0x007c82 (0x191e bytes) 000229 1786734273.891332 write 0x03366e thru 0x03ffff (0xc992 bytes) 000230 1786734273.905562 fallocate 0x03b940 thru 0x03e7fd (0x2ebe bytes) 000231 1786734273.920162 trunc from 0x02ba29 to 0x02c9b3 (0xf8b bytes) 000232 1786734273.935990 mapwrite 0x0010c5 thru 0x001d17 (0xc53 bytes) 000233 1786734273.957927 punch from 0x02aee8 to 0x035817 (0xa930 bytes) 000234 1786734273.966335 write 0x00b713 thru 0x00b9bf (0x2ad bytes) 000235 1786734273.968924 mapwrite 0x02fe95 thru 0x03ecc4 (0xee30 bytes) 000236 1786734274.030949 mapread 0x02745c thru 0x027a4f (0x5f4 bytes) 000237 1786734274.052508 fallocate 0x019ba1 thru 0x0236e4 (0x9b44 bytes) 000238 1786734274.082399 punch from 0x00ae19 to 0x018d7f (0xdf67 bytes) 000239 1786734274.097925 punch from 0x03b360 to 0x03ffff (0x4ca0 bytes) 000241 1786734274.114054 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8260c502 (stale) Mirror 5: 0x8260c502 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf260fca6 Mirror 6: 0xf260fca6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1786734275.286675 fallocate 0x01a666 thru 0x02a3dd (0xfd78 bytes) 000243 1786734275.333700 write 0x0006a4 thru 0x0090b5 (0x8a12 bytes) 000244 1786734275.355039 trunc from 0x040000 to 0x02d66b (0x12994 bytes) 000245 1786734275.387140 read 0x02a0b0 thru 0x02afb8 (0xf09 bytes) 000246 1786734275.394060 trunc from 0x020a20 to 0x02401e (0x35ff bytes) 000247 1786734275.399188 trunc from 0x02d66c to 0x03019c (0x2b31 bytes) 000248 1786734275.418614 trunc from 0x03019d to 0x00b724 (0x24a78 bytes) 000249 1786734275.437160 trunc from 0x007206 to 0x00b724 (0x451f bytes) 000251 1786734275.500211 trunc from 0x006f86 to 0x007a6a (0xae5 bytes) 000252 1786734275.516104 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb725): Mirror 5: 0xa7b84d57 (stale) Mirror 6: 0xd144e03e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb725): Mirror 5: 0xa7b84d57 (stale) Mirror 6: 0xd144e03e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb725: skipped 000255 1786734275.884338 read 0x0059a4 thru 0x00b724 (0x5d81 bytes) 000257 1786734275.901379 mapread 0x006c3c thru 0x00b724 (0x4ae9 bytes) 000258 1786734275.906055 mapwrite 0x0300d1 thru 0x03d23d (0xd16d bytes) 000259 1786734276.025328 trunc from 0x018feb to 0x01de83 (0x4e99 bytes) 000261 1786734276.045001 mapwrite 0x001861 thru 0x00bbbc (0xa35c bytes) 000262 1786734276.096369 mapread 0x03884a thru 0x03b91f (0x30d6 bytes) 000263 1786734276.097932 fallocate 0x034259 thru 0x03d798 (0x9540 bytes) 000264 1786734276.108273 trunc from 0x03d23e to 0x03d861 (0x624 bytes) 000265 1786734276.135914 read 0x005278 thru 0x00b853 (0x65dc bytes) 000267 1786734276.140757 punch from 0x024271 to 0x032332 (0xe0c2 bytes) 000268 1786734276.146675 mapwrite 0x007898 thru 0x0162ae (0xea17 bytes) 000269 1786734276.209202 mapwrite 0x00cdc9 thru 0x012979 (0x5bb1 bytes) 000270 1786734276.227639 read 0x037fe7 thru 0x0380a2 (0xbc bytes) 000271 1786734276.235009 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d862): Mirror 5: 0x805ce1d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d862): Mirror 5: 0x805ce1d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d862: skipped 000272 1786734276.825196 trunc from 0x033c0f to 0x037c55 (0x4047 bytes) 000273 1786734276.834207 write 0x00588a thru 0x015869 (0xffe0 bytes) 000274 1786734276.856308 mapread 0x03d715 thru 0x03d861 (0x14d bytes) 000276 1786734276.868493 punch from 0x006d91 to 0x00e3e3 (0x7653 bytes) 000277 1786734276.896333 read 0x01d7f9 thru 0x02206f (0x4877 bytes) 000278 1786734276.912588 mapread 0x031530 thru 0x03d861 (0xc332 bytes) 000281 1786734276.942204 read 0x027be9 thru 0x029fa4 (0x23bc bytes) 000282 1786734276.982612 punch from 0x001419 to 0x003750 (0x2338 bytes) 000283 1786734277.002911 read 0x03d3a2 thru 0x03d861 (0x4c0 bytes) 000284 1786734277.007018 trunc from 0x014567 to 0x014740 (0x1da bytes) 000285 1786734277.032283 fallocate 0x030bc8 thru 0x03e012 (0xd44b bytes) 000286 1786734277.048028 fallocate 0x0290c7 thru 0x032960 (0x989a bytes) 000287 1786734277.065765 trunc from 0x03d862 to 0x02b32a (0x12537 bytes) 000288 1786734277.107575 trunc from 0x02b32b to 0x034da3 (0x9a79 bytes) 000290 1786734277.137613 punch from 0x00d1f0 to 0x00f6c2 (0x24d3 bytes) 000291 1786734277.150595 MIRROR_RESYNC 2 mirrors 000292 1786734278.204567 write 0x024c45 thru 0x029547 (0x4903 bytes) 000293 1786734278.326077 write 0x011560 thru 0x01a9cd (0x946e bytes) 000294 1786734278.340605 read 0x00fa5b thru 0x01d4fe (0xdaa4 bytes) 000295 1786734278.361653 fallocate 0x0287b2 thru 0x029de3 (0x1632 bytes) 000296 1786734278.379059 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34da4): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34da4): Mirror 5: 0x12cec62c (stale) CRC-32 checksum value for chunk [0, 0x34da4): Mirror 1: 0xa7b92f6c Mirror 6: 0xa7b92f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34da4: skipped 000297 1786734280.149635 fallocate 0x0067eb thru 0x00bab9 (0x52cf bytes) 000298 1786734280.191363 trunc from 0x005274 to 0x0089ca (0x3757 bytes) 000299 1786734280.199663 read 0x01cd52 thru 0x0275e7 (0xa896 bytes) 000300 1786734280.211004 trunc from 0x033587 to 0x034da3 (0x181d bytes) 000301 1786734280.216906 read 0x004512 thru 0x008289 (0x3d78 bytes) 000302 1786734280.227124 mapread 0x0033ef thru 0x00cd14 (0x9926 bytes) 000303 1786734280.228846 mapread 0x0192cc thru 0x0239ee (0xa723 bytes) 000304 1786734280.231357 trunc from 0x034da4 to 0x031a75 (0x332e bytes) 000305 1786734280.256183 trunc from 0x031a76 to 0x01b52b (0x1654a bytes) 000306 1786734280.273185 read 0x0180c3 thru 0x019045 (0xf83 bytes) 000307 1786734280.274237 mapwrite 0x016f37 thru 0x01e625 (0x76ef bytes) 000308 1786734280.326775 MIRROR_RESYNC 3 mirrors 000309 1786734281.114189 mapread 0x018220 thru 0x01ba71 (0x3852 bytes) 000310 1786734281.148119 trunc from 0x014754 to 0x01ce09 (0x86b6 bytes) 000311 1786734281.210395 trunc from 0x0015cb to 0x006bb4 (0x55ea bytes) 000312 1786734281.216328 mapwrite 0x0383da thru 0x039488 (0x10af bytes) 000313 1786734281.291834 mapwrite 0x0033b5 thru 0x009459 (0x60a5 bytes) 000315 1786734281.327981 punch from 0x02a86a to 0x0328f8 (0x808f bytes) 000317 1786734281.337296 trunc from 0x00f679 to 0x018520 (0x8ea8 bytes) 000318 1786734281.343854 punch from 0x005439 to 0x0054ac (0x74 bytes) 000319 1786734281.349276 trunc from 0x019757 to 0x021df0 (0x869a bytes) 000320 1786734281.357854 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39489): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39489): Mirror 5: 0xf7f93164 (stale) Mirror 6: 0xf7f93164 (stale) CRC-32 checksum value for chunk [0, 0x39489): Mirror 1: 0x22a72c37 Mirror 7: 0x22a72c37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39489: skipped 000321 1786734282.283610 trunc from 0x00787b to 0x00ccd9 (0x545f bytes) 000322 1786734282.325674 punch from 0x014454 to 0x01ef43 (0xaaf0 bytes) 000323 1786734282.333795 fallocate 0x00b71f thru 0x016e0f (0xb6f1 bytes) 000324 1786734282.340391 trunc from 0x014fd1 to 0x01691c (0x194c bytes) 000325 1786734282.349580 MIRROR_RESYNC 4 mirrors 000328 1786734283.792311 trunc from 0x039489 to 0x03b3c1 (0x1f39 bytes) 000329 1786734283.866887 punch from 0x00442d to 0x0069ff (0x25d3 bytes) 000330 1786734283.873340 mapread 0x003a83 thru 0x010f66 (0xd4e4 bytes) 000331 1786734283.897119 trunc from 0x03b3c2 to 0x008ec9 (0x324f8 bytes) 000333 1786734283.919698 fallocate 0x001783 thru 0x00e098 (0xc916 bytes) 000334 1786734283.927416 fallocate 0x0372b2 thru 0x0394ba (0x2209 bytes) 000335 1786734283.937550 mapread 0x02100d thru 0x02778c (0x6780 bytes) 000336 1786734283.967207 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x394bb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394bb): Mirror 5: 0xf03bc3d9 (stale) Mirror 6: 0xf03bc3d9 (stale) Mirror 7: 0xf03bc3d9 (stale) CRC-32 checksum value for chunk [0, 0x394bb): Mirror 1: 0x163dbbe9 Mirror 8: 0x163dbbe9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394bb: skipped 000337 1786734285.440691 trunc from 0x0394bb to 0x02eb6e (0xa94c bytes) 000338 1786734285.511513 read 0x003310 thru 0x009d33 (0x6a24 bytes) 000339 1786734285.524779 trunc from 0x02eb6f to 0x0351db (0x666d bytes) 000340 1786734285.567373 trunc from 0x013e2a to 0x01b32d (0x7504 bytes) 000341 1786734285.573387 fallocate 0x015c8d thru 0x022f24 (0xd298 bytes) 000342 1786734285.583381 trunc from 0x0351dc to 0x02c670 (0x8b6b bytes) 000343 1786734285.615436 fallocate 0x01b05f thru 0x02889e (0xd840 bytes) 000344 1786734285.626330 trunc from 0x0138e7 to 0x013d4f (0x469 bytes) 000345 1786734285.633132 read 0x013971 thru 0x021ce3 (0xe373 bytes) 000346 1786734285.655269 trunc from 0x02a7dd to 0x02abf2 (0x416 bytes) 000349 1786734285.664360 trunc from 0x00fd5b to 0x01b816 (0xbabc bytes) 000351 1786734285.680881 mapread 0x0197ce thru 0x020f4f (0x7782 bytes) 000352 1786734285.712079 read 0x01c12b thru 0x027750 (0xb626 bytes) 000353 1786734285.719287 write 0x02992e thru 0x02ae65 (0x1538 bytes) 000354 1786734285.734269 read 0x01e5a6 thru 0x02bfbb (0xda16 bytes) 000355 1786734285.735289 trunc from 0x00b53e to 0x0154b2 (0x9f75 bytes) 000356 1786734285.746079 write 0x008735 thru 0x0183c9 (0xfc95 bytes) 000357 1786734285.748142 trunc from 0x02c671 to 0x024aac (0x7bc4 bytes) 000359 1786734285.791262 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24aad): Mirror 6: 0x33ebb30 (stale) Mirror 7: 0x33ebb30 (stale) Mirror 8: 0xf21a3c19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24aad): Mirror 6: 0x33ebb30 (stale) Mirror 7: 0x33ebb30 (stale) Mirror 8: 0xf21a3c19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24aad: skipped 000360 1786734286.389280 trunc from 0x024aad to 0x00b6c9 (0x193e3 bytes) 000361 1786734286.417414 read 0x009ccf thru 0x00b6c9 (0x19fb bytes) 000362 1786734286.430893 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb6ca): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6ca): Mirror 6: 0x16b2da22 (stale) Mirror 7: 0x16b2da22 (stale) Mirror 8: 0x1d85df93 (stale) CRC-32 checksum value for chunk [0, 0xb6ca): Mirror 1: 0x9913e98 Mirror 9: 0x9913e98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6ca: skipped 000363 1786734287.402574 write 0x03b708 thru 0x03ffff (0x48f8 bytes) 000364 1786734287.473759 mapread 0x020ddf thru 0x026b25 (0x5d47 bytes) 000365 1786734287.509018 punch from 0x03df4c to 0x03ffff (0x20b4 bytes) 000366 1786734287.534982 trunc from 0x040000 to 0x038eed (0x7112 bytes) 000367 1786734287.625128 fallocate 0x02ef06 thru 0x0319cf (0x2aca bytes) 000368 1786734287.636182 fallocate 0x01416d thru 0x0200b4 (0xbf48 bytes) 000369 1786734287.646688 fallocate 0x01b55a thru 0x01fa9b (0x4542 bytes) 000370 1786734287.654755 read 0x028a52 thru 0x034a7b (0xc02a bytes) 000371 1786734287.673494 mapwrite 0x023247 thru 0x02b28c (0x8046 bytes) 000372 1786734287.712476 trunc from 0x038eee to 0x021dff (0x170ee bytes) 000373 1786734287.737101 punch from 0x006ff7 to 0x014ae7 (0xdaf1 bytes) 000374 1786734287.745243 punch from 0x01ba0a to 0x021dff (0x63f6 bytes) 000375 1786734287.751646 fallocate 0x0223aa thru 0x026264 (0x3ebb bytes) 000376 1786734287.767114 MIRROR_RESYNC 5 mirrors 000377 1786734289.251195 trunc from 0x026265 to 0x0343ce (0xe16a bytes) 000378 1786734289.295299 mapread 0x016009 thru 0x022375 (0xc36d bytes) 000379 1786734289.325935 trunc from 0x0343cf to 0x01fd9c (0x14632 bytes) 000380 1786734289.346934 trunc from 0x01fd9d to 0x00e057 (0x11d45 bytes) 000381 1786734289.372639 trunc from 0x00e058 to 0x03a8e7 (0x2c890 bytes) 000382 1786734289.393028 fallocate 0x02572d thru 0x0268b3 (0x1187 bytes) 000384 1786734289.399869 read 0x022af5 thru 0x02783f (0x4d4b bytes) 000385 1786734289.409916 mapwrite 0x012b50 thru 0x01b3f2 (0x88a3 bytes) 000386 1786734289.451813 trunc from 0x01bf63 to 0x01cef1 (0xf8f bytes) 000387 1786734289.458348 write 0x01b47c thru 0x020086 (0x4c0b bytes) 000388 1786734289.464880 mapread 0x01aeec thru 0x026fe5 (0xc0fa bytes) 000389 1786734289.466411 fallocate 0x01e9d8 thru 0x022508 (0x3b31 bytes) 000390 1786734289.472904 fallocate 0x031620 thru 0x033986 (0x2367 bytes) 000391 1786734289.478954 punch from 0x022d11 to 0x02b9f9 (0x8ce9 bytes) 000392 1786734289.485193 write 0x01d98b thru 0x02d6bf (0xfd35 bytes) 000393 1786734289.494084 mapwrite 0x039667 thru 0x03ec47 (0x55e1 bytes) 000394 1786734289.543371 punch from 0x005a4b to 0x00a104 (0x46ba bytes) 000395 1786734289.551191 trunc from 0x0349b5 to 0x0389ba (0x4006 bytes) 000396 1786734289.557977 trunc from 0x03ec48 to 0x03ed89 (0x142 bytes) 000397 1786734289.575108 mapread 0x019dbd thru 0x021131 (0x7375 bytes) 000398 1786734289.581859 MIRROR_RESYNC 5 mirrors 000399 1786734291.267512 trunc from 0x01b3cd to 0x0215d7 (0x620b bytes) 000400 1786734291.302277 trunc from 0x02fc69 to 0x036fc5 (0x735d bytes) 000401 1786734291.309325 trunc from 0x03ed8a to 0x01036e (0x2ea1b bytes) 000403 1786734291.330824 read 0x007939 thru 0x00febb (0x8583 bytes) 000406 1786734291.347617 mapread 0x006bed thru 0x007b01 (0xf15 bytes) 000407 1786734291.354957 read 0x002e19 thru 0x01036e (0xd556 bytes) 000408 1786734291.365782 read 0x003979 thru 0x01036e (0xc9f6 bytes) 000409 1786734291.367196 read 0x00e80e thru 0x01036e (0x1b61 bytes) 000410 1786734291.368825 trunc from 0x00b5a8 to 0x01036e (0x4dc7 bytes) 000411 1786734291.374326 punch from 0x00f75a to 0x01036e (0xc15 bytes) 000412 1786734291.380919 mapwrite 0x038554 thru 0x03ffff (0x7aac bytes) 000413 1786734291.431889 punch from 0x017e7c to 0x024c07 (0xcd8c bytes) 000414 1786734291.440951 trunc from 0x040000 to 0x030789 (0xf876 bytes) 000415 1786734291.469054 read 0x00349a thru 0x0041c4 (0xd2b bytes) 000416 1786734291.469946 read 0x0043b7 thru 0x00e5d8 (0xa222 bytes) 000417 1786734291.471976 write 0x0143a7 thru 0x01e019 (0x9c73 bytes) 000418 1786734291.479249 trunc from 0x03078a to 0x03c8a1 (0xc118 bytes) 000419 1786734291.495119 write 0x01e939 thru 0x02af5e (0xc626 bytes) 000420 1786734291.497922 read 0x02cd14 thru 0x03ac1b (0xdf08 bytes) 000421 1786734291.509185 write 0x036096 thru 0x037eda (0x1e45 bytes) 000422 1786734291.511933 write 0x01f82b thru 0x020e40 (0x1616 bytes) 000426 1786734291.515304 mapread 0x003686 thru 0x010efa (0xd875 bytes) 000427 1786734291.517688 trunc from 0x00a035 to 0x010a69 (0x6a35 bytes) 000428 1786734291.522680 trunc from 0x03c8a2 to 0x00a57f (0x32322 bytes) 000429 1786734291.545032 trunc from 0x003e47 to 0x00a57f (0x6739 bytes) 000430 1786734291.555383 MIRROR_RESYNC 5 mirrors 000431 1786734292.809987 read 0x003863 thru 0x004a86 (0x1224 bytes) 000432 1786734292.819326 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000433 1786734293.211094 trunc from 0x005299 to 0x00a57f (0x52e7 bytes) 000434 1786734293.250362 mapread 0x004f89 thru 0x00a57f (0x55f7 bytes) 000435 1786734293.267868 punch from 0x0084b6 to 0x00a57f (0x20ca bytes) 000436 1786734293.272126 punch from 0x0019db to 0x002258 (0x87e bytes) 000437 1786734293.276944 mapwrite 0x02571f thru 0x02aef8 (0x57da bytes) 000438 1786734293.335061 trunc from 0x02aef9 to 0x038c3c (0xdd44 bytes) 000439 1786734293.358962 mapwrite 0x008a76 thru 0x0156ca (0xcc55 bytes) 000440 1786734293.397686 mapread 0x025c11 thru 0x02ca8e (0x6e7e bytes) 000441 1786734293.409257 read 0x022f38 thru 0x031b62 (0xec2b bytes) 000442 1786734293.410198 punch from 0x0047ae to 0x0090fe (0x4951 bytes) 000443 1786734293.416879 read 0x017555 thru 0x01cda7 (0x5853 bytes) 000444 1786734293.427585 fallocate 0x03f97f thru 0x03ffff (0x681 bytes) 000445 1786734293.432487 mapread 0x00cd69 thru 0x017976 (0xac0e bytes) 000446 1786734293.438849 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38c3d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c3d): Mirror 6: 0x9ac474d1 (stale) Mirror 7: 0x9ac474d1 (stale) Mirror 8: 0x9ac474d1 (stale) CRC-32 checksum value for chunk [0, 0x38c3d): Mirror 1: 0x532b17af Mirror 9: 0x532b17af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c3d: skipped 000447 1786734294.742680 MIRROR_RESYNC 5 mirrors 000448 1786734296.337651 mapread 0x01565a thru 0x0215a0 (0xbf47 bytes) 000449 1786734296.391996 mapwrite 0x036d3c thru 0x038345 (0x160a bytes) 000450 1786734296.553320 trunc from 0x038c3d to 0x0222dd (0x1695f bytes) 000451 1786734296.601102 mapwrite 0x018f16 thru 0x020767 (0x7852 bytes) 000453 1786734296.635632 trunc from 0x01088d to 0x01bf72 (0xb6e6 bytes) 000454 1786734296.647182 MIRROR_RESYNC 5 mirrors 000455 1786734298.289990 mapread 0x019216 thru 0x0222dd (0x90c8 bytes) 000456 1786734298.302101 fallocate 0x00e2a0 thru 0x01b0e0 (0xce41 bytes) 000457 1786734298.330383 trunc from 0x0222de to 0x029a22 (0x7745 bytes) 000458 1786734298.364626 read 0x01a4ad thru 0x01a81c (0x370 bytes) 000459 1786734298.372956 punch from 0x0136d6 to 0x01e746 (0xb071 bytes) 000460 1786734298.379570 trunc from 0x01fb5c to 0x029a22 (0x9ec7 bytes) 000461 1786734298.385265 trunc from 0x029a23 to 0x00e624 (0x1b3fe bytes) 000462 1786734298.403059 trunc from 0x00e625 to 0x00cf3e (0x16e6 bytes) 000463 1786734298.422237 punch from 0x002e12 to 0x007362 (0x4551 bytes) 000464 1786734298.428745 punch from 0x00cb26 to 0x00cf3e (0x419 bytes) 000466 1786734298.439556 fallocate 0x007d75 thru 0x014815 (0xcaa1 bytes) 000467 1786734298.445794 fallocate 0x0348f3 thru 0x03aacc (0x61da bytes) 000468 1786734298.453111 read 0x01e7dc thru 0x0218a9 (0x30ce bytes) 000469 1786734298.472040 punch from 0x01556a to 0x01f621 (0xa0b8 bytes) 000470 1786734298.477737 trunc from 0x03aacd to 0x036551 (0x457b bytes) 000471 1786734298.491506 trunc from 0x00470a to 0x012981 (0xe278 bytes) 000472 1786734298.497641 read 0x029774 thru 0x036551 (0xcdde bytes) 000473 1786734298.499068 fallocate 0x002af2 thru 0x00804f (0x555e bytes) 000474 1786734298.504917 read 0x0194e0 thru 0x01e0a3 (0x4bc4 bytes) 000475 1786734298.524678 write 0x034d26 thru 0x03a643 (0x591e bytes) 000476 1786734298.526987 trunc from 0x0339ef to 0x037d0a (0x431c bytes) 000477 1786734298.541044 read 0x00e62d thru 0x014598 (0x5f6c bytes) 000478 1786734298.542196 trunc from 0x00d35b to 0x010b16 (0x37bc bytes) 000479 1786734298.547660 mapread 0x01895d thru 0x02778d (0xee31 bytes) 000480 1786734298.549626 punch from 0x034440 to 0x0389bc (0x457d bytes) 000481 1786734298.555958 write 0x0358e6 thru 0x03a147 (0x4862 bytes) 000483 1786734298.567906 fallocate 0x01ebf4 thru 0x02d104 (0xe511 bytes) 000484 1786734298.573753 trunc from 0x03a644 to 0x024034 (0x1660f bytes) 000485 1786734298.595862 mapread 0x0134a5 thru 0x018ced (0x5849 bytes) 000486 1786734298.598065 trunc from 0x012766 to 0x016d8a (0x4625 bytes) 000487 1786734298.607629 trunc from 0x024035 to 0x0009c0 (0x23674 bytes) 000488 1786734298.629696 read 0x00005f thru 0x0009c0 (0x962 bytes) 000489 1786734298.630331 trunc from 0x0007ac to 0x0009c0 (0x215 bytes) 000490 1786734298.635226 read 0x0007f3 thru 0x0009c0 (0x1ce bytes) 000491 1786734298.640049 read 0x000939 thru 0x0009c0 (0x88 bytes) 000492 1786734298.641078 write 0x02242b thru 0x02de73 (0xba49 bytes) 000494 1786734298.648197 read 0x01654c thru 0x0258f0 (0xf3a5 bytes) 000495 1786734298.655757 punch from 0x024843 to 0x0284d2 (0x3c90 bytes) 000496 1786734298.666204 read 0x008481 thru 0x0087f1 (0x371 bytes) 000498 1786734298.667873 punch from 0x01b632 to 0x026146 (0xab15 bytes) 000500 1786734298.672987 write 0x029baa thru 0x034395 (0xa7ec bytes) 000501 1786734298.674451 trunc from 0x034396 to 0x004527 (0x2fe6e bytes) 000502 1786734298.698771 write 0x03581b thru 0x037a7a (0x2260 bytes) 000503 1786734298.705247 read 0x0140e2 thru 0x018799 (0x46b8 bytes) 000504 1786734298.719454 trunc from 0x037a7b to 0x03b06c (0x35f2 bytes) 000505 1786734298.749208 read 0x01597a thru 0x023be2 (0xe269 bytes) 000506 1786734298.750942 trunc from 0x03b06d to 0x01d7ef (0x1d87d bytes) 000507 1786734298.772553 trunc from 0x01d7f0 to 0x025908 (0x8119 bytes) 000509 1786734298.792316 mapread 0x012399 thru 0x01d56d (0xb1d5 bytes) 000511 1786734298.796271 mapread 0x013db6 thru 0x0155bf (0x180a bytes) 000512 1786734298.798659 punch from 0x00189c to 0x00adbe (0x9523 bytes) 000513 1786734298.804258 fallocate 0x02cbf7 thru 0x03303e (0x6448 bytes) 000514 1786734298.810789 write 0x002dc9 thru 0x00fc3e (0xce76 bytes) 000515 1786734298.823633 write 0x02d1ad thru 0x0375c0 (0xa414 bytes) 000516 1786734298.825871 fallocate 0x02a2d2 thru 0x0305cc (0x62fb bytes) 000517 1786734298.841014 write 0x021d5b thru 0x029f38 (0x81de bytes) 000518 1786734298.842651 read 0x0123c0 thru 0x01d663 (0xb2a4 bytes) 000519 1786734298.844365 trunc from 0x03482d to 0x0375c0 (0x2d94 bytes) 000520 1786734298.862253 write 0x008d7d thru 0x013a3d (0xacc1 bytes) 000521 1786734298.870243 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x375c1): Mirror 6: 0xe468beb6 (stale) Mirror 8: 0xe468beb6 (stale) Mirror 9: 0xe468beb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x375c1): Mirror 6: 0xe468beb6 (stale) Mirror 8: 0xe468beb6 (stale) Mirror 9: 0xe468beb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x375c1: skipped 000523 1786734299.302734 trunc from 0x0375c1 to 0x03d4f1 (0x5f31 bytes) 000525 1786734299.335804 mapwrite 0x03f569 thru 0x03ffff (0xa97 bytes) 000526 1786734299.425452 fallocate 0x0025ac thru 0x0071db (0x4c30 bytes) 000527 1786734299.434646 punch from 0x02b4ce to 0x02d5dc (0x210f bytes) 000529 1786734299.453131 write 0x03db21 thru 0x03ffff (0x24df bytes) 000530 1786734299.458167 write 0x020317 thru 0x02f0d8 (0xedc2 bytes) 000531 1786734299.507908 write 0x02a98e thru 0x03a27a (0xf8ed bytes) 000532 1786734299.529168 mapwrite 0x025a9b thru 0x02d97c (0x7ee2 bytes) 000534 1786734299.553235 trunc from 0x040000 to 0x02b501 (0x14afe bytes) 000535 1786734299.578434 trunc from 0x02b502 to 0x00eeb9 (0x1c648 bytes) 000536 1786734299.601454 trunc from 0x00eeba to 0x0013e2 (0xdad7 bytes) 000537 1786734299.641889 trunc from 0x0013e3 to 0x0286e3 (0x27301 bytes) 000538 1786734299.677669 write 0x02a8c3 thru 0x03857c (0xdcba bytes) 000539 1786734299.682422 trunc from 0x01336e to 0x01ba3d (0x86d0 bytes) 000540 1786734299.695864 MIRROR_RESYNC 4 mirrors 000541 1786734300.619647 write 0x01ef68 thru 0x024b8a (0x5c23 bytes) 000542 1786734300.649187 trunc from 0x03857d to 0x015549 (0x23033 bytes) 000543 1786734300.676652 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1554a): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1554a): Mirror 6: 0x8a1080bf (stale) Mirror 8: 0x8a1080bf (stale) Mirror 9: 0x8a1080bf (stale) CRC-32 checksum value for chunk [0, 0x1554a): Mirror 1: 0x8a1080bf Mirror 10: 0x8a1080bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1554a: skipped 000544 1786734301.768618 mapread 0x0122bd thru 0x015549 (0x328d bytes) 000545 1786734301.820211 mapwrite 0x03e17f thru 0x03ffff (0x1e81 bytes) 000546 1786734301.916142 punch from 0x01a41c to 0x022638 (0x821d bytes) 000547 1786734301.922745 trunc from 0x00f008 to 0x01b4cd (0xc4c6 bytes) 000548 1786734301.930738 punch from 0x0182be to 0x024f4f (0xcc92 bytes) 000549 1786734301.938400 fallocate 0x011ba7 thru 0x01fa89 (0xdee3 bytes) 000550 1786734301.948262 trunc from 0x040000 to 0x03abdb (0x5424 bytes) 000551 1786734301.973246 mapwrite 0x03de54 thru 0x03ffff (0x21ac bytes) 000553 1786734302.031992 trunc from 0x040000 to 0x023432 (0x1cbcd bytes) 000554 1786734302.059224 read 0x01c79a thru 0x023432 (0x6c99 bytes) 000555 1786734302.090004 mapread 0x011176 thru 0x0196ea (0x8575 bytes) 000556 1786734302.118111 read 0x002b55 thru 0x00e8c8 (0xbd74 bytes) 000557 1786734302.140348 read 0x00044c thru 0x00516d (0x4d22 bytes) 000558 1786734302.141752 trunc from 0x011f03 to 0x0216ed (0xf7eb bytes) 000559 1786734302.155896 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23433): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23433): Mirror 6: 0x21f1f929 (stale) Mirror 8: 0x21f1f929 (stale) Mirror 9: 0x21f1f929 (stale) Mirror 10: 0x21f1f929 (stale) CRC-32 checksum value for chunk [0, 0x23433): Mirror 1: 0x21f1f929 Mirror 11: 0x21f1f929 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23433: skipped 000560 1786734302.991215 write 0x01eafc thru 0x026f9d (0x84a2 bytes) 000561 1786734303.042398 trunc from 0x026f9e to 0x007745 (0x1f858 bytes) 000562 1786734303.095360 read 0x0064a4 thru 0x007745 (0x12a2 bytes) 000563 1786734303.102353 fallocate 0x002128 thru 0x011c89 (0xfb62 bytes) 000564 1786734303.113613 MIRROR_RESYNC 6 mirrors 000565 1786734304.700952 trunc from 0x007746 to 0x000131 (0x7614 bytes) 000566 1786734304.741614 trunc from 0x000132 to 0x035104 (0x34fd3 bytes) 000567 1786734304.771346 punch from 0x031941 to 0x03338e (0x1a4e bytes) 000568 1786734304.777816 punch from 0x030339 to 0x035104 (0x4dcc bytes) 000569 1786734304.783338 trunc from 0x035105 to 0x022df6 (0x1230e bytes) 000570 1786734304.800814 punch from 0x01e942 to 0x022df6 (0x44b5 bytes) 000571 1786734304.808095 write 0x02f998 thru 0x03b72b (0xbd94 bytes) 000572 1786734304.810364 trunc from 0x004401 to 0x00816f (0x3d6f bytes) 000573 1786734304.815107 write 0x0194bb thru 0x0284a9 (0xefef bytes) 000574 1786734304.827103 mapread 0x00b74e thru 0x012ddd (0x7690 bytes) 000575 1786734304.846592 fallocate 0x01988f thru 0x019a11 (0x183 bytes) 000576 1786734304.853902 fallocate 0x01766f thru 0x025a94 (0xe426 bytes) 000577 1786734304.860807 mapwrite 0x027908 thru 0x02f239 (0x7932 bytes) 000578 1786734304.922343 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b72c): Mirror 6: 0xa38b8bb8 (stale) Mirror 8: 0xa38b8bb8 (stale) Mirror 9: 0xa38b8bb8 (stale) Mirror 10: 0xa38b8bb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b72c): Mirror 6: 0xa38b8bb8 (stale) Mirror 8: 0xa38b8bb8 (stale) Mirror 9: 0xa38b8bb8 (stale) Mirror 10: 0xa38b8bb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b72c: skipped 000579 1786734305.499337 mapwrite 0x034a9c thru 0x03f1fc (0xa761 bytes) 000580 1786734305.581634 trunc from 0x03f1fd to 0x02202c (0x1d1d0 bytes) 000581 1786734305.611262 MIRROR_RESYNC 5 mirrors 000583 1786734306.849423 fallocate 0x014633 thru 0x014d1a (0x6e8 bytes) 000584 1786734306.905752 trunc from 0x02202d to 0x026870 (0x4844 bytes) 000585 1786734306.936578 read 0x000b6c thru 0x003577 (0x2a0c bytes) 000587 1786734306.955014 mapwrite 0x03cc03 thru 0x03ffff (0x33fd bytes) 000589 1786734307.063541 mapwrite 0x034fe7 thru 0x035fe9 (0x1003 bytes) 000590 1786734307.094520 mapwrite 0x003362 thru 0x009cb2 (0x6951 bytes) 000591 1786734307.164697 mapwrite 0x01e07a thru 0x020b93 (0x2b1a bytes) 000592 1786734307.206364 mapwrite 0x0300fc thru 0x03a805 (0xa70a bytes) 000593 1786734307.234335 punch from 0x00f3b1 to 0x015666 (0x62b6 bytes) 000594 1786734307.240366 trunc from 0x040000 to 0x0111fa (0x2ee05 bytes) 000595 1786734307.260607 mapwrite 0x01be04 thru 0x027643 (0xb840 bytes) 000596 1786734307.326433 punch from 0x007be2 to 0x00fbd7 (0x7ff6 bytes) 000597 1786734307.333131 mapwrite 0x02ef33 thru 0x0335a0 (0x466e bytes) 000599 1786734307.382284 read 0x00d533 thru 0x01148a (0x3f58 bytes) 000600 1786734307.394126 mapread 0x01e7c7 thru 0x02c538 (0xdd72 bytes) 000601 1786734307.396630 trunc from 0x0335a1 to 0x008d9b (0x2a805 bytes) 000602 1786734307.411307 mapwrite 0x00cb01 thru 0x012f71 (0x6471 bytes) 000603 1786734307.483908 punch from 0x006597 to 0x0069e6 (0x450 bytes) 000604 1786734307.491317 mapread 0x0072ea thru 0x00ab4e (0x3865 bytes) 000605 1786734307.502603 trunc from 0x012a21 to 0x012f71 (0x551 bytes) 000607 1786734307.509142 read 0x0011f3 thru 0x00d140 (0xbf4e bytes) 000610 1786734307.530464 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12f72): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f72): Mirror 6: 0xed49af69 (stale) Mirror 8: 0xed49af69 (stale) Mirror 9: 0xed49af69 (stale) Mirror 10: 0xed49af69 (stale) CRC-32 checksum value for chunk [0, 0x12f72): Mirror 1: 0x56ef7460 Mirror 11: 0x56ef7460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f72: skipped 000611 1786734308.389499 trunc from 0x009908 to 0x009a16 (0x10f bytes) 000612 1786734308.444380 write 0x019500 thru 0x01f6f2 (0x61f3 bytes) 000613 1786734308.465409 read 0x01f023 thru 0x01f6f2 (0x6d0 bytes) 000614 1786734308.469171 mapwrite 0x0211fd thru 0x02e513 (0xd317 bytes) 000615 1786734308.611890 fallocate 0x03d19c thru 0x03ffff (0x2e64 bytes) 000616 1786734308.620452 read 0x01b43f thru 0x01f6d4 (0x4296 bytes) 000617 1786734308.626342 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe2ba6049 (stale) Mirror 8: 0xe2ba6049 (stale) Mirror 9: 0xe2ba6049 (stale) Mirror 10: 0xe2ba6049 (stale) Mirror 11: 0xfa350fad (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xffb03aaf Mirror 12: 0xffb03aaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000618 1786734309.871097 read 0x01874c thru 0x025c2b (0xd4e0 bytes) 000620 1786734309.909696 MIRROR_RESYNC 7 mirrors 000621 1786734312.326652 write 0x01def5 thru 0x0240d4 (0x61e0 bytes) 000622 1786734312.381824 trunc from 0x02ec75 to 0x039efb (0xb287 bytes) 000623 1786734312.393308 MIRROR_RESYNC 7 mirrors 000624 1786734314.614616 write 0x002015 thru 0x0050d4 (0x30c0 bytes) 000625 1786734314.658954 mapread 0x029385 thru 0x03129d (0x7f19 bytes) 000626 1786734314.684890 punch from 0x01571f to 0x01ed65 (0x9647 bytes) 000628 1786734314.698082 read 0x039116 thru 0x03ffff (0x6eea bytes) 000629 1786734314.711916 mapread 0x002df6 thru 0x011372 (0xe57d bytes) 000630 1786734314.736035 trunc from 0x040000 to 0x01b722 (0x248dd bytes) 000631 1786734314.765839 write 0x032a05 thru 0x03cc19 (0xa215 bytes) 000632 1786734314.770184 mapread 0x01fe22 thru 0x02f051 (0xf230 bytes) 000633 1786734314.791368 write 0x011b8c thru 0x020a81 (0xeef6 bytes) 000634 1786734314.794906 fallocate 0x031c4a thru 0x03e283 (0xc63a bytes) 000635 1786734314.802579 trunc from 0x00690e to 0x00eeb2 (0x85a5 bytes) 000636 1786734314.809348 mapread 0x01b089 thru 0x01c8e3 (0x185b bytes) 000637 1786734314.816963 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 6: 0x914e1a3f (stale) Mirror 8: 0x914e1a3f (stale) Mirror 9: 0x914e1a3f (stale) Mirror 10: 0x914e1a3f (stale) Mirror 11: 0x914e1a3f (stale) Mirror 12: 0x914e1a3f (stale) CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 1: 0xd3f14ee9 Mirror 13: 0xd3f14ee9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc1a: skipped 000639 1786734316.301592 punch from 0x02e518 to 0x0398c7 (0xb3b0 bytes) 000641 1786734316.376557 trunc from 0x03cc1a to 0x032663 (0xa5b6 bytes) 000642 1786734316.420340 trunc from 0x02b742 to 0x032663 (0x6f22 bytes) 000643 1786734316.430580 punch from 0x004d42 to 0x0121f2 (0xd4b1 bytes) 000644 1786734316.440004 mapread 0x001a83 thru 0x00441e (0x299c bytes) 000645 1786734316.463048 read 0x00f94a thru 0x017d83 (0x843a bytes) 000646 1786734316.472235 mapread 0x01f2b7 thru 0x0225ee (0x3338 bytes) 000647 1786734316.485457 MIRROR_RESYNC 8 mirrors 000648 1786734318.718580 mapwrite 0x000e45 thru 0x000f0f (0xcb bytes) 000649 1786734318.832408 read 0x00eb98 thru 0x00f0d3 (0x53c bytes) 000651 1786734318.841176 punch from 0x00e58d to 0x01c210 (0xdc84 bytes) 000652 1786734318.848056 trunc from 0x0163f5 to 0x0189a0 (0x25ac bytes) 000653 1786734318.854419 mapwrite 0x023ee8 thru 0x02cfd0 (0x90e9 bytes) 000654 1786734318.920611 punch from 0x0038fa to 0x00c6f6 (0x8dfd bytes) 000656 1786734318.929847 punch from 0x019739 to 0x01f61c (0x5ee4 bytes) 000657 1786734318.941452 punch from 0x0263fa to 0x030c7e (0xa885 bytes) 000658 1786734318.952384 mapread 0x01ef23 thru 0x028a95 (0x9b73 bytes) 000659 1786734318.972897 trunc from 0x032664 to 0x01a202 (0x18461 bytes) 000660 1786734318.995172 punch from 0x010d12 to 0x01a202 (0x94f1 bytes) 000661 1786734319.001062 punch from 0x0061d9 to 0x012ed4 (0xccfc bytes) 000662 1786734319.006807 trunc from 0x01a203 to 0x0364b2 (0x1c2b0 bytes) 000663 1786734319.027178 write 0x0311c7 thru 0x03b9a4 (0xa7de bytes) 000664 1786734319.041189 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: 0xd1c7cdb3 (stale) Mirror 8: 0xd1c7cdb3 (stale) Mirror 9: 0xd1c7cdb3 (stale) Mirror 10: 0xd1c7cdb3 (stale) Mirror 11: 0xd1c7cdb3 (stale) Mirror 12: 0xd1c7cdb3 (stale) Mirror 13: 0xd1c7cdb3 (stale) CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 1: 0xf017a01b Mirror 14: 0xf017a01b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9a5: skipped 000665 1786734320.339596 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: 0xd1c7cdb3 (stale) Mirror 8: 0xd1c7cdb3 (stale) Mirror 9: 0xd1c7cdb3 (stale) Mirror 10: 0xd1c7cdb3 (stale) Mirror 11: 0xd1c7cdb3 (stale) Mirror 12: 0xd1c7cdb3 (stale) Mirror 13: 0xd1c7cdb3 (stale) CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 1: 0xf017a01b Mirror 14: 0xf017a01b Mirror 15: 0xf017a01b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9a5: skipped 000666 1786734321.379307 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: 0xd1c7cdb3 (stale) Mirror 9: 0xd1c7cdb3 (stale) Mirror 10: 0xd1c7cdb3 (stale) Mirror 11: 0xd1c7cdb3 (stale) Mirror 12: 0xd1c7cdb3 (stale) Mirror 13: 0xd1c7cdb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 6: 0xd1c7cdb3 (stale) Mirror 9: 0xd1c7cdb3 (stale) Mirror 10: 0xd1c7cdb3 (stale) Mirror 11: 0xd1c7cdb3 (stale) Mirror 12: 0xd1c7cdb3 (stale) Mirror 13: 0xd1c7cdb3 (stale) CRC-32 checksum value for chunk [0, 0x3b9a5): Mirror 1: 0xf017a01b Mirror 14: 0xf017a01b Mirror 15: 0xf017a01b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9a5: skipped 000667 1786734322.132264 mapwrite 0x026c0c thru 0x0332eb (0xc6e0 bytes) 000668 1786734322.298032 read 0x03450d thru 0x03b1a6 (0x6c9a bytes) 000670 1786734322.322261 fallocate 0x00f6fa thru 0x01e5ce (0xeed5 bytes) 000671 1786734322.330685 mapwrite 0x02d5c6 thru 0x035e26 (0x8861 bytes) 000672 1786734322.349590 punch from 0x021f37 to 0x02d15f (0xb229 bytes) 000673 1786734322.355677 write 0x028b93 thru 0x02f0b4 (0x6522 bytes) 000675 1786734322.369184 mapwrite 0x0266b2 thru 0x02d1e8 (0x6b37 bytes) 000676 1786734322.405764 trunc from 0x0321a2 to 0x03b9a4 (0x9803 bytes) 000677 1786734322.413340 read 0x02fa84 thru 0x033e95 (0x4412 bytes) 000678 1786734322.424302 punch from 0x02624b to 0x0304dc (0xa292 bytes) 000679 1786734322.430459 trunc from 0x03b9a5 to 0x034ecb (0x6ad9 bytes) 000680 1786734322.450108 trunc from 0x034ecc to 0x009b06 (0x2b3c5 bytes) 000681 1786734322.474520 punch from 0x0018b7 to 0x009b06 (0x8250 bytes) 000682 1786734322.514849 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9b07): Mirror 6: 0x98181725 (stale) Mirror 10: 0x98181725 (stale) Mirror 11: 0x98181725 (stale) Mirror 12: 0x98181725 (stale) Mirror 13: 0x98181725 (stale) Mirror 14: 0x49666bc3 (stale) Mirror 15: 0x49666bc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9b07): Mirror 6: 0x98181725 (stale) Mirror 10: 0x98181725 (stale) Mirror 11: 0x98181725 (stale) Mirror 12: 0x98181725 (stale) Mirror 13: 0x98181725 (stale) Mirror 14: 0x49666bc3 (stale) Mirror 15: 0x49666bc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9b07: skipped 000683 1786734323.123645 fallocate 0x0274b1 thru 0x02f054 (0x7ba4 bytes) 000684 1786734323.130417 fallocate 0x012cc0 thru 0x01de66 (0xb1a7 bytes) 000685 1786734323.137196 mapread 0x01e837 thru 0x025a5d (0x7227 bytes) 000686 1786734323.159422 read 0x0243c0 thru 0x02ad3d (0x697e bytes) 000687 1786734323.165668 mapwrite 0x034bb1 thru 0x0392b4 (0x4704 bytes) 000688 1786734323.218410 punch from 0x03272c to 0x0346cb (0x1fa0 bytes) 000689 1786734323.225664 fallocate 0x00b940 thru 0x013875 (0x7f36 bytes) 000690 1786734323.236434 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x392b5): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x392b5): Mirror 6: 0x12b01b9c (stale) Mirror 10: 0x12b01b9c (stale) Mirror 11: 0x12b01b9c (stale) Mirror 12: 0x12b01b9c (stale) Mirror 13: 0x12b01b9c (stale) Mirror 14: 0xeecac83c (stale) Mirror 15: 0xeecac83c (stale) CRC-32 checksum value for chunk [0, 0x392b5): Mirror 1: 0xcb7a9cca Mirror 16: 0xcb7a9cca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x392b5: skipped 000691 1786734324.333564 trunc from 0x0392b5 to 0x00eede (0x2a3d6 bytes) 000692 1786734324.483421 mapwrite 0x01505f thru 0x019ea0 (0x4e42 bytes) 000693 1786734324.552501 mapwrite 0x01372e thru 0x014117 (0x9ea bytes) 000694 1786734324.573920 mapread 0x012d79 thru 0x0131fa (0x482 bytes) 000696 1786734324.584315 mapread 0x0155d8 thru 0x019ea0 (0x48c9 bytes) 000697 1786734324.588064 fallocate 0x01ef87 thru 0x027032 (0x80ac bytes) 000698 1786734324.596978 trunc from 0x019ea1 to 0x036305 (0x1c465 bytes) 000699 1786734324.623913 trunc from 0x036306 to 0x005ae4 (0x30821 bytes) 000700 1786734324.644775 fallocate 0x024479 thru 0x02b2d3 (0x6e5b bytes) 000701 1786734324.650994 fallocate 0x006a72 thru 0x00e4ef (0x7a7e bytes) 000702 1786734324.656443 fallocate 0x023696 thru 0x02e6bc (0xb027 bytes) 000703 1786734324.669606 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2e6bd): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6bd): Mirror 6: 0xb00a50a4 (stale) Mirror 10: 0xb00a50a4 (stale) Mirror 11: 0xb00a50a4 (stale) Mirror 12: 0xb00a50a4 (stale) Mirror 13: 0xb00a50a4 (stale) Mirror 14: 0x4aa94296 (stale) Mirror 15: 0x4aa94296 (stale) Mirror 16: 0x37f751f8 (stale) CRC-32 checksum value for chunk [0, 0x2e6bd): Mirror 1: 0x37f751f8 Mirror 17: 0x37f751f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6bd: skipped 000704 1786734326.397354 trunc from 0x02e6bd to 0x00a967 (0x23d55 bytes) 000705 1786734326.482658 fallocate 0x00b999 thru 0x0199ed (0xe055 bytes) 000706 1786734326.489032 trunc from 0x010147 to 0x0199ed (0x98a7 bytes) 000707 1786734326.500273 write 0x00416f thru 0x00a9da (0x686c bytes) 000708 1786734326.521522 mapwrite 0x032d85 thru 0x03cc2d (0x9ea9 bytes) 000709 1786734326.629323 trunc from 0x0106c2 to 0x01fde6 (0xf725 bytes) 000710 1786734326.634864 write 0x01bacc thru 0x02715b (0xb690 bytes) 000711 1786734326.675516 fallocate 0x035ca3 thru 0x03720e (0x156c bytes) 000713 1786734326.684938 mapread 0x028c30 thru 0x0307f0 (0x7bc1 bytes) 000714 1786734326.727078 read 0x008235 thru 0x00de9a (0x5c66 bytes) 000715 1786734326.745076 mapwrite 0x035029 thru 0x0388b9 (0x3891 bytes) 000716 1786734326.782601 punch from 0x036490 to 0x03cc2d (0x679e bytes) 000717 1786734326.793254 punch from 0x012a12 to 0x01d661 (0xac50 bytes) 000719 1786734326.810019 fallocate 0x008fc6 thru 0x012751 (0x978c bytes) 000720 1786734326.822583 mapread 0x018fe3 thru 0x026c98 (0xdcb6 bytes) 000721 1786734326.845413 trunc from 0x0308a2 to 0x0393e5 (0x8b44 bytes) 000722 1786734326.859747 read 0x024c97 thru 0x02dd34 (0x909e bytes) 000723 1786734326.861425 punch from 0x0136d0 to 0x0234bc (0xfded bytes) 000724 1786734326.882852 trunc from 0x03cc2e to 0x00ac38 (0x31ff5 bytes) 000725 1786734326.907786 mapread 0x0091cb thru 0x00ac38 (0x1a6e bytes) 000726 1786734326.913164 punch from 0x0052d9 to 0x00ac38 (0x5960 bytes) 000728 1786734326.925621 trunc from 0x00ac39 to 0x03970a (0x2ead2 bytes) 000729 1786734326.961405 read 0x01c901 thru 0x028bd5 (0xc2d5 bytes) 000730 1786734326.987678 trunc from 0x01ac9f to 0x01ed8a (0x40ec bytes) 000731 1786734326.997171 write 0x028895 thru 0x030005 (0x7771 bytes) 000732 1786734327.001743 trunc from 0x019144 to 0x028948 (0xf805 bytes) 000733 1786734327.029862 punch from 0x001d6a to 0x0096aa (0x7941 bytes) 000734 1786734327.042580 fallocate 0x03e83c thru 0x03ffff (0x17c4 bytes) 000735 1786734327.054313 mapwrite 0x00608a thru 0x0098bd (0x3834 bytes) 000736 1786734327.117682 MIRROR_RESYNC 10 mirrors 000737 1786734329.358049 punch from 0x03100c to 0x037c57 (0x6c4c bytes) 000738 1786734329.381222 write 0x00b54f thru 0x0108ae (0x5360 bytes) 000739 1786734329.394569 trunc from 0x03970b to 0x009a4e (0x2fcbc bytes) 000740 1786734329.413586 read 0x007aea thru 0x009a4e (0x1f65 bytes) 000741 1786734329.421568 trunc from 0x007c16 to 0x009a4e (0x1e39 bytes) 000742 1786734329.426793 mapwrite 0x037f53 thru 0x03f28e (0x733c bytes) 000743 1786734329.480391 write 0x020c26 thru 0x02281a (0x1bf5 bytes) 000744 1786734329.491097 punch from 0x01e119 to 0x0248d7 (0x67bf bytes) 000745 1786734329.503253 trunc from 0x03602c to 0x03f28e (0x9263 bytes) 000746 1786734329.509842 fallocate 0x003ebc thru 0x00536b (0x14b0 bytes) 000747 1786734329.515598 trunc from 0x019e66 to 0x01d139 (0x32d4 bytes) 000748 1786734329.521432 read 0x013ab5 thru 0x01a486 (0x69d2 bytes) 000749 1786734329.533078 trunc from 0x03f28f to 0x03e306 (0xf88 bytes) 000750 1786734329.547626 mapread 0x01f789 thru 0x0261ab (0x6a23 bytes) 000751 1786734329.549727 trunc from 0x03e307 to 0x02610a (0x181fc bytes) 000753 1786734329.570307 write 0x034858 thru 0x03de77 (0x9620 bytes) 000755 1786734329.572998 write 0x013870 thru 0x01d3c2 (0x9b53 bytes) 000756 1786734329.575176 fallocate 0x00e6b4 thru 0x0145c1 (0x5f0e bytes) 000757 1786734329.581599 write 0x031447 thru 0x031c71 (0x82b bytes) 000758 1786734329.586971 fallocate 0x02374e thru 0x02c0a0 (0x8953 bytes) 000759 1786734329.609790 MIRROR_RESYNC 10 mirrors 000760 1786734332.906319 read 0x01efa5 thru 0x02e03a (0xf096 bytes) 000761 1786734332.916213 fallocate 0x0344e0 thru 0x03ffff (0xbb20 bytes) 000762 1786734332.955499 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc82eae0d (stale) Mirror 10: 0xc82eae0d (stale) Mirror 11: 0xc82eae0d (stale) Mirror 12: 0xc82eae0d (stale) Mirror 13: 0xc82eae0d (stale) Mirror 14: 0xc82eae0d (stale) Mirror 16: 0xc82eae0d (stale) Mirror 17: 0xc82eae0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc82eae0d (stale) Mirror 10: 0xc82eae0d (stale) Mirror 11: 0xc82eae0d (stale) Mirror 12: 0xc82eae0d (stale) Mirror 13: 0xc82eae0d (stale) Mirror 14: 0xc82eae0d (stale) Mirror 16: 0xc82eae0d (stale) Mirror 17: 0xc82eae0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1786734333.532295 trunc from 0x040000 to 0x034bcb (0xb434 bytes) 000764 1786734333.555675 mapread 0x00d2b1 thru 0x01919a (0xbeea bytes) 000765 1786734333.597182 write 0x02fcf1 thru 0x039522 (0x9832 bytes) 000766 1786734333.603298 read 0x009de3 thru 0x0137d7 (0x99f5 bytes) 000767 1786734333.604271 trunc from 0x039523 to 0x0218e1 (0x17c41 bytes) 000768 1786734333.633229 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x218e2): Mirror 10: 0xa79999d4 (stale) Mirror 11: 0xa79999d4 (stale) Mirror 12: 0xa79999d4 (stale) Mirror 13: 0xa79999d4 (stale) Mirror 14: 0xa79999d4 (stale) Mirror 16: 0xa79999d4 (stale) Mirror 17: 0xa79999d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x218e2): Mirror 10: 0xa79999d4 (stale) Mirror 11: 0xa79999d4 (stale) Mirror 12: 0xa79999d4 (stale) Mirror 13: 0xa79999d4 (stale) Mirror 14: 0xa79999d4 (stale) Mirror 16: 0xa79999d4 (stale) Mirror 17: 0xa79999d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x218e2: skipped 000769 1786734334.441336 write 0x0034c8 thru 0x00f25c (0xbd95 bytes) 000770 1786734334.463191 punch from 0x0206b6 to 0x0218e1 (0x122c bytes) 000771 1786734334.476134 MIRROR_RESYNC 8 mirrors 000772 1786734336.725980 read 0x00e688 thru 0x01d3bc (0xed35 bytes) 000773 1786734336.743348 mapread 0x0142f5 thru 0x01959e (0x52aa bytes) 000774 1786734336.747405 read 0x0019da thru 0x0088c7 (0x6eee bytes) 000775 1786734336.776688 mapread 0x012428 thru 0x01b872 (0x944b bytes) 000776 1786734336.779697 trunc from 0x009144 to 0x00b856 (0x2713 bytes) 000777 1786734336.821675 trunc from 0x0218e2 to 0x036112 (0x14831 bytes) 000778 1786734336.841544 trunc from 0x036113 to 0x02f5a3 (0x6b6f bytes) 000779 1786734336.856154 punch from 0x014124 to 0x016c33 (0x2b10 bytes) 000780 1786734336.875022 mapwrite 0x03cc17 thru 0x03ffff (0x33e9 bytes) 000781 1786734336.937285 trunc from 0x03e842 to 0x03fcd9 (0x1498 bytes) 000782 1786734336.950578 mapread 0x02551e thru 0x027bde (0x26c1 bytes) 000783 1786734336.951644 trunc from 0x040000 to 0x0157e6 (0x2a819 bytes) 000785 1786734336.968466 fallocate 0x0208ab thru 0x022a14 (0x216a bytes) 000786 1786734336.976548 mapwrite 0x017898 thru 0x017c19 (0x382 bytes) 000787 1786734336.998421 write 0x03d342 thru 0x03ffff (0x2cbe bytes) 000789 1786734337.001230 write 0x03b9da thru 0x03e9cb (0x2ff2 bytes) 000790 1786734337.007022 mapread 0x032382 thru 0x03345f (0x10de bytes) 000791 1786734337.016640 punch from 0x007a62 to 0x007b3b (0xda bytes) 000792 1786734337.021423 trunc from 0x040000 to 0x00660d (0x399f2 bytes) 000793 1786734337.038107 read 0x005fd7 thru 0x00660d (0x637 bytes) 000795 1786734337.051040 trunc from 0x004d5a to 0x00660d (0x18b4 bytes) 000796 1786734337.056989 read 0x006516 thru 0x00660d (0xf8 bytes) 000797 1786734337.063265 read 0x001afd thru 0x006111 (0x4615 bytes) 000798 1786734337.063880 write 0x0399d7 thru 0x03ffff (0x6629 bytes) 000799 1786734337.069576 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5bdd8a23 (stale) Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 13: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x707752f9 Mirror 18: 0x707752f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1786734338.095650 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5bdd8a23 (stale) Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5bdd8a23 (stale) Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x707752f9 Mirror 18: 0x707752f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1786734339.142373 mapwrite 0x012dfd thru 0x017f17 (0x511b bytes) 000802 1786734339.251347 read 0x00e401 thru 0x011287 (0x2e87 bytes) 000803 1786734339.252586 write 0x01cd4f thru 0x023c1d (0x6ecf bytes) 000804 1786734339.269516 write 0x03812e thru 0x03ffff (0x7ed2 bytes) 000805 1786734339.276435 mapread 0x026498 thru 0x032788 (0xc2f1 bytes) 000806 1786734339.295851 write 0x00c627 thru 0x01515b (0x8b35 bytes) 000807 1786734339.298676 fallocate 0x001c31 thru 0x0118cc (0xfc9c bytes) 000809 1786734339.305619 fallocate 0x006006 thru 0x01264d (0xc648 bytes) 000810 1786734339.311875 fallocate 0x037adf thru 0x03e08b (0x65ad bytes) 000811 1786734339.317377 read 0x0221fa thru 0x030cfb (0xeb02 bytes) 000812 1786734339.319273 mapread 0x0126aa thru 0x022483 (0xfdda bytes) 000813 1786734339.326042 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) Mirror 18: 0x707752f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) Mirror 18: 0x707752f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1786734339.804520 mapread 0x002db4 thru 0x00cadf (0x9d2c bytes) 000816 1786734339.834337 mapread 0x001f67 thru 0x00b1c9 (0x9263 bytes) 000817 1786734339.836371 mapwrite 0x002074 thru 0x0037f5 (0x1782 bytes) 000819 1786734339.850664 punch from 0x0310f4 to 0x037361 (0x626e bytes) 000820 1786734339.859370 fallocate 0x01312c thru 0x013b82 (0xa57 bytes) 000821 1786734339.872840 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5bdd8a23 (stale) Mirror 12: 0x5bdd8a23 (stale) Mirror 14: 0x5bdd8a23 (stale) Mirror 16: 0x5bdd8a23 (stale) Mirror 17: 0x5bdd8a23 (stale) Mirror 18: 0x707752f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdebfe908 Mirror 19: 0xdebfe908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000822 1786734341.376125 trunc from 0x040000 to 0x025cdc (0x1a323 bytes) 000823 1786734341.458219 write 0x0033ad thru 0x008937 (0x558b bytes) 000824 1786734341.479414 mapread 0x0239ba thru 0x02422d (0x874 bytes) 000825 1786734341.493424 read 0x0091ff thru 0x018040 (0xee42 bytes) 000827 1786734341.516803 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25cdd): Mirror 11: 0xf4d4d297 (stale) Mirror 12: 0xf4d4d297 (stale) Mirror 14: 0xf4d4d297 (stale) Mirror 17: 0xf4d4d297 (stale) Mirror 18: 0xe901e22d (stale) Mirror 19: 0x46c100c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cdd): Mirror 11: 0xf4d4d297 (stale) Mirror 12: 0xf4d4d297 (stale) Mirror 14: 0xf4d4d297 (stale) Mirror 17: 0xf4d4d297 (stale) Mirror 18: 0xe901e22d (stale) Mirror 19: 0x46c100c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cdd: skipped 000829 1786734341.954647 trunc from 0x003e98 to 0x008371 (0x44da bytes) 000830 1786734341.962220 mapread 0x024f78 thru 0x025cdc (0xd65 bytes) 000831 1786734341.986025 MIRROR_RESYNC 7 mirrors 000832 1786734344.100318 trunc from 0x01130a to 0x011ac2 (0x7b9 bytes) 000833 1786734344.143379 punch from 0x00bb8e to 0x015141 (0x95b4 bytes) 000834 1786734344.149419 trunc from 0x013e3d to 0x021085 (0xd249 bytes) 000835 1786734344.154688 trunc from 0x025cdd to 0x01f36d (0x696f bytes) 000836 1786734344.179192 punch from 0x01d4af to 0x01f36d (0x1ebf bytes) 000837 1786734344.184598 write 0x00d060 thru 0x00ea07 (0x19a8 bytes) 000839 1786734344.197395 mapwrite 0x02b2d3 thru 0x03424d (0x8f7b bytes) 000840 1786734344.270819 trunc from 0x02ade3 to 0x030521 (0x573f bytes) 000841 1786734344.280009 write 0x01b526 thru 0x024588 (0x9063 bytes) 000842 1786734344.296971 fallocate 0x02f68f thru 0x0353e5 (0x5d57 bytes) 000843 1786734344.307824 trunc from 0x03096e to 0x032c2d (0x22c0 bytes) 000844 1786734344.323271 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3424e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3424e): Mirror 11: 0xed9d9c23 (stale) Mirror 12: 0xed9d9c23 (stale) Mirror 14: 0xed9d9c23 (stale) Mirror 17: 0xed9d9c23 (stale) Mirror 18: 0xed9d9c23 (stale) Mirror 19: 0xed9d9c23 (stale) CRC-32 checksum value for chunk [0, 0x3424e): Mirror 1: 0xbc9162aa Mirror 20: 0xbc9162aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3424e: skipped 000845 1786734345.470435 punch from 0x016196 to 0x01b147 (0x4fb2 bytes) 000846 1786734345.574264 mapread 0x01ef80 thru 0x026746 (0x77c7 bytes) 000847 1786734345.632655 mapread 0x0000fa thru 0x006cdb (0x6be2 bytes) 000849 1786734345.647137 mapwrite 0x03ada3 thru 0x03fe0a (0x5068 bytes) 000850 1786734345.733630 MIRROR_RESYNC 8 mirrors 000851 1786734349.613592 mapread 0x00519e thru 0x00ceae (0x7d11 bytes) 000852 1786734349.653352 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000853 1786734350.044447 read 0x02cdef thru 0x034af3 (0x7d05 bytes) 000855 1786734350.055352 mapwrite 0x01485c thru 0x015d64 (0x1509 bytes) 000856 1786734350.151408 write 0x0040c1 thru 0x006771 (0x26b1 bytes) 000857 1786734350.171056 mapread 0x013c07 thru 0x01643c (0x2836 bytes) 000858 1786734350.179183 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fe0b): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe0b): Mirror 12: 0x3fe02503 (stale) Mirror 14: 0x3fe02503 (stale) Mirror 17: 0x3fe02503 (stale) Mirror 18: 0x3fe02503 (stale) Mirror 19: 0x3fe02503 (stale) Mirror 20: 0x3fe02503 (stale) CRC-32 checksum value for chunk [0, 0x3fe0b): Mirror 1: 0x4978d167 Mirror 21: 0x4978d167 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe0b: skipped 000859 1786734351.821308 punch from 0x034962 to 0x03544b (0xaea bytes) 000860 1786734351.875934 fallocate 0x00186d thru 0x009596 (0x7d2a bytes) 000861 1786734351.884241 trunc from 0x01d1a2 to 0x0289c9 (0xb828 bytes) 000862 1786734351.890989 fallocate 0x00b16f thru 0x01a35a (0xf1ec bytes) 000863 1786734351.899228 trunc from 0x0227ae to 0x02365b (0xeae bytes) 000864 1786734351.906407 write 0x00166d thru 0x00950b (0x7e9f bytes) 000865 1786734351.923686 read 0x00df66 thru 0x0118f9 (0x3994 bytes) 000866 1786734351.935680 read 0x039046 thru 0x03fe0a (0x6dc5 bytes) 000867 1786734351.946075 trunc from 0x03fe0b to 0x004d92 (0x3b078 bytes) 000868 1786734351.997653 trunc from 0x004d93 to 0x00055f (0x4833 bytes) 000869 1786734352.025466 trunc from 0x000560 to 0x0060aa (0x5b4b bytes) 000870 1786734352.037699 fallocate 0x00619b thru 0x00d70f (0x7575 bytes) 000871 1786734352.044539 fallocate 0x027e0d thru 0x028204 (0x3f8 bytes) 000872 1786734352.049909 read 0x0222fa thru 0x028204 (0x5f0b bytes) 000873 1786734352.060241 trunc from 0x0066d2 to 0x00f776 (0x90a5 bytes) 000874 1786734352.070442 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28205): Mirror 14: 0xd28cacbc (stale) Mirror 17: 0xd28cacbc (stale) Mirror 18: 0xd28cacbc (stale) Mirror 19: 0xd28cacbc (stale) Mirror 20: 0xd28cacbc (stale) Mirror 21: 0x90e1fe6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28205): Mirror 14: 0xd28cacbc (stale) Mirror 17: 0xd28cacbc (stale) Mirror 18: 0xd28cacbc (stale) Mirror 19: 0xd28cacbc (stale) Mirror 20: 0xd28cacbc (stale) Mirror 21: 0x90e1fe6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28205: skipped 000875 1786734352.503118 mapwrite 0x012518 thru 0x0144c8 (0x1fb1 bytes) 000876 1786734352.537833 mapread 0x021e1f thru 0x024f72 (0x3154 bytes) 000877 1786734352.551285 trunc from 0x028205 to 0x01df49 (0xa2bb bytes) 000878 1786734352.570232 fallocate 0x00e6c1 thru 0x017eac (0x97ec bytes) 000879 1786734352.579454 write 0x000458 thru 0x00a73f (0xa2e8 bytes) 000880 1786734352.624115 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1df4a): Mirror 14: 0x6593a32d (stale) Mirror 17: 0x6593a32d (stale) Mirror 18: 0x6593a32d (stale) Mirror 19: 0x6593a32d (stale) Mirror 20: 0x6593a32d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df4a): Mirror 14: 0x6593a32d (stale) Mirror 17: 0x6593a32d (stale) Mirror 18: 0x6593a32d (stale) Mirror 19: 0x6593a32d (stale) Mirror 20: 0x6593a32d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df4a: skipped 000881 1786734353.115263 punch from 0x00a217 to 0x013d10 (0x9afa bytes) 000882 1786734353.123361 write 0x0026b6 thru 0x008433 (0x5d7e bytes) 000883 1786734353.135669 trunc from 0x007705 to 0x0083a0 (0xc9c bytes) 000884 1786734353.162935 write 0x00c458 thru 0x00d275 (0xe1e bytes) 000885 1786734353.183228 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1df4a): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df4a): Mirror 14: 0x6593a32d (stale) Mirror 17: 0x6593a32d (stale) Mirror 18: 0x6593a32d (stale) Mirror 19: 0x6593a32d (stale) Mirror 20: 0x6593a32d (stale) CRC-32 checksum value for chunk [0, 0x1df4a): Mirror 1: 0x5cff8e0f Mirror 21: 0x5cff8e0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df4a: skipped 000886 1786734354.303879 trunc from 0x01df4a to 0x00bab6 (0x12493 bytes) 000887 1786734354.407591 punch from 0x005d0d to 0x00bab6 (0x5daa bytes) 000888 1786734354.435493 MIRROR_RESYNC 7 mirrors 000889 1786734355.764130 punch from 0x000375 to 0x0058b3 (0x553f bytes) 000890 1786734355.807409 trunc from 0x002c58 to 0x005403 (0x27ac bytes) 000892 1786734355.815318 punch from 0x00b09e to 0x00bab6 (0xa19 bytes) 000893 1786734355.821387 read 0x00915d thru 0x00bab6 (0x295a bytes) 000894 1786734355.828420 read 0x00300c thru 0x00bab6 (0x8aab bytes) 000895 1786734355.843547 trunc from 0x009f46 to 0x00bab6 (0x1b71 bytes) 000896 1786734355.855930 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xbab7): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbab7): Mirror 14: 0x39c81856 (stale) Mirror 17: 0x39c81856 (stale) Mirror 18: 0x39c81856 (stale) Mirror 19: 0x39c81856 (stale) Mirror 20: 0x39c81856 (stale) Mirror 21: 0x39c81856 (stale) CRC-32 checksum value for chunk [0, 0xbab7): Mirror 1: 0x9f31efc5 Mirror 22: 0x9f31efc5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbab7: skipped 000897 1786734357.214664 fallocate 0x007860 thru 0x00b21a (0x39bb bytes) 000898 1786734357.293692 mapread 0x009015 thru 0x00bab6 (0x2aa2 bytes) 000899 1786734357.325352 trunc from 0x00bab7 to 0x03c466 (0x309b0 bytes) 000900 1786734357.353379 punch from 0x0382b5 to 0x03ab08 (0x2854 bytes) 000902 1786734357.361295 trunc from 0x03c467 to 0x01ce0c (0x1f65a bytes) 000903 1786734357.391680 mapwrite 0x028833 thru 0x033f51 (0xb71f bytes) 000904 1786734357.549376 write 0x00c9ca thru 0x012bb6 (0x61ed bytes) 000905 1786734357.576935 mapwrite 0x03873e thru 0x03ffff (0x78c2 bytes) 000907 1786734357.660287 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x775b5cb0 (stale) Mirror 17: 0x775b5cb0 (stale) Mirror 18: 0x775b5cb0 (stale) Mirror 19: 0x775b5cb0 (stale) Mirror 21: 0x775b5cb0 (stale) Mirror 22: 0xdcb3b750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x775b5cb0 (stale) Mirror 17: 0x775b5cb0 (stale) Mirror 18: 0x775b5cb0 (stale) Mirror 19: 0x775b5cb0 (stale) Mirror 21: 0x775b5cb0 (stale) Mirror 22: 0xdcb3b750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000908 1786734358.116435 mapwrite 0x0098e2 thru 0x019227 (0xf946 bytes) 000909 1786734358.168399 read 0x00924d thru 0x00a2a3 (0x1057 bytes) 000910 1786734358.169559 fallocate 0x022172 thru 0x02fc84 (0xdb13 bytes) 000911 1786734358.176593 trunc from 0x021237 to 0x02e9f8 (0xd7c2 bytes) 000912 1786734358.182681 read 0x0017d1 thru 0x0095ba (0x7dea bytes) 000914 1786734358.203254 mapread 0x0057dd thru 0x0071d4 (0x19f8 bytes) 000915 1786734358.205232 mapwrite 0x036863 thru 0x03ffff (0x979d bytes) 000916 1786734358.229231 mapwrite 0x00e9d2 thru 0x013329 (0x4958 bytes) 000917 1786734358.250144 trunc from 0x023a99 to 0x031716 (0xdc7e bytes) 000918 1786734358.255917 fallocate 0x03b475 thru 0x03ffff (0x4b8b bytes) 000919 1786734358.262455 trunc from 0x000299 to 0x001a43 (0x17ab bytes) 000920 1786734358.272832 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x775b5cb0 (stale) Mirror 17: 0x775b5cb0 (stale) Mirror 18: 0x775b5cb0 (stale) Mirror 19: 0x775b5cb0 (stale) Mirror 21: 0x775b5cb0 (stale) Mirror 22: 0xdcb3b750 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf474009b Mirror 23: 0xf474009b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000921 1786734359.350146 mapread 0x014c4f thru 0x01f75e (0xab10 bytes) 000922 1786734359.401402 read 0x03c18a thru 0x03ffff (0x3e76 bytes) 000923 1786734359.417611 fallocate 0x01ca56 thru 0x01ede0 (0x238b bytes) 000924 1786734359.458039 fallocate 0x0221b2 thru 0x0276ca (0x5519 bytes) 000925 1786734359.475629 write 0x03097a thru 0x03e019 (0xd6a0 bytes) 000926 1786734359.493373 mapread 0x02d19b thru 0x02f394 (0x21fa bytes) 000927 1786734359.512400 trunc from 0x0055f3 to 0x00eb62 (0x9570 bytes) 000928 1786734359.533400 write 0x0365d5 thru 0x03ac0d (0x4639 bytes) 000929 1786734359.534351 mapread 0x012a7a thru 0x022500 (0xfa87 bytes) 000930 1786734359.618323 mapwrite 0x028ee7 thru 0x029e73 (0xf8d bytes) 000931 1786734359.665406 punch from 0x024402 to 0x02857a (0x4179 bytes) 000932 1786734359.678850 write 0x03bb97 thru 0x03ffff (0x4469 bytes) 000933 1786734359.688067 trunc from 0x040000 to 0x035998 (0xa667 bytes) 000934 1786734359.754243 punch from 0x026164 to 0x035998 (0xf835 bytes) 000935 1786734359.766195 mapwrite 0x035c0c thru 0x03c4bb (0x68b0 bytes) 000936 1786734359.836667 mapread 0x006fbb thru 0x00dafb (0x6b41 bytes) 000937 1786734359.851863 trunc from 0x03c4bc to 0x024760 (0x17d5b bytes) 000938 1786734359.887351 trunc from 0x024761 to 0x00d9f9 (0x16d67 bytes) 000939 1786734359.907777 fallocate 0x026642 thru 0x032957 (0xc316 bytes) 000940 1786734359.929272 fallocate 0x005178 thru 0x00efca (0x9e53 bytes) 000941 1786734359.937616 write 0x00409c thru 0x00f055 (0xafba bytes) 000942 1786734359.950308 mapwrite 0x01bbec thru 0x028f0c (0xd321 bytes) 000943 1786734359.992464 punch from 0x0184e3 to 0x01b521 (0x303f bytes) 000944 1786734359.999996 mapread 0x010c9a thru 0x01f757 (0xeabe bytes) 000945 1786734360.016257 trunc from 0x02bb48 to 0x0326fd (0x6bb6 bytes) 000946 1786734360.024244 mapwrite 0x01d10f thru 0x02cf67 (0xfe59 bytes) 000947 1786734360.060187 punch from 0x009078 to 0x00d2d8 (0x4261 bytes) 000949 1786734360.075677 trunc from 0x010ebd to 0x018af9 (0x7c3d bytes) 000950 1786734360.091844 MIRROR_RESYNC 8 mirrors 000951 1786734362.493636 fallocate 0x038ef5 thru 0x039ce1 (0xded bytes) 000952 1786734362.528352 mapread 0x00cc74 thru 0x00f4d2 (0x285f bytes) 000953 1786734362.556425 mapread 0x0307af thru 0x039ce1 (0x9533 bytes) 000954 1786734362.572539 mapwrite 0x034337 thru 0x03ffff (0xbcc9 bytes) 000955 1786734362.626533 punch from 0x03b3a4 to 0x03ffff (0x4c5c bytes) 000957 1786734362.636509 write 0x021549 thru 0x02f3b5 (0xde6d bytes) 000958 1786734362.644852 trunc from 0x02f9bf to 0x03a405 (0xaa47 bytes) 000959 1786734362.660350 read 0x02dd41 thru 0x037a95 (0x9d55 bytes) 000960 1786734362.670366 trunc from 0x01cd76 to 0x02688e (0x9b19 bytes) 000961 1786734362.680592 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x17d901c4 (stale) Mirror 17: 0x17d901c4 (stale) Mirror 18: 0x17d901c4 (stale) Mirror 19: 0x17d901c4 (stale) Mirror 21: 0x17d901c4 (stale) Mirror 23: 0x17d901c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x17d901c4 (stale) Mirror 17: 0x17d901c4 (stale) Mirror 18: 0x17d901c4 (stale) Mirror 19: 0x17d901c4 (stale) Mirror 21: 0x17d901c4 (stale) Mirror 23: 0x17d901c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000962 1786734363.149398 mapwrite 0x006d05 thru 0x00b567 (0x4863 bytes) 000964 1786734363.191624 read 0x02cf56 thru 0x033783 (0x682e bytes) 000966 1786734363.200419 trunc from 0x040000 to 0x00007e (0x3ff81 bytes) 000967 1786734363.224210 mapwrite 0x01794b thru 0x020ef8 (0x95ae bytes) 000968 1786734363.286234 read 0x004c5d thru 0x01347c (0xe820 bytes) 000969 1786734363.304541 mapread 0x00ffe8 thru 0x0143a8 (0x43c1 bytes) 000971 1786734363.306444 mapread 0x00f515 thru 0x01a2ff (0xadeb bytes) 000972 1786734363.313349 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20ef9): Mirror 14: 0x3e37f040 (stale) Mirror 17: 0x3e37f040 (stale) Mirror 18: 0x3e37f040 (stale) Mirror 19: 0x3e37f040 (stale) Mirror 21: 0x3e37f040 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20ef9): Mirror 14: 0x3e37f040 (stale) Mirror 17: 0x3e37f040 (stale) Mirror 18: 0x3e37f040 (stale) Mirror 19: 0x3e37f040 (stale) Mirror 21: 0x3e37f040 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20ef9: skipped 000973 1786734363.803485 fallocate 0x01590d thru 0x01db73 (0x8267 bytes) 000974 1786734363.814387 MIRROR_RESYNC 6 mirrors 000975 1786734364.763508 trunc from 0x020ef9 to 0x03cedb (0x1bfe3 bytes) 000976 1786734364.813031 write 0x01a6c8 thru 0x0234af (0x8de8 bytes) 000977 1786734364.847615 read 0x02ad19 thru 0x039a35 (0xed1d bytes) 000978 1786734364.865622 trunc from 0x03cedc to 0x01710b (0x25dd0 bytes) 000979 1786734364.896814 write 0x021812 thru 0x02f9ff (0xe1ee bytes) 000980 1786734364.909233 mapread 0x0176a4 thru 0x0258c7 (0xe224 bytes) 000981 1786734364.940512 punch from 0x01e52d to 0x01f716 (0x11ea bytes) 000982 1786734364.948091 trunc from 0x02fa00 to 0x00e3ce (0x21631 bytes) 000983 1786734364.975513 mapread 0x000620 thru 0x00de3a (0xd81b bytes) 000984 1786734364.998191 write 0x011d82 thru 0x01a098 (0x8317 bytes) 000985 1786734365.000359 fallocate 0x002205 thru 0x006978 (0x4774 bytes) 000986 1786734365.013549 fallocate 0x03b294 thru 0x03ce08 (0x1b75 bytes) 000987 1786734365.030374 read 0x0376a9 thru 0x03b04d (0x39a5 bytes) 000988 1786734365.046473 mapread 0x039f3a thru 0x03ce08 (0x2ecf bytes) 000989 1786734365.048123 mapread 0x0388bb thru 0x03ce08 (0x454e bytes) 000990 1786734365.051628 punch from 0x02dac9 to 0x038888 (0xadc0 bytes) 000991 1786734365.067849 fallocate 0x03de8b thru 0x03ffff (0x2175 bytes) 000992 1786734365.086259 punch from 0x00f595 to 0x018178 (0x8be4 bytes) 000993 1786734365.124162 punch from 0x034248 to 0x03ffff (0xbdb8 bytes) 000994 1786734365.130447 write 0x00c994 thru 0x015eb3 (0x9520 bytes) 000995 1786734365.144606 write 0x029c6b thru 0x038a2a (0xedc0 bytes) 000996 1786734365.163203 MIRROR_RESYNC 6 mirrors 000997 1786734367.346744 fallocate 0x01111f thru 0x01468d (0x356f bytes) 000999 1786734367.389487 mapwrite 0x03d303 thru 0x03eb51 (0x184f bytes) 001000 1786734367.421609 mapwrite 0x03eea9 thru 0x03ffff (0x1157 bytes) All operations completed A-OK! PASS 70a (120s) == sanity-flr test 71: check mirror extend parent fid ==== 15:06:13 (1786734373) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x87:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 15:06:25 (1786734385) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161099 s, 13.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815878 s, 12.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 15:06:37 (1786734397) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7526560 .. Extending file size to 6871392 .. Extending file size to 7055488 .. Extending file size to 8242144 .. Extending file size to 8927904 .. Extending file size to 8276576 .. Extending file size to 8333344 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8370656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8475616 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8576000 .. Extending file size to 8801632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61274 61276 61278 61279 61281 done 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:07:59 (1786734479) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9103232 .. done Extending file size to 3467040 .. Extending file size to 5766816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8064576 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8287616 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8743904 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62881 62882 62884 62885 62886 done 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:09:17 (1786734557) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25572 3581448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17712 3589308 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43284 7170756 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25572 3581448 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17712 3589308 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43284 7170756 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120039 s, 17.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193497 s, 10.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf8152808 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:09:37 (1786734577) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10446 s, 20.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 15:09:44 (1786734584) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x96:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:09:53 (1786734593) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0658828 s, 31.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:10:01 (1786734601) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106611 s, 19.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ea:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:10:11 (1786734611) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17106 s, 12.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:10:45 (1786734645) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074917 s, 14.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e9:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:11:37 (1786734697) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0614804 s, 34.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:12:34 (1786734754) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg248-server: Pool lustre.test_204f created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0954527 s, 22.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg248-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:13:52 (1786734832) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:14:00 (1786734840) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:14:09 (1786734849) oleg248-server: Pool lustre.test_206 created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg248-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:14:38 (1786734878) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145337 s, 7.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366915 s, 28.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4bcb0d47 Mirror 3: 0x4bcb0d47 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:14:51 (1786734891) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:14:55 (1786734895) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:14:59 (1786734899) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0527869 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2630000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2630000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2630000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19e0000 Read this (4 bytes): BBBB 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:15:39 (1786734939) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0113876 s, 719 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:15:45 (1786734945) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602106 s, 17.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb3 lmm_fid: [0x200000405:0xb3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb3 lmm_fid: [0x200000405:0xb3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611315 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 15:15:47.000000000 -0400 Modify: 2026-08-14 15:15:47.000000000 -0400 Change: 2026-08-14 15:15:48.000000000 -0400 Birth: 2026-08-14 15:15:46.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:15:55 (1786734955) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753788 s, 13.9 MB/s oleg248-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg248-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg248-server: O/280000401/d24/158: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:16:18 (1786734978) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.192415 s, 213 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0190891 s, 215 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 15:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 15:16 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:16:25 (1786734985) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:16:34 (1786734994) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1883 sec =========== 15:16:43 (1786735003) === sanity-flr: start cleanup 15:16:45 (1786735005) === === sanity-flr: finish cleanup 15:16:53 (1786735013) ===