-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 17:55:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:55:49 (1787781349) === oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f2bd13b8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f2bd13b8000.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:56:35 (1787781395) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8224] == sanity-flr test 0a: lfs mirror create with -N option == 17:56:38 (1787781398) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:57:00 (1787781420) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:57:04 (1787781424) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:57:08 (1787781428) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:57:35 (1787781455) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:57:38 (1787781458) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:57:42 (1787781462) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f2bd13b8000.stats=clear osc.lustre-OST0001-osc-ffff8f2bd13b8000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:57:49 (1787781469) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:58:00 (1787781480) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:58:08 (1787781488) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:58:11 (1787781491) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:58:19 (1787781499) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 17:58:29 (1787781509) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:58:37 (1787781517) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:58:46 (1787781526) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:58:55 (1787781535) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:59:06 (1787781546) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:59:13 (1787781553) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.83936 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150053 s, 7.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:59:24 (1787781564) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.701611 s, 20.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:59:35 (1787781575) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:59:46 (1787781586) Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:01:07 (1787781667) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.267434 s, 15.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:01:49 (1787781709) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:03:21 (1787781801) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.70328 s, 19.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 18:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.950s user 0m0.001s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.187s user 0m0.000s sys 0m0.207s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.239s user 0m0.002s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f2bd13b8000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:05:00 (1787781900) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:05:04 (1787781904) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:05:08 (1787781908) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 18:05:12 (1787781912) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 18:05:19 (1787781919) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.758734 s, 16.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:05:33 (1787781933) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405318 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792504 s, 13.2 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:05:42 (1787781942) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:05:51 (1787781951) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00629005 s, 163 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00674299 s, 152 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0063004 s, 163 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0031665 s, 323 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 18:06:05 (1787781965) create test files with size 15749962 .. d580d38c393a0917bc3a541bd0364ddc - b20a9a69db8bf8d2694c46100a34931c - 39d680ac630881a13a2f0de19b233550 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 18:06:36 (1787781996) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:06:53 (1787782013) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:07:03 (1787782023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636845 s, 16.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796871 s, 13.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 18:07:14 (1787782034) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105908 s, 19.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.347173 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165184 s, 12.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 18:07:28 (1787782048) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:07:31 (1787782051) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:07:35 (1787782055) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452541 s, 23.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:07:43 (1787782063) llite.lustre-ffff8f2bc6533000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 18:07:52 (1787782072) llite.lustre-ffff8f2bc6533000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290762 s, 36.1 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251073 s, 41.8 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 18:08:02 (1787782082) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.354184 s, 8.9 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:08:13 (1787782093) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:08:22 (1787782102) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353216 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xa:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:08:33 (1787782113) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f2bd4c85000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.378016 s, 27.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f2bc78b1800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:08:50 (1787782130) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:09:29 (1787782169) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:10:03 (1787782203) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 18:10:07 (1787782207) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196308 s, 16.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.141419 s, 22.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaf5c0c7d (nosync,stale) Mirror 2: 0xaf5c0c7d (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5947cebfa2d217f9b1a9a450439604b1 - mirror 1 checksum: ade0b8e4f36eb035dd0e016c17d59266 - mirror 2 checksum: 5947cebfa2d217f9b1a9a450439604b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T18:10:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T18:10:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5947cebfa2d217f9b1a9a450439604b1 - mirror 1 checksum: ade0b8e4f36eb035dd0e016c17d59266 - mirror 2 checksum: ade0b8e4f36eb035dd0e016c17d59266 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T18:10:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T18:10:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:10:17 (1787782217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249351 s, 42.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:10:23 (1787782223) Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server 18:10:35 (1787782235) shut down facet: mds1 facet_host: oleg450-server facet_failover_host: oleg450-server Failover mds1 to oleg450-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 18:11:00 (1787782260) targets are mounted 18:11:00 (1787782260) facet_failover done oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351141 s, 29.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:11:18 (1787782278) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0682684 s, 300 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0147981 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:11:29 (1787782289) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0906569 s, 226 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0120199 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:11:40 (1787782300) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0140718 s, 1.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:11:48 (1787782308) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0522698 s, 392 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:11:57 (1787782317) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100032 s, 409 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0433573 s, 94.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:12:04 (1787782324) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:12:29 (1787782349) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:13:00 (1787782380) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:13:26 (1787782406) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:13:51 (1787782431) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 18:13:59 (1787782439) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58873 mirror split pid 58874 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58873 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58874 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 18:15:06 (1787782506) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2508 000002 1787782508.841882 mapwrite 0x02ce41 thru 0x0357e5 (0x89a5 bytes) 000004 1787782508.956936 read 0x02866c thru 0x03527e (0xcc13 bytes) 000005 1787782508.958179 mapread 0x024cc3 thru 0x0310ca (0xc408 bytes) 000006 1787782508.980381 MIRROR_EXTEND to 2 mirrors zero_range to largest ever: 0xd6b2 000007 1787782509.725678 trunc from 0x003429 to 0x00d6b1 (0xa289 bytes) fallocating to largest ever: 0x14056 000008 1787782509.767110 fallocate 0x0056e1 thru 0x014055 (0xe975 bytes) 000009 1787782509.773001 mapwrite 0x00b40d thru 0x01a3e6 (0xefda bytes) 000010 1787782509.840830 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) 000012 1787782510.061093 mapread 0x0082cf thru 0x016caa (0xe9dc bytes) fallocating to largest ever: 0x1c655 000013 1787782510.100107 fallocate 0x00e612 thru 0x01c654 (0xe043 bytes) 000014 1787782510.105806 mapwrite 0x0324be thru 0x03e93c (0xc47f bytes) fallocating to largest ever: 0x285a9 000015 1787782510.203192 fallocate 0x02615d thru 0x0285a8 (0x244c bytes) 000016 1787782510.212173 punch from 0x019245 to 0x02669f (0xd45b bytes) 000017 1787782510.219089 trunc from 0x03e93d to 0x00613c (0x38800 bytes) 000018 1787782510.239556 read 0x0043e0 thru 0x00613c (0x1d5d bytes) 000019 1787782510.240159 read 0x004025 thru 0x00613c (0x2118 bytes) 000020 1787782510.240703 trunc from 0x0030d9 to 0x0049fa (0x1922 bytes) fallocating to largest ever: 0x2de96 000021 1787782510.246604 fallocate 0x01f692 thru 0x02de95 (0xe804 bytes) 000022 1787782510.251965 trunc from 0x02de96 to 0x00ca32 (0x21463 bytes) 000023 1787782510.275141 MIRROR_EXTEND to 2 mirrors 000024 1787782510.922544 trunc from 0x001e22 to 0x004679 (0x2858 bytes) 000025 1787782510.996602 punch from 0x0022f5 to 0x0068eb (0x45f7 bytes) 000027 1787782511.005873 punch from 0x00320b to 0x008a90 (0x5886 bytes) 000028 1787782511.018528 read 0x003dad thru 0x00943b (0x568f bytes) 000029 1787782511.051072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xca33): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca33): Mirror 2: 0x47e43199 (stale) CRC-32 checksum value for chunk [0, 0xca33): Mirror 1: 0x47e43199 Mirror 3: 0x47e43199 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca33: skipped 000030 1787782511.984025 write 0x014e38 thru 0x01d1f2 (0x83bb bytes) 000032 1787782512.037435 mapread 0x0052f6 thru 0x013b7c (0xe887 bytes) 000033 1787782512.062055 mapread 0x018137 thru 0x01d1f2 (0x50bc bytes) 000034 1787782512.069266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d1f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d1f3): Mirror 2: 0xd0f257b2 (stale) Mirror 3: 0xd0f257b2 (stale) CRC-32 checksum value for chunk [0, 0x1d1f3): Mirror 1: 0xc95e5fa Mirror 4: 0xc95e5fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d1f3: skipped 000035 1787782512.930312 mapread 0x00972c thru 0x00febe (0x6793 bytes) 000036 1787782512.959076 read 0x005a1b thru 0x00cdb0 (0x7396 bytes) 000037 1787782512.959599 trunc from 0x006360 to 0x00a6a1 (0x4342 bytes) 000038 1787782513.003575 punch from 0x016a8c to 0x01d1f2 (0x6767 bytes) 000039 1787782513.009852 mapwrite 0x03c194 thru 0x03ffff (0x3e6c bytes) 000040 1787782513.094356 read 0x02fe72 thru 0x0351d5 (0x5364 bytes) 000041 1787782513.129535 mapwrite 0x023d02 thru 0x030d4e (0xd04d bytes) 000042 1787782513.153947 punch from 0x015eb1 to 0x01a5c2 (0x4712 bytes) 000043 1787782513.161103 fallocate 0x018297 thru 0x0219dc (0x9746 bytes) 000044 1787782513.168193 punch from 0x00be02 to 0x015605 (0x9804 bytes) 000046 1787782513.177129 mapread 0x01ff60 thru 0x02652b (0x65cc bytes) 000048 1787782513.179575 mapwrite 0x01156d thru 0x01e2f3 (0xcd87 bytes) 000049 1787782513.258538 fallocate 0x01afe5 thru 0x01fd53 (0x4d6f bytes) 000050 1787782513.264020 trunc from 0x040000 to 0x02d6b7 (0x12948 bytes) 000051 1787782513.281422 mapwrite 0x0123c0 thru 0x012919 (0x55a bytes) truncating to largest ever: 0x36ea4 000052 1787782513.294946 trunc from 0x02d6b8 to 0x036ea3 (0x97ec bytes) 000053 1787782513.316412 trunc from 0x03103f to 0x0322cd (0x128f bytes) 000054 1787782513.321919 read 0x026ccd thru 0x034806 (0xdb3a bytes) 000055 1787782513.331943 fallocate 0x024238 thru 0x02df17 (0x9ce0 bytes) 000056 1787782513.337201 punch from 0x02d43a to 0x031a64 (0x462b bytes) 000057 1787782513.343423 read 0x01c565 thru 0x01dd6b (0x1807 bytes) 000058 1787782513.349388 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 2: 0xbf0153d2 (stale) Mirror 3: 0xbf0153d2 (stale) Mirror 4: 0xc31161bf (stale) CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 1: 0xf7d6480 Mirror 5: 0xf7d6480 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ea4: skipped 000059 1787782514.242008 write 0x0162d2 thru 0x024433 (0xe162 bytes) 000061 1787782514.358600 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 2: 0xbf0153d2 (stale) Mirror 3: 0xbf0153d2 (stale) Mirror 4: 0xc31161bf (stale) Mirror 5: 0xf7d6480 (stale) CRC-32 checksum value for chunk [0, 0x36ea4): Mirror 1: 0xacbbee27 Mirror 6: 0xacbbee27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ea4: skipped 000062 1787782515.566961 trunc from 0x036ea4 to 0x01b68b (0x1b818 bytes) 000063 1787782515.634372 punch from 0x00ae88 to 0x016206 (0xb37f bytes) 000064 1787782515.639398 write 0x026e19 thru 0x02c702 (0x58ea bytes) 000065 1787782515.640682 write 0x02fed1 thru 0x03a47e (0xa5ae bytes) 000067 1787782515.643106 read 0x02508a thru 0x027e43 (0x2dba bytes) 000068 1787782515.648432 trunc from 0x00b720 to 0x0163ae (0xac8f bytes) 000069 1787782515.653452 mapwrite 0x030a14 thru 0x0363b9 (0x59a6 bytes) fallocating to largest ever: 0x37378 000070 1787782515.677570 fallocate 0x02c825 thru 0x037377 (0xab53 bytes) 000071 1787782515.691124 read 0x00e780 thru 0x01ae24 (0xc6a5 bytes) 000072 1787782515.724596 write 0x0103b9 thru 0x011494 (0x10dc bytes) 000073 1787782515.727973 read 0x02d7a1 thru 0x0339b9 (0x6219 bytes) 000074 1787782515.729542 read 0x018aa4 thru 0x024491 (0xb9ee bytes) 000075 1787782515.730388 mapwrite 0x03edab thru 0x03ffff (0x1255 bytes) 000076 1787782515.800877 write 0x025928 thru 0x029547 (0x3c20 bytes) 000077 1787782515.802024 fallocate 0x024d66 thru 0x02b363 (0x65fe bytes) 000078 1787782515.807065 trunc from 0x040000 to 0x007b08 (0x384f7 bytes) 000079 1787782515.827911 MIRROR_RESYNC 6 mirrors 000080 1787782516.866041 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) 000081 1787782517.239391 trunc from 0x007b09 to 0x01efb0 (0x174a8 bytes) 000082 1787782517.323723 trunc from 0x01efb1 to 0x00bc92 (0x1331e bytes) 000083 1787782517.345966 punch from 0x0036ea to 0x009de5 (0x66fc bytes) 000084 1787782517.354012 trunc from 0x007c66 to 0x00bc92 (0x402d bytes) 000085 1787782517.364851 punch from 0x001f43 to 0x00bc92 (0x9d50 bytes) 000086 1787782517.371786 punch from 0x00760e to 0x00bc92 (0x4685 bytes) 000087 1787782517.377676 write 0x0217f9 thru 0x024be2 (0x33ea bytes) 000088 1787782517.381755 read 0x00f8cb thru 0x01cd92 (0xd4c8 bytes) 000089 1787782517.393334 trunc from 0x00392d to 0x0047d5 (0xea9 bytes) 000090 1787782517.411507 MIRROR_RESYNC 5 mirrors 000091 1787782518.701645 punch from 0x011597 to 0x01ba4e (0xa4b8 bytes) 000092 1787782518.753369 write 0x034ea5 thru 0x03c876 (0x79d2 bytes) 000093 1787782518.754849 write 0x01a60a thru 0x01b1bd (0xbb4 bytes) 000094 1787782518.768113 punch from 0x038bc7 to 0x03c876 (0x3cb0 bytes) 000095 1787782518.795860 write 0x02b995 thru 0x03195a (0x5fc6 bytes) 000096 1787782518.812387 mapwrite 0x019fcf thru 0x02314f (0x9181 bytes) 000097 1787782518.896331 MIRROR_RESYNC 5 mirrors 000098 1787782521.020094 mapread 0x013609 thru 0x0165c2 (0x2fba bytes) 000099 1787782521.055075 mapwrite 0x03ad3f thru 0x03d1a0 (0x2462 bytes) fallocating to largest ever: 0x3984b 000101 1787782521.190196 fallocate 0x0397b6 thru 0x03984a (0x95 bytes) 000102 1787782521.205329 mapwrite 0x03346f thru 0x03d21d (0x9daf bytes) 000103 1787782521.272860 read 0x0100ee thru 0x01a013 (0x9f26 bytes) 000104 1787782521.281707 read 0x03c359 thru 0x03d21d (0xec5 bytes) 000105 1787782521.282490 mapwrite 0x005c84 thru 0x00885a (0x2bd7 bytes) 000106 1787782521.317074 MIRROR_RESYNC 5 mirrors 000107 1787782523.485115 mapwrite 0x00445c thru 0x00556c (0x1111 bytes) 000108 1787782523.594088 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d21e): Mirror 2: 0x48cddfac (stale) Mirror 5: 0x48cddfac (stale) Mirror 6: 0x48cddfac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d21e): Mirror 2: 0x48cddfac (stale) Mirror 5: 0x48cddfac (stale) Mirror 6: 0x48cddfac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d21e: skipped 000109 1787782523.985374 punch from 0x03d0e4 to 0x03d21d (0x13a bytes) 000110 1787782524.004135 read 0x036098 thru 0x03d21d (0x7186 bytes) 000111 1787782524.038979 punch from 0x037f65 to 0x038a4a (0xae6 bytes) 000112 1787782524.053850 mapwrite 0x03b3a8 thru 0x03ffff (0x4c58 bytes) fallocating to largest ever: 0x40000 000113 1787782524.241453 fallocate 0x03f0c3 thru 0x03ffff (0xf3d bytes) 000114 1787782524.262738 trunc from 0x0296ba to 0x035aa3 (0xc3ea bytes) 000115 1787782524.291631 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x13b60747 (stale) Mirror 5: 0x13b60747 (stale) Mirror 6: 0x13b60747 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fa46d3a Mirror 7: 0x8fa46d3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000116 1787782525.571933 mapread 0x000494 thru 0x000c28 (0x795 bytes) 000117 1787782525.587196 mapread 0x01e55c thru 0x024aaa (0x654f bytes) 000118 1787782525.620851 write 0x037a87 thru 0x03954e (0x1ac8 bytes) 000119 1787782525.769007 trunc from 0x040000 to 0x00c993 (0x3366c bytes) 000120 1787782525.843121 mapread 0x007f65 thru 0x00c993 (0x4a2f bytes) 000121 1787782525.863955 punch from 0x00a97d to 0x00c993 (0x2017 bytes) 000122 1787782525.871440 mapread 0x00535b thru 0x00c993 (0x7639 bytes) 000123 1787782525.881714 trunc from 0x00a4b7 to 0x00c993 (0x24dd bytes) 000124 1787782525.887276 punch from 0x009abe to 0x00c993 (0x2ed6 bytes) 000125 1787782525.908182 read 0x009f99 thru 0x00c993 (0x29fb bytes) 000127 1787782525.916797 mapwrite 0x0030a0 thru 0x007395 (0x42f6 bytes) 000128 1787782525.937615 mapwrite 0x0028e1 thru 0x00b534 (0x8c54 bytes) 000129 1787782525.961679 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc994): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc994): Mirror 2: 0xf00771b5 (stale) Mirror 5: 0xf00771b5 (stale) Mirror 6: 0xf00771b5 (stale) Mirror 7: 0xa8d17d07 (stale) CRC-32 checksum value for chunk [0, 0xc994): Mirror 1: 0xa60d7a6b Mirror 8: 0xa60d7a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc994: skipped 000130 1787782527.432902 mapwrite 0x000862 thru 0x002692 (0x1e31 bytes) 000131 1787782527.542917 read 0x00ac62 thru 0x00c993 (0x1d32 bytes) 000132 1787782527.553164 punch from 0x00c09b to 0x00c993 (0x8f9 bytes) 000133 1787782527.566942 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc994): Mirror 2: 0xf00771b5 (stale) Mirror 6: 0xf00771b5 (stale) Mirror 7: 0xa8d17d07 (stale) Mirror 8: 0xa60d7a6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc994): Mirror 2: 0xf00771b5 (stale) Mirror 6: 0xf00771b5 (stale) Mirror 7: 0xa8d17d07 (stale) Mirror 8: 0xa60d7a6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc994: skipped 000134 1787782528.078939 MIRROR_RESYNC 5 mirrors 000135 1787782529.571005 MIRROR_EXTEND to 6 mirrors 000136 1787782530.780152 punch from 0x004f7c to 0x00c993 (0x7a18 bytes) 000137 1787782530.867268 read 0x0069d9 thru 0x009b4f (0x3177 bytes) 000138 1787782530.893949 read 0x000328 thru 0x00b3f6 (0xb0cf bytes) 000139 1787782530.910967 read 0x003d4c thru 0x00c993 (0x8c48 bytes) 000140 1787782530.911615 trunc from 0x00c994 to 0x00435f (0x8634 bytes) 000141 1787782531.007003 mapwrite 0x01eef9 thru 0x0212a9 (0x23b1 bytes) 000143 1787782531.137011 write 0x02ff9f thru 0x037d71 (0x7dd3 bytes) 000144 1787782531.147086 trunc from 0x037ae0 to 0x037d71 (0x292 bytes) 000145 1787782531.193444 mapread 0x017cfc thru 0x01ff11 (0x8216 bytes) 000146 1787782531.195695 mapwrite 0x033756 thru 0x03d79b (0xa046 bytes) 000147 1787782531.309199 trunc from 0x01b98b to 0x02338e (0x7a04 bytes) 000149 1787782531.327110 punch from 0x02ac77 to 0x02d9b5 (0x2d3f bytes) 000150 1787782531.335288 mapwrite 0x022eb5 thru 0x0248cf (0x1a1b bytes) 000151 1787782531.373588 fallocate 0x023e75 thru 0x032260 (0xe3ec bytes) 000152 1787782531.382134 write 0x001e25 thru 0x00b84c (0x9a28 bytes) 000153 1787782531.383624 mapread 0x03543a thru 0x03ba3d (0x6604 bytes) 000154 1787782531.387061 read 0x007813 thru 0x007dfe (0x5ec bytes) 000155 1787782531.387793 read 0x000a2a thru 0x00fd2c (0xf303 bytes) 000156 1787782531.391692 punch from 0x016c8d to 0x0228a1 (0xbc15 bytes) 000158 1787782531.414664 write 0x01a994 thru 0x024818 (0x9e85 bytes) 000159 1787782531.447619 punch from 0x00c98f to 0x01b300 (0xe972 bytes) 000160 1787782531.468142 fallocate 0x01dffb thru 0x02902c (0xb032 bytes) 000161 1787782531.478034 mapread 0x004257 thru 0x007e2e (0x3bd8 bytes) 000162 1787782531.478873 mapwrite 0x01afcc thru 0x01e357 (0x338c bytes) 000163 1787782531.527905 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d79c): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d79c): Mirror 2: 0x54554a35 (stale) Mirror 6: 0x54554a35 (stale) Mirror 7: 0x54554a35 (stale) Mirror 8: 0x54554a35 (stale) Mirror 9: 0x54554a35 (stale) CRC-32 checksum value for chunk [0, 0x3d79c): Mirror 1: 0x98e17563 Mirror 10: 0x98e17563 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d79c: skipped 000164 1787782532.454361 trunc from 0x03bbb9 to 0x03d79b (0x1be3 bytes) 000166 1787782532.496201 trunc from 0x03bc04 to 0x03d79b (0x1b98 bytes) 000167 1787782532.502427 read 0x003d92 thru 0x010fc0 (0xd22f bytes) 000168 1787782532.515040 mapread 0x010aba thru 0x012c3f (0x2186 bytes) 000169 1787782532.523509 mapread 0x00f4d9 thru 0x011b8f (0x26b7 bytes) 000170 1787782532.524325 punch from 0x010ea9 to 0x01ecdd (0xde35 bytes) 000171 1787782532.529968 read 0x02c163 thru 0x031b92 (0x5a30 bytes) 000172 1787782532.539885 mapread 0x02e649 thru 0x0378e6 (0x929e bytes) 000173 1787782532.541007 mapwrite 0x01fc06 thru 0x021fd3 (0x23ce bytes) 000175 1787782532.553143 trunc from 0x03d79c to 0x00a354 (0x33447 bytes) 000178 1787782532.612536 trunc from 0x00a355 to 0x01d216 (0x12ec2 bytes) 000179 1787782532.637568 mapread 0x01698c thru 0x01d216 (0x688b bytes) 000180 1787782532.681581 mapread 0x018c69 thru 0x01d216 (0x45ae bytes) 000181 1787782532.682799 read 0x00cafd thru 0x017b6c (0xb070 bytes) 000182 1787782532.696762 mapread 0x003a3a thru 0x007a2f (0x3ff6 bytes) 000184 1787782532.706076 mapwrite 0x01ab3c thru 0x0265f6 (0xbabb bytes) 000185 1787782532.781705 fallocate 0x012682 thru 0x014204 (0x1b83 bytes) 000186 1787782532.799648 trunc from 0x01a535 to 0x01f537 (0x5003 bytes) 000187 1787782532.806711 mapwrite 0x03ec48 thru 0x03ffff (0x13b8 bytes) 000188 1787782532.845694 trunc from 0x040000 to 0x010de3 (0x2f21c bytes) 000189 1787782532.860136 trunc from 0x00c993 to 0x010de3 (0x4451 bytes) 000190 1787782532.864732 trunc from 0x010de4 to 0x028366 (0x17583 bytes) 000191 1787782532.883794 mapread 0x00c6f1 thru 0x01478b (0x809b bytes) 000192 1787782533.003520 mapwrite 0x016dc6 thru 0x01dba5 (0x6de0 bytes) 000193 1787782533.102998 write 0x01d5af thru 0x01e4cd (0xf1f bytes) 000194 1787782533.116075 fallocate 0x00a5b7 thru 0x0188da (0xe324 bytes) 000195 1787782533.135118 MIRROR_RESYNC 7 mirrors 000196 1787782535.419932 fallocate 0x007ae5 thru 0x0125e7 (0xab03 bytes) 000198 1787782535.494653 mapread 0x01a811 thru 0x024e73 (0xa663 bytes) 000199 1787782535.517108 mapread 0x00f1ce thru 0x019622 (0xa455 bytes) 000200 1787782535.530086 mapwrite 0x03bd0f thru 0x03ffff (0x42f1 bytes) 000201 1787782535.597027 trunc from 0x040000 to 0x02c483 (0x13b7c bytes) 000202 1787782535.616805 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c484): Mirror 2: 0x24ff58ca (stale) Mirror 6: 0x24ff58ca (stale) Mirror 7: 0x24ff58ca (stale) Mirror 9: 0x24ff58ca (stale) Mirror 10: 0x24ff58ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c484): Mirror 2: 0x24ff58ca (stale) Mirror 6: 0x24ff58ca (stale) Mirror 7: 0x24ff58ca (stale) Mirror 9: 0x24ff58ca (stale) Mirror 10: 0x24ff58ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c484: skipped 000203 1787782536.135304 mapwrite 0x034678 thru 0x03ffff (0xb988 bytes) 000204 1787782536.207557 trunc from 0x00bb62 to 0x00f852 (0x3cf1 bytes) 000205 1787782536.213141 trunc from 0x008765 to 0x015b0b (0xd3a7 bytes) 000206 1787782536.218762 fallocate 0x01f865 thru 0x023ec2 (0x465e bytes) 000207 1787782536.231321 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7218b1b5 (stale) Mirror 7: 0x7218b1b5 (stale) Mirror 9: 0x7218b1b5 (stale) Mirror 10: 0x7218b1b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7218b1b5 (stale) Mirror 7: 0x7218b1b5 (stale) Mirror 9: 0x7218b1b5 (stale) Mirror 10: 0x7218b1b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000208 1787782536.825912 trunc from 0x040000 to 0x011977 (0x2e688 bytes) 000209 1787782536.849129 trunc from 0x011978 to 0x03a672 (0x28cfb bytes) 000210 1787782536.871651 write 0x01b7fa thru 0x02b642 (0xfe49 bytes) 000211 1787782536.895465 fallocate 0x0044f2 thru 0x0128b7 (0xe3c6 bytes) 000212 1787782536.901958 mapwrite 0x02a217 thru 0x02b700 (0x14ea bytes) 000213 1787782536.928008 read 0x02dc72 thru 0x03a245 (0xc5d4 bytes) 000214 1787782536.955332 fallocate 0x0012c1 thru 0x005e2d (0x4b6d bytes) 000215 1787782536.966319 read 0x02dec9 thru 0x034f50 (0x7088 bytes) 000216 1787782536.971965 trunc from 0x03a673 to 0x0237d3 (0x16e9f bytes) 000217 1787782537.058531 mapread 0x01a02b thru 0x0237d3 (0x97a9 bytes) 000218 1787782537.083292 mapread 0x01ae79 thru 0x01b275 (0x3fd bytes) 000219 1787782537.095481 write 0x02ca7d thru 0x0392c3 (0xc847 bytes) 000220 1787782537.105591 trunc from 0x019e71 to 0x02446c (0xa5fc bytes) 000222 1787782537.114043 read 0x02f3fd thru 0x034219 (0x4e1d bytes) 000223 1787782537.116505 mapwrite 0x033476 thru 0x03ffff (0xcb8a bytes) 000224 1787782537.223420 fallocate 0x0345ae thru 0x03a207 (0x5c5a bytes) 000225 1787782537.238245 read 0x015be3 thru 0x01bacb (0x5ee9 bytes) 000226 1787782537.240172 write 0x038184 thru 0x03b8c9 (0x3746 bytes) 000227 1787782537.243050 trunc from 0x040000 to 0x03910f (0x6ef0 bytes) 000229 1787782537.267591 fallocate 0x0230ad thru 0x026ae7 (0x3a3b bytes) 000230 1787782537.275973 trunc from 0x016727 to 0x0185ec (0x1ec6 bytes) 000231 1787782537.286055 mapread 0x0180f1 thru 0x020450 (0x8360 bytes) 000233 1787782537.297437 trunc from 0x00ce63 to 0x01842a (0xb5c8 bytes) 000234 1787782537.304577 write 0x028980 thru 0x03129e (0x891f bytes) 000235 1787782537.314173 MIRROR_RESYNC 5 mirrors 000236 1787782538.630599 fallocate 0x03c44e thru 0x03ffff (0x3bb2 bytes) 000237 1787782538.676011 fallocate 0x00c706 thru 0x0153fd (0x8cf8 bytes) 000238 1787782538.682300 mapwrite 0x018606 thru 0x01e022 (0x5a1d bytes) 000239 1787782538.730303 read 0x03f042 thru 0x03ffff (0xfbe bytes) 000240 1787782538.736137 write 0x020ce3 thru 0x02e342 (0xd660 bytes) 000241 1787782538.746121 trunc from 0x040000 to 0x004046 (0x3bfb9 bytes) 000242 1787782538.769153 read 0x00257f thru 0x004046 (0x1ac8 bytes) 000243 1787782538.780824 punch from 0x001da9 to 0x00339d (0x15f5 bytes) 000244 1787782538.786860 mapwrite 0x03f382 thru 0x03ffff (0xc7e bytes) 000245 1787782538.836827 write 0x00691c thru 0x0101cb (0x98b0 bytes) 000246 1787782538.838364 punch from 0x01dab5 to 0x025683 (0x7bcf bytes) 000247 1787782538.845034 punch from 0x034048 to 0x0381b3 (0x416c bytes) 000248 1787782538.850381 mapread 0x027084 thru 0x035113 (0xe090 bytes) 000250 1787782538.861958 trunc from 0x007f17 to 0x017d19 (0xfe03 bytes) 000251 1787782538.876156 mapwrite 0x027c8f thru 0x0329c1 (0xad33 bytes) 000252 1787782538.924351 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x967f6e16 (stale) Mirror 9: 0x967f6e16 (stale) Mirror 10: 0x967f6e16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x967f6e16 (stale) Mirror 9: 0x967f6e16 (stale) Mirror 10: 0x967f6e16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000253 1787782539.660035 punch from 0x011cbb to 0x01dc50 (0xbf96 bytes) 000254 1787782539.672016 MIRROR_RESYNC 4 mirrors 000255 1787782541.367998 mapread 0x01e372 thru 0x022c36 (0x48c5 bytes) 000256 1787782541.400115 mapread 0x022552 thru 0x02a7b2 (0x8261 bytes) 000257 1787782541.412672 punch from 0x0364bb to 0x03ffff (0x9b45 bytes) 000258 1787782541.464950 fallocate 0x03edf3 thru 0x03ffff (0x120d bytes) 000259 1787782541.474526 fallocate 0x00c260 thru 0x016be2 (0xa983 bytes) 000260 1787782541.480582 trunc from 0x03081c to 0x03388f (0x3074 bytes) 000261 1787782541.493612 trunc from 0x0048d3 to 0x00b32c (0x6a5a bytes) 000262 1787782541.504480 trunc from 0x040000 to 0x015bea (0x2a415 bytes) 000263 1787782541.536070 trunc from 0x015beb to 0x02fbca (0x19fe0 bytes) 000264 1787782541.554752 mapread 0x00722e thru 0x0126bb (0xb48e bytes) 000265 1787782541.577973 fallocate 0x010116 thru 0x018336 (0x8221 bytes) 000266 1787782541.583786 trunc from 0x02fbcb to 0x001739 (0x2e491 bytes) 000267 1787782541.606008 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x173a): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x173a): Mirror 6: 0xf2e6addf (stale) Mirror 9: 0xf2e6addf (stale) Mirror 10: 0xf2e6addf (stale) CRC-32 checksum value for chunk [0, 0x173a): Mirror 1: 0xf2e6addf Mirror 11: 0xf2e6addf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x173a: skipped 000268 1787782542.606557 read 0x001087 thru 0x001739 (0x6b3 bytes) 000269 1787782542.612568 read 0x000bf6 thru 0x00108b (0x496 bytes) 000270 1787782542.617165 trunc from 0x00173a to 0x03601f (0x348e6 bytes) 000271 1787782542.686915 trunc from 0x036020 to 0x029540 (0xcadf bytes) 000272 1787782542.704996 write 0x0301e3 thru 0x03e71b (0xe539 bytes) 000273 1787782542.707969 trunc from 0x03e71c to 0x0383ab (0x6370 bytes) 000274 1787782542.721563 trunc from 0x0383ac to 0x003f9e (0x3440d bytes) 000275 1787782542.740063 read 0x002a4d thru 0x003f9e (0x1552 bytes) 000277 1787782542.753018 punch from 0x00329d to 0x003734 (0x498 bytes) 000281 1787782542.759479 fallocate 0x02fa69 thru 0x038676 (0x8c0e bytes) 000282 1787782542.790061 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38677): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38677): Mirror 6: 0x5da33911 (stale) Mirror 9: 0x5da33911 (stale) Mirror 10: 0x5da33911 (stale) Mirror 11: 0xfc2191d0 (stale) CRC-32 checksum value for chunk [0, 0x38677): Mirror 1: 0xfc2191d0 Mirror 12: 0xfc2191d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38677: skipped 000284 1787782543.799427 trunc from 0x038677 to 0x03e207 (0x5b91 bytes) 000285 1787782543.874622 mapread 0x02db71 thru 0x038afb (0xaf8b bytes) 000286 1787782543.935495 trunc from 0x00e28e to 0x012a61 (0x47d4 bytes) 000287 1787782543.943557 read 0x00c7bc thru 0x01a3b1 (0xdbf6 bytes) 000288 1787782543.959474 mapread 0x03a208 thru 0x03e207 (0x4000 bytes) 000289 1787782543.960747 trunc from 0x004cf9 to 0x012f23 (0xe22b bytes) 000290 1787782543.974898 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e208): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e208): Mirror 6: 0xfed4fe9e (stale) Mirror 9: 0xfed4fe9e (stale) Mirror 10: 0xfed4fe9e (stale) Mirror 11: 0xdc55b0ea (stale) Mirror 12: 0xdc55b0ea (stale) CRC-32 checksum value for chunk [0, 0x3e208): Mirror 1: 0xdc55b0ea Mirror 13: 0xdc55b0ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e208: skipped 000291 1787782545.224907 read 0x0235ca thru 0x02832c (0x4d63 bytes) 000292 1787782545.242472 read 0x033651 thru 0x033fb8 (0x968 bytes) 000293 1787782545.254122 mapwrite 0x006199 thru 0x00c2ec (0x6154 bytes) 000294 1787782545.373946 read 0x022ab2 thru 0x02e233 (0xb782 bytes) 000295 1787782545.389343 read 0x00cf9a thru 0x00e5ff (0x1666 bytes) 000296 1787782545.390082 mapwrite 0x02d569 thru 0x038d83 (0xb81b bytes) 000297 1787782545.413561 trunc from 0x033912 to 0x03e207 (0xa8f6 bytes) 000298 1787782545.421018 trunc from 0x03e208 to 0x018190 (0x26077 bytes) 000299 1787782545.445524 write 0x0321ac thru 0x03ffff (0xde54 bytes) 000300 1787782545.447330 write 0x008e14 thru 0x014a76 (0xbc63 bytes) 000301 1787782545.450903 trunc from 0x03a8c8 to 0x03ffff (0x5738 bytes) 000302 1787782545.475371 MIRROR_RESYNC 7 mirrors 000304 1787782548.105076 MIRROR_EXTEND to 8 mirrors 000305 1787782549.262624 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) 000307 1787782549.916991 punch from 0x037402 to 0x03c93e (0x553d bytes) 000308 1787782550.003773 punch from 0x009829 to 0x009c04 (0x3dc bytes) 000309 1787782550.012298 mapwrite 0x01d73d thru 0x01f890 (0x2154 bytes) 000310 1787782550.092310 mapwrite 0x02ed77 thru 0x03ec5b (0xfee5 bytes) 000311 1787782550.164371 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x40d2272d (stale) Mirror 9: 0x40d2272d (stale) Mirror 10: 0x40d2272d (stale) Mirror 11: 0x40d2272d (stale) Mirror 12: 0x40d2272d (stale) Mirror 14: 0x40d2272d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96d96ed5 Mirror 15: 0x96d96ed5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1787782551.394117 trunc from 0x040000 to 0x00e609 (0x319f6 bytes) 000313 1787782551.534636 trunc from 0x00e60a to 0x01ca24 (0xe41b bytes) 000314 1787782551.567584 trunc from 0x01ca25 to 0x03b3e5 (0x1e9c1 bytes) 000315 1787782551.593662 trunc from 0x0159d9 to 0x018a74 (0x309c bytes) 000316 1787782551.603990 write 0x00fb20 thru 0x01263c (0x2b1d bytes) 000317 1787782551.618617 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b3e6): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3e6): Mirror 6: 0xcaeb5490 (stale) Mirror 9: 0xcaeb5490 (stale) Mirror 10: 0xcaeb5490 (stale) Mirror 11: 0xcaeb5490 (stale) Mirror 12: 0xcaeb5490 (stale) Mirror 14: 0xcaeb5490 (stale) Mirror 15: 0xe4b3694b (stale) CRC-32 checksum value for chunk [0, 0x3b3e6): Mirror 1: 0xdb6e9890 Mirror 16: 0xdb6e9890 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3e6: skipped 000318 1787782552.981937 fallocate 0x0102e0 thru 0x016327 (0x6048 bytes) 000319 1787782553.070323 trunc from 0x0217fc to 0x02de8c (0xc691 bytes) 000320 1787782553.082851 write 0x01d031 thru 0x029114 (0xc0e4 bytes) 000321 1787782553.114116 trunc from 0x01f7ca to 0x0243b7 (0x4bee bytes) 000322 1787782553.134807 trunc from 0x03b3e6 to 0x01c69e (0x1ed47 bytes) 000323 1787782553.194169 trunc from 0x01c69f to 0x0186ad (0x3ff1 bytes) 000324 1787782553.241635 MIRROR_RESYNC 9 mirrors 000325 1787782556.120095 trunc from 0x006eb4 to 0x012993 (0xbae0 bytes) 000326 1787782556.167599 mapread 0x009600 thru 0x00d4e4 (0x3ee5 bytes) 000327 1787782556.190975 mapwrite 0x01cf63 thru 0x0257e0 (0x887e bytes) 000328 1787782556.249149 read 0x01a1c3 thru 0x01b4bc (0x12fa bytes) 000329 1787782556.250361 mapread 0x003493 thru 0x00e134 (0xaca2 bytes) 000330 1787782556.262051 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: 0x7a1be77c (stale) Mirror 9: 0x7a1be77c (stale) Mirror 10: 0x7a1be77c (stale) Mirror 11: 0x7a1be77c (stale) Mirror 12: 0x7a1be77c (stale) Mirror 14: 0x7a1be77c (stale) Mirror 15: 0x7a1be77c (stale) Mirror 16: 0x7a1be77c (stale) CRC-32 checksum value for chunk [0, 0x257e1): Mirror 1: 0x5d729707 Mirror 17: 0x5d729707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x257e1: skipped 000331 1787782557.628897 read 0x008bb7 thru 0x016ee5 (0xe32f bytes) 000332 1787782557.642692 read 0x016415 thru 0x0197eb (0x33d7 bytes) 000333 1787782557.655161 punch from 0x01adf5 to 0x01b133 (0x33f bytes) 000334 1787782557.708193 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: 0x7a1be77c (stale) Mirror 9: 0x7a1be77c (stale) Mirror 10: 0x7a1be77c (stale) Mirror 11: 0x7a1be77c (stale) Mirror 12: 0x7a1be77c (stale) Mirror 14: 0x7a1be77c (stale) Mirror 15: 0x7a1be77c (stale) Mirror 16: 0x7a1be77c (stale) Mirror 17: 0x5d729707 (stale) CRC-32 checksum value for chunk [0, 0x257e1): Mirror 1: 0x5d729707 Mirror 18: 0x5d729707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x257e1: skipped 000335 1787782558.984053 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x257e1): Mirror 6: 0x7a1be77c (stale) Mirror 9: 0x7a1be77c (stale) Mirror 10: 0x7a1be77c (stale) Mirror 11: 0x7a1be77c (stale) Mirror 12: 0x7a1be77c (stale) Mirror 14: 0x7a1be77c (stale) Mirror 15: 0x7a1be77c (stale) Mirror 16: 0x7a1be77c (stale) Mirror 17: 0x5d729707 (stale) CRC-32 checksum value for chunk [0, 0x257e1): Mirror 1: 0x5d729707 Mirror 18: 0x5d729707 Mirror 19: 0x5d729707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x257e1: skipped 000336 1787782560.492136 trunc from 0x019cfc to 0x02363d (0x9942 bytes) 000337 1787782560.562178 MIRROR_RESYNC 12 mirrors 000338 1787782564.914729 mapwrite 0x026760 thru 0x0308dc (0xa17d bytes) 000339 1787782565.051984 trunc from 0x0308dd to 0x03e7fe (0xdf22 bytes) 000340 1787782565.088973 mapread 0x0178d7 thru 0x020a6a (0x9194 bytes) 000341 1787782565.105625 mapread 0x024edc thru 0x02e2dc (0x9401 bytes) 000342 1787782565.107034 trunc from 0x03e7ff to 0x02fd47 (0xeab7 bytes) 000343 1787782565.133665 read 0x024663 thru 0x02e7fd (0xa19b bytes) 000345 1787782565.142554 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fd48): Mirror 6: 0x38de2fc8 (stale) Mirror 9: 0x38de2fc8 (stale) Mirror 10: 0x38de2fc8 (stale) Mirror 11: 0x38de2fc8 (stale) Mirror 12: 0x38de2fc8 (stale) Mirror 14: 0x38de2fc8 (stale) Mirror 16: 0x38de2fc8 (stale) Mirror 17: 0x38de2fc8 (stale) Mirror 18: 0x38de2fc8 (stale) Mirror 19: 0x38de2fc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd48): Mirror 6: 0x38de2fc8 (stale) Mirror 9: 0x38de2fc8 (stale) Mirror 10: 0x38de2fc8 (stale) Mirror 11: 0x38de2fc8 (stale) Mirror 12: 0x38de2fc8 (stale) Mirror 14: 0x38de2fc8 (stale) Mirror 16: 0x38de2fc8 (stale) Mirror 17: 0x38de2fc8 (stale) Mirror 18: 0x38de2fc8 (stale) Mirror 19: 0x38de2fc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd48: skipped 000346 1787782565.916055 MIRROR_RESYNC 11 mirrors 000347 1787782569.731596 trunc from 0x02fd48 to 0x01546c (0x1a8db bytes) 000348 1787782569.799162 mapwrite 0x020a8e thru 0x02c3a3 (0xb916 bytes) 000350 1787782569.906812 trunc from 0x02c3a4 to 0x00ff25 (0x1c47e bytes) 000352 1787782569.930682 read 0x00aabb thru 0x00ff25 (0x546b bytes) 000353 1787782569.947362 punch from 0x0039ec to 0x00ff25 (0xc53a bytes) 000354 1787782569.953974 trunc from 0x00ff26 to 0x0372c7 (0x273a2 bytes) 000355 1787782569.982795 fallocate 0x014d97 thru 0x01a96f (0x5bd9 bytes) 000356 1787782569.998544 MIRROR_RESYNC 11 mirrors 000357 1787782571.625404 trunc from 0x00254c to 0x008c20 (0x66d5 bytes) 000358 1787782571.705918 trunc from 0x0372c8 to 0x0293d1 (0xdef6 bytes) 000359 1787782571.736013 trunc from 0x00d48b to 0x016e27 (0x999d bytes) 000361 1787782571.747885 mapread 0x00753a thru 0x015a3f (0xe506 bytes) 000362 1787782571.823172 trunc from 0x0293d2 to 0x035ea0 (0xcacf bytes) 000364 1787782571.867661 read 0x02eaae thru 0x02f85e (0xdb1 bytes) 000365 1787782571.876703 mapwrite 0x025a68 thru 0x02a047 (0x45e0 bytes) 000367 1787782571.926874 fallocate 0x01319b thru 0x017e43 (0x4ca9 bytes) 000368 1787782571.933884 fallocate 0x00cca1 thru 0x0160c3 (0x9423 bytes) 000369 1787782571.941944 trunc from 0x035ea1 to 0x020ec4 (0x14fdc bytes) 000372 1787782571.964367 punch from 0x01b274 to 0x020ec4 (0x5c51 bytes) 000373 1787782571.973312 trunc from 0x020ec5 to 0x017c80 (0x9244 bytes) 000374 1787782571.996009 mapread 0x001a72 thru 0x00f006 (0xd595 bytes) 000375 1787782572.002747 write 0x02ffca thru 0x03408e (0x40c5 bytes) 000376 1787782572.004651 write 0x02004f thru 0x02db0e (0xdac0 bytes) 000377 1787782572.015642 trunc from 0x03408f to 0x034d0a (0xc7c bytes) 000378 1787782572.033898 trunc from 0x02e123 to 0x034d0a (0x6be8 bytes) 000379 1787782572.063877 write 0x00713f thru 0x011f84 (0xae46 bytes) 000380 1787782572.069037 fallocate 0x003db2 thru 0x009f03 (0x6152 bytes) 000381 1787782572.076086 mapwrite 0x03e05a thru 0x03ffff (0x1fa6 bytes) 000382 1787782572.148291 fallocate 0x0261a8 thru 0x02f78f (0x95e8 bytes) 000383 1787782572.156501 fallocate 0x008750 thru 0x016266 (0xdb17 bytes) 000384 1787782572.167579 trunc from 0x040000 to 0x031040 (0xefbf bytes) 000385 1787782572.198978 mapread 0x02bad4 thru 0x02d823 (0x1d50 bytes) 000386 1787782572.201042 trunc from 0x031041 to 0x03619d (0x515d bytes) 000387 1787782572.233723 write 0x02a571 thru 0x035709 (0xb199 bytes) 000388 1787782572.243012 mapread 0x00525b thru 0x00f33a (0xa0e0 bytes) 000389 1787782572.246476 mapread 0x00bdcc thru 0x0132c1 (0x74f6 bytes) 000390 1787782572.249993 write 0x00b02d thru 0x011f5d (0x6f31 bytes) 000391 1787782572.252728 trunc from 0x03619e to 0x034aa5 (0x16f8 bytes) 000392 1787782572.288610 fallocate 0x00178c thru 0x006894 (0x5109 bytes) 000394 1787782572.306385 mapread 0x0197bf thru 0x01de42 (0x4684 bytes) 000395 1787782572.341745 trunc from 0x0034f3 to 0x0113e6 (0xdef4 bytes) 000397 1787782572.384359 read 0x03447d thru 0x034aa5 (0x629 bytes) 000398 1787782572.392486 trunc from 0x01d894 to 0x022a2b (0x5198 bytes) 000399 1787782572.409051 trunc from 0x00311f to 0x0052f7 (0x21d9 bytes) 000400 1787782572.414866 fallocate 0x00fdf1 thru 0x019e51 (0xa061 bytes) 000401 1787782572.420991 read 0x011acb thru 0x01e92a (0xce60 bytes) 000402 1787782572.439725 read 0x0092da thru 0x00ba90 (0x27b7 bytes) 000403 1787782572.440650 trunc from 0x034aa6 to 0x007e7d (0x2cc28 bytes) 000404 1787782572.460702 trunc from 0x004aa9 to 0x007e7d (0x33d5 bytes) 000405 1787782572.466272 mapwrite 0x000dc6 thru 0x008056 (0x7291 bytes) 000407 1787782572.512969 punch from 0x006549 to 0x008056 (0x1b0e bytes) 000408 1787782572.519961 write 0x006ffc thru 0x0077db (0x7e0 bytes) 000409 1787782572.539760 MIRROR_RESYNC 11 mirrors 000411 1787782573.956982 punch from 0x006c64 to 0x008056 (0x13f3 bytes) 000412 1787782574.012785 mapread 0x0048ec thru 0x004b9e (0x2b3 bytes) 000413 1787782574.024594 MIRROR_RESYNC 11 mirrors 000414 1787782576.170636 trunc from 0x000f30 to 0x008056 (0x7127 bytes) 000415 1787782576.257861 mapread 0x00522a thru 0x008056 (0x2e2d bytes) 000416 1787782576.326466 trunc from 0x008057 to 0x031cc8 (0x29c72 bytes) 000417 1787782576.421439 mapread 0x019692 thru 0x020404 (0x6d73 bytes) 000419 1787782576.501727 trunc from 0x013ea4 to 0x017c0e (0x3d6b bytes) 000420 1787782576.522830 mapwrite 0x0270f9 thru 0x032b78 (0xba80 bytes) 000421 1787782576.603461 punch from 0x002121 to 0x00dfd3 (0xbeb3 bytes) 000422 1787782576.611180 fallocate 0x035b53 thru 0x03d290 (0x773e bytes) 000423 1787782576.622173 fallocate 0x038ba7 thru 0x03ffff (0x7459 bytes) 000424 1787782576.630339 mapread 0x0048c7 thru 0x00fd1b (0xb455 bytes) 000425 1787782576.674009 fallocate 0x00e90b thru 0x01b5c2 (0xccb8 bytes) 000426 1787782576.693297 punch from 0x0341e5 to 0x03ae0a (0x6c26 bytes) 000428 1787782576.702560 trunc from 0x038551 to 0x0386a1 (0x151 bytes) 000429 1787782576.709610 fallocate 0x014316 thru 0x0207df (0xc4ca bytes) 000430 1787782576.717278 write 0x01c236 thru 0x02b42d (0xf1f8 bytes) 000431 1787782576.719888 trunc from 0x040000 to 0x00237f (0x3dc80 bytes) 000432 1787782576.745545 mapwrite 0x03364f thru 0x038564 (0x4f16 bytes) 000433 1787782576.803890 fallocate 0x01f499 thru 0x020bf5 (0x175d bytes) 000435 1787782576.811388 trunc from 0x015e8e to 0x01755a (0x16cd bytes) 000436 1787782576.817461 trunc from 0x03442e to 0x038564 (0x4137 bytes) 000437 1787782576.823682 write 0x015e1d thru 0x01be4b (0x602f bytes) 000438 1787782576.841713 write 0x005e5a thru 0x00c0ff (0x62a6 bytes) 000440 1787782576.855156 read 0x009559 thru 0x018874 (0xf31c bytes) 000441 1787782576.869335 trunc from 0x038565 to 0x0141eb (0x24379 bytes) 000442 1787782576.903274 punch from 0x003e8b to 0x00cfd8 (0x914e bytes) 000444 1787782576.926732 mapread 0x00ed38 thru 0x0141eb (0x54b4 bytes) 000445 1787782576.933604 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x141ec): Mirror 9: 0x42d1da8 (stale) Mirror 10: 0x42d1da8 (stale) Mirror 11: 0x42d1da8 (stale) Mirror 12: 0x42d1da8 (stale) Mirror 14: 0x42d1da8 (stale) Mirror 16: 0x42d1da8 (stale) Mirror 17: 0x42d1da8 (stale) Mirror 18: 0x42d1da8 (stale) Mirror 19: 0x42d1da8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141ec): Mirror 9: 0x42d1da8 (stale) Mirror 10: 0x42d1da8 (stale) Mirror 11: 0x42d1da8 (stale) Mirror 12: 0x42d1da8 (stale) Mirror 14: 0x42d1da8 (stale) Mirror 16: 0x42d1da8 (stale) Mirror 17: 0x42d1da8 (stale) Mirror 18: 0x42d1da8 (stale) Mirror 19: 0x42d1da8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141ec: skipped 000446 1787782577.638173 mapread 0x00675f thru 0x00dfac (0x784e bytes) 000447 1787782577.671599 trunc from 0x0141ec to 0x03e398 (0x2a1ad bytes) 000448 1787782577.716566 read 0x0292c7 thru 0x02fa29 (0x6763 bytes) 000449 1787782577.730620 punch from 0x02f11a to 0x03404e (0x4f35 bytes) 000450 1787782577.737342 fallocate 0x03ed97 thru 0x03ffff (0x1269 bytes) 000452 1787782577.742684 punch from 0x012caa to 0x01ba67 (0x8dbe bytes) 000453 1787782577.751541 write 0x01763e thru 0x01bc56 (0x4619 bytes) 000454 1787782577.776100 write 0x02f75c thru 0x03d85c (0xe101 bytes) 000455 1787782577.794735 write 0x03ebfe thru 0x03ffff (0x1402 bytes) 000456 1787782577.802760 fallocate 0x01c3ec thru 0x028b9a (0xc7af bytes) 000458 1787782577.813176 read 0x038d87 thru 0x03ffff (0x7279 bytes) 000459 1787782577.815996 trunc from 0x040000 to 0x03c6bc (0x3943 bytes) 000460 1787782577.852132 read 0x033f70 thru 0x037db4 (0x3e45 bytes) 000461 1787782577.853437 trunc from 0x036c59 to 0x03c6bc (0x5a64 bytes) 000462 1787782577.913853 fallocate 0x03153f thru 0x037022 (0x5ae4 bytes) 000463 1787782577.930828 punch from 0x02a2c5 to 0x032591 (0x82cd bytes) 000464 1787782577.943119 fallocate 0x027e5b thru 0x031eea (0xa090 bytes) 000465 1787782577.955711 trunc from 0x00ead9 to 0x016813 (0x7d3b bytes) 000466 1787782577.966415 read 0x038da9 thru 0x038ee4 (0x13c bytes) 000468 1787782577.995740 write 0x0213ea thru 0x02d5f1 (0xc208 bytes) 000469 1787782578.007303 write 0x00d007 thru 0x017039 (0xa033 bytes) 000470 1787782578.019760 fallocate 0x01ad8d thru 0x02081b (0x5a8f bytes) 000471 1787782578.039971 mapwrite 0x03558e thru 0x03aaeb (0x555e bytes) 000473 1787782578.080913 trunc from 0x03c6bd to 0x01ff28 (0x1c794 bytes) 000474 1787782578.099148 mapwrite 0x0084d4 thru 0x01809f (0xfbcc bytes) 000476 1787782578.131490 trunc from 0x01ff29 to 0x015737 (0xa7f1 bytes) 000477 1787782578.152802 trunc from 0x015738 to 0x018da4 (0x366d bytes) 000478 1787782578.169888 read 0x003441 thru 0x0037cc (0x38c bytes) 000479 1787782578.170639 punch from 0x00ef86 to 0x017612 (0x868d bytes) 000480 1787782578.177274 fallocate 0x02dbcb thru 0x03d5d8 (0xfa0e bytes) 000481 1787782578.185993 trunc from 0x004891 to 0x01470a (0xfe7a bytes) 000483 1787782578.199745 trunc from 0x018da5 to 0x015baf (0x31f5 bytes) 000485 1787782578.230663 write 0x008e10 thru 0x0140ea (0xb2db bytes) 000487 1787782578.254744 mapwrite 0x0071e9 thru 0x00a4fc (0x3314 bytes) 000488 1787782578.301594 mapwrite 0x02184e thru 0x027b0d (0x62c0 bytes) 000489 1787782578.442180 trunc from 0x0226ec to 0x027b0d (0x5422 bytes) 000490 1787782578.452715 write 0x009237 thru 0x012f0f (0x9cd9 bytes) 000491 1787782578.455995 fallocate 0x02da55 thru 0x039db8 (0xc364 bytes) 000492 1787782578.465727 fallocate 0x024aca thru 0x02e240 (0x9777 bytes) 000493 1787782578.475923 mapwrite 0x011adb thru 0x01e7b4 (0xccda bytes) 000494 1787782578.519716 mapread 0x02528e thru 0x02d994 (0x8707 bytes) 000495 1787782578.553749 mapread 0x014f14 thru 0x015039 (0x126 bytes) 000496 1787782578.557941 trunc from 0x039db9 to 0x0046d1 (0x356e7 bytes) 000498 1787782578.590114 punch from 0x002855 to 0x0046d1 (0x1e7d bytes) 000499 1787782578.599121 trunc from 0x001d9b to 0x0046d1 (0x2937 bytes) 000500 1787782578.609126 write 0x00043c thru 0x0103eb (0xffb0 bytes) 000501 1787782578.612968 punch from 0x00726b to 0x00f85b (0x85f1 bytes) 000502 1787782578.657288 trunc from 0x0060d7 to 0x00b033 (0x4f5d bytes) 000503 1787782578.672814 mapread 0x000e5d thru 0x00532f (0x44d3 bytes) 000504 1787782578.678891 read 0x004fc0 thru 0x0103eb (0xb42c bytes) 000505 1787782578.700467 trunc from 0x0103ec to 0x00cd02 (0x36e9 bytes) 000507 1787782578.763280 read 0x00c207 thru 0x00cd02 (0xafc bytes) 000508 1787782578.766690 read 0x009e57 thru 0x00c853 (0x29fd bytes) 000509 1787782578.771197 punch from 0x0060aa to 0x00996f (0x38c6 bytes) 000510 1787782578.780027 punch from 0x00132a to 0x00ccd9 (0xb9b0 bytes) 000512 1787782578.794930 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xcd03): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcd03): Mirror 9: 0x950ed71a (stale) Mirror 10: 0x950ed71a (stale) Mirror 11: 0x950ed71a (stale) Mirror 12: 0x950ed71a (stale) Mirror 14: 0x950ed71a (stale) Mirror 16: 0x950ed71a (stale) Mirror 17: 0x950ed71a (stale) Mirror 18: 0x950ed71a (stale) Mirror 19: 0x950ed71a (stale) CRC-32 checksum value for chunk [0, 0xcd03): Mirror 1: 0x670d19fb Mirror 20: 0x670d19fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcd03: skipped 000514 1787782580.306877 mapread 0x00b81e thru 0x00bcda (0x4bd bytes) 000515 1787782580.325272 trunc from 0x00cd03 to 0x003f73 (0x8d8f bytes) 000517 1787782580.415667 mapwrite 0x03aacb thru 0x03ffff (0x5535 bytes) 000518 1787782580.477626 write 0x03561e thru 0x03704f (0x1a32 bytes) 000519 1787782580.479007 trunc from 0x01a527 to 0x01eae7 (0x45c1 bytes) 000520 1787782580.486144 punch from 0x02f3d0 to 0x030397 (0xfc8 bytes) 000522 1787782580.491847 read 0x035a97 thru 0x03ffff (0xa569 bytes) 000523 1787782580.492885 trunc from 0x00d932 to 0x01ba2a (0xe0f9 bytes) 000526 1787782580.500181 write 0x01f365 thru 0x0260c1 (0x6d5d bytes) 000527 1787782580.511650 mapread 0x01d8b7 thru 0x029595 (0xbcdf bytes) 000528 1787782580.545350 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2e2c2d94 (stale) Mirror 10: 0x2e2c2d94 (stale) Mirror 11: 0x2e2c2d94 (stale) Mirror 12: 0x2e2c2d94 (stale) Mirror 14: 0x2e2c2d94 (stale) Mirror 16: 0x2e2c2d94 (stale) Mirror 17: 0x2e2c2d94 (stale) Mirror 18: 0x2e2c2d94 (stale) Mirror 19: 0x2e2c2d94 (stale) Mirror 20: 0x4d400143 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68a9b2e1 Mirror 21: 0x68a9b2e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000529 1787782582.092085 punch from 0x00427d to 0x01065a (0xc3de bytes) 000530 1787782582.155814 fallocate 0x0269b9 thru 0x03255b (0xbba3 bytes) 000531 1787782582.162817 trunc from 0x040000 to 0x032e62 (0xd19d bytes) 000532 1787782582.199869 trunc from 0x00eb30 to 0x01832c (0x97fd bytes) 000533 1787782582.207418 write 0x009797 thru 0x0146f1 (0xaf5b bytes) 000534 1787782582.221180 trunc from 0x0248ae to 0x031bfc (0xd34f bytes) 000535 1787782582.234653 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x32e63): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e63): Mirror 9: 0x718e084b (stale) Mirror 10: 0x718e084b (stale) Mirror 11: 0x718e084b (stale) Mirror 12: 0x718e084b (stale) Mirror 14: 0x718e084b (stale) Mirror 16: 0x718e084b (stale) Mirror 17: 0x718e084b (stale) Mirror 18: 0x718e084b (stale) Mirror 19: 0x718e084b (stale) Mirror 20: 0x1612d0ba (stale) Mirror 21: 0x44d69c9a (stale) CRC-32 checksum value for chunk [0, 0x32e63): Mirror 1: 0xfe448981 Mirror 22: 0xfe448981 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e63: skipped 000536 1787782583.882292 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x32e63): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e63): Mirror 9: 0x718e084b (stale) Mirror 10: 0x718e084b (stale) Mirror 11: 0x718e084b (stale) Mirror 12: 0x718e084b (stale) Mirror 14: 0x718e084b (stale) Mirror 16: 0x718e084b (stale) Mirror 17: 0x718e084b (stale) Mirror 18: 0x718e084b (stale) Mirror 19: 0x718e084b (stale) Mirror 20: 0x1612d0ba (stale) Mirror 21: 0x44d69c9a (stale) CRC-32 checksum value for chunk [0, 0x32e63): Mirror 1: 0xfe448981 Mirror 22: 0xfe448981 Mirror 23: 0xfe448981 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e63: skipped 000537 1787782585.332040 MIRROR_RESYNC 14 mirrors 000538 1787782588.739238 fallocate 0x0102dc thru 0x01ec13 (0xe938 bytes) 000539 1787782588.792896 fallocate 0x01266c thru 0x01e216 (0xbbab bytes) 000540 1787782588.806034 mapwrite 0x03bdca thru 0x03ea31 (0x2c68 bytes) 000541 1787782588.880059 mapread 0x03a154 thru 0x03ea31 (0x48de bytes) 000542 1787782588.881564 read 0x03c11f thru 0x03ea31 (0x2913 bytes) 000543 1787782588.887560 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 10: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 17: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 10: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 17: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea32: skipped 000544 1787782589.857510 fallocate 0x00a189 thru 0x00f80f (0x5687 bytes) 000546 1787782589.873301 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 10: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 10: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea32: skipped 000547 1787782590.611973 fallocate 0x0283bc thru 0x03626e (0xdeb3 bytes) 000548 1787782590.618964 punch from 0x0146c5 to 0x0244a8 (0xfde4 bytes) 000549 1787782590.625549 fallocate 0x017443 thru 0x024120 (0xccde bytes) 000550 1787782590.631651 punch from 0x03ca56 to 0x03ea31 (0x1fdc bytes) 000551 1787782590.637986 mapread 0x00a11b thru 0x017ad6 (0xd9bc bytes) 000552 1787782590.668384 write 0x021061 thru 0x0223bf (0x135f bytes) 000553 1787782590.681518 fallocate 0x016928 thru 0x01b4eb (0x4bc4 bytes) 000554 1787782590.687470 mapwrite 0x024466 thru 0x02f9aa (0xb545 bytes) 000555 1787782590.752069 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 10: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 1: 0xe69f4b57 Mirror 24: 0xe69f4b57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea32: skipped 000556 1787782592.252311 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 12: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 1: 0xe69f4b57 Mirror 24: 0xe69f4b57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea32: skipped 000557 1787782593.439324 fallocate 0x01826e thru 0x01a8e0 (0x2673 bytes) 000558 1787782593.491369 mapread 0x03123c thru 0x03bd4e (0xab13 bytes) 000559 1787782593.510386 trunc from 0x006b24 to 0x01687e (0xfd5b bytes) 000560 1787782593.516108 trunc from 0x033fd6 to 0x03c643 (0x866e bytes) 000561 1787782593.522052 fallocate 0x02bd10 thru 0x02c878 (0xb69 bytes) 000562 1787782593.536325 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) Mirror 24: 0xe69f4b57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea32): Mirror 9: 0x76ede8c (stale) Mirror 14: 0x76ede8c (stale) Mirror 16: 0x76ede8c (stale) Mirror 18: 0x76ede8c (stale) Mirror 19: 0x76ede8c (stale) Mirror 20: 0x76ede8c (stale) Mirror 21: 0x76ede8c (stale) Mirror 22: 0x76ede8c (stale) Mirror 23: 0x76ede8c (stale) Mirror 24: 0xe69f4b57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea32: skipped 000563 1787782594.389369 write 0x00244f thru 0x00f7e4 (0xd396 bytes) 000564 1787782594.408069 write 0x024c3a thru 0x026dcf (0x2196 bytes) 000565 1787782594.419612 trunc from 0x02a207 to 0x02e2f3 (0x40ed bytes) 000566 1787782594.429468 trunc from 0x03ea32 to 0x00d2b8 (0x31779 bytes) 000567 1787782594.468461 MIRROR_RESYNC 11 mirrors 000568 1787782596.279403 trunc from 0x006dee to 0x00d2b8 (0x64cb bytes) 000569 1787782596.326943 read 0x00baba thru 0x00d2b8 (0x17ff bytes) 000570 1787782596.334258 fallocate 0x01e17c thru 0x02a668 (0xc4ed bytes) 000571 1787782596.339676 trunc from 0x00d2b9 to 0x0348d5 (0x2761d bytes) 000573 1787782596.364174 trunc from 0x0348d6 to 0x0085bd (0x2c318 bytes) 000574 1787782596.385702 punch from 0x008460 to 0x0085bd (0x15e bytes) 000575 1787782596.391136 mapread 0x005652 thru 0x0085bd (0x2f6c bytes) 000576 1787782596.411789 write 0x00de65 thru 0x01a557 (0xc6f3 bytes) 000577 1787782596.417821 MIRROR_RESYNC 11 mirrors 000578 1787782598.386629 fallocate 0x02b829 thru 0x02fe94 (0x466c bytes) 000579 1787782598.460282 mapwrite 0x02db3f thru 0x03879b (0xac5d bytes) 000580 1787782598.547441 fallocate 0x01d93e thru 0x0219a1 (0x4064 bytes) 000581 1787782598.553244 trunc from 0x032b84 to 0x03879b (0x5c18 bytes) 000582 1787782598.567502 mapread 0x018bfc thru 0x020893 (0x7c98 bytes) 000583 1787782598.585715 mapread 0x0361c7 thru 0x03879b (0x25d5 bytes) 000586 1787782598.597965 trunc from 0x028616 to 0x03002a (0x7a15 bytes) 000587 1787782598.611273 mapread 0x00113e thru 0x0083e7 (0x72aa bytes) 000588 1787782598.626166 trunc from 0x02b5db to 0x03209b (0x6ac1 bytes) 000589 1787782598.633588 mapwrite 0x01b31a thru 0x02089b (0x5582 bytes) 000590 1787782598.655730 trunc from 0x03879c to 0x039028 (0x88d bytes) 000591 1787782598.695822 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x39029): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39029): Mirror 9: 0xfcaf5515 (stale) Mirror 14: 0xfcaf5515 (stale) Mirror 16: 0xfcaf5515 (stale) Mirror 18: 0xfcaf5515 (stale) Mirror 19: 0xfcaf5515 (stale) Mirror 20: 0xfcaf5515 (stale) Mirror 21: 0xfcaf5515 (stale) Mirror 22: 0xfcaf5515 (stale) Mirror 23: 0xfcaf5515 (stale) Mirror 24: 0xfcaf5515 (stale) CRC-32 checksum value for chunk [0, 0x39029): Mirror 1: 0xad2cb5e7 Mirror 25: 0xad2cb5e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39029: skipped 000593 1787782600.086786 punch from 0x015ee7 to 0x01ef90 (0x90aa bytes) 000594 1787782600.137898 punch from 0x003457 to 0x005b65 (0x270f bytes) 000595 1787782600.144430 punch from 0x00ce2e to 0x011dad (0x4f80 bytes) 000597 1787782600.151702 trunc from 0x039029 to 0x03ad93 (0x1d6b bytes) 000598 1787782600.198519 fallocate 0x02613b thru 0x031282 (0xb148 bytes) 000599 1787782600.204876 trunc from 0x01cb6b to 0x01cded (0x283 bytes) 000600 1787782600.211042 punch from 0x00cd41 to 0x01330a (0x65ca bytes) 000601 1787782600.216561 punch from 0x02d971 to 0x034047 (0x66d7 bytes) 000602 1787782600.222087 read 0x023dbd thru 0x024657 (0x89b bytes) 000603 1787782600.227928 read 0x00be1d thru 0x019223 (0xd407 bytes) 000604 1787782600.239372 mapwrite 0x0174db thru 0x02553f (0xe065 bytes) 000605 1787782600.266322 MIRROR_RESYNC 12 mirrors 000606 1787782605.310622 trunc from 0x035405 to 0x03ad93 (0x598f bytes) 000607 1787782605.370309 punch from 0x019508 to 0x01b60c (0x2105 bytes) 000608 1787782605.377629 punch from 0x02111e to 0x029c54 (0x8b37 bytes) 000609 1787782605.383104 mapwrite 0x00a181 thru 0x0184b7 (0xe337 bytes) 000610 1787782605.441864 MIRROR_RESYNC 12 mirrors 000611 1787782609.247970 read 0x02d080 thru 0x02ea5d (0x19de bytes) 000612 1787782609.259562 mapread 0x00d934 thru 0x00eebd (0x158a bytes) 000613 1787782609.272171 trunc from 0x03ad94 to 0x00b913 (0x2f480 bytes) 000614 1787782609.386905 trunc from 0x001d76 to 0x00b913 (0x9b9e bytes) 000615 1787782609.397558 mapread 0x002393 thru 0x002891 (0x4ff bytes) 000616 1787782609.426346 mapwrite 0x01a533 thru 0x029f21 (0xf9ef bytes) 000617 1787782609.573264 fallocate 0x016410 thru 0x020952 (0xa543 bytes) 000618 1787782609.597203 punch from 0x026655 to 0x029356 (0x2d02 bytes) 000619 1787782609.622361 mapwrite 0x02e525 thru 0x032d98 (0x4874 bytes) 000620 1787782609.724048 fallocate 0x032fbe thru 0x03b514 (0x8557 bytes) 000622 1787782609.731973 fallocate 0x02a519 thru 0x031a35 (0x751d bytes) 000623 1787782609.743380 punch from 0x032c6e to 0x03b514 (0x88a7 bytes) 000624 1787782609.765293 trunc from 0x03b515 to 0x023033 (0x184e1 bytes) 000625 1787782609.810516 fallocate 0x00d308 thru 0x01b28c (0xdf85 bytes) 000626 1787782609.832803 fallocate 0x01ca6b thru 0x0206da (0x3c70 bytes) 000627 1787782609.848634 fallocate 0x032c50 thru 0x033e33 (0x11e4 bytes) 000628 1787782609.889477 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xd3563bfd (stale) Mirror 14: 0xd3563bfd (stale) Mirror 16: 0xd3563bfd (stale) Mirror 18: 0xd3563bfd (stale) Mirror 19: 0xd3563bfd (stale) Mirror 20: 0xd3563bfd (stale) Mirror 22: 0xd3563bfd (stale) Mirror 23: 0xd3563bfd (stale) Mirror 24: 0xd3563bfd (stale) Mirror 25: 0xd3563bfd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xd3563bfd (stale) Mirror 14: 0xd3563bfd (stale) Mirror 16: 0xd3563bfd (stale) Mirror 18: 0xd3563bfd (stale) Mirror 19: 0xd3563bfd (stale) Mirror 20: 0xd3563bfd (stale) Mirror 22: 0xd3563bfd (stale) Mirror 23: 0xd3563bfd (stale) Mirror 24: 0xd3563bfd (stale) Mirror 25: 0xd3563bfd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e34: skipped 000629 1787782610.796777 mapwrite 0x0246b9 thru 0x030404 (0xbd4c bytes) 000630 1787782610.897508 punch from 0x003234 to 0x00acb1 (0x7a7e bytes) 000631 1787782610.907946 read 0x02acc0 thru 0x033e33 (0x9174 bytes) 000632 1787782610.918091 MIRROR_RESYNC 11 mirrors 000633 1787782614.679698 mapwrite 0x019a10 thru 0x01e2df (0x48d0 bytes) 000634 1787782614.800530 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xf29be841 (stale) Mirror 14: 0xf29be841 (stale) Mirror 18: 0xf29be841 (stale) Mirror 19: 0xf29be841 (stale) Mirror 20: 0xf29be841 (stale) Mirror 22: 0xf29be841 (stale) Mirror 23: 0xf29be841 (stale) Mirror 24: 0xf29be841 (stale) Mirror 25: 0xf29be841 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xf29be841 (stale) Mirror 14: 0xf29be841 (stale) Mirror 18: 0xf29be841 (stale) Mirror 19: 0xf29be841 (stale) Mirror 20: 0xf29be841 (stale) Mirror 22: 0xf29be841 (stale) Mirror 23: 0xf29be841 (stale) Mirror 24: 0xf29be841 (stale) Mirror 25: 0xf29be841 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e34: skipped 000635 1787782615.570673 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xf29be841 (stale) Mirror 14: 0xf29be841 (stale) Mirror 18: 0xf29be841 (stale) Mirror 19: 0xf29be841 (stale) Mirror 20: 0xf29be841 (stale) Mirror 22: 0xf29be841 (stale) Mirror 23: 0xf29be841 (stale) Mirror 24: 0xf29be841 (stale) Mirror 25: 0xf29be841 (stale) CRC-32 checksum value for chunk [0, 0x33e34): Mirror 1: 0xc1d52acd Mirror 26: 0xc1d52acd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e34: skipped 000636 1787782616.693134 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xf29be841 (stale) Mirror 14: 0xf29be841 (stale) Mirror 19: 0xf29be841 (stale) Mirror 20: 0xf29be841 (stale) Mirror 22: 0xf29be841 (stale) Mirror 23: 0xf29be841 (stale) Mirror 24: 0xf29be841 (stale) Mirror 25: 0xf29be841 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e34): Mirror 9: 0xf29be841 (stale) Mirror 14: 0xf29be841 (stale) Mirror 19: 0xf29be841 (stale) Mirror 20: 0xf29be841 (stale) Mirror 22: 0xf29be841 (stale) Mirror 23: 0xf29be841 (stale) Mirror 24: 0xf29be841 (stale) Mirror 25: 0xf29be841 (stale) CRC-32 checksum value for chunk [0, 0x33e34): Mirror 1: 0xc1d52acd Mirror 26: 0xc1d52acd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e34: skipped 000637 1787782617.639571 fallocate 0x0012d1 thru 0x009ab5 (0x87e5 bytes) 000638 1787782617.712913 punch from 0x008ea5 to 0x012abf (0x9c1b bytes) 000639 1787782617.721688 fallocate 0x003782 thru 0x00603a (0x28b9 bytes) 000640 1787782617.729399 fallocate 0x018b53 thru 0x019c88 (0x1136 bytes) 000641 1787782617.735520 mapwrite 0x032708 thru 0x03ea95 (0xc38e bytes) 000642 1787782617.827726 mapread 0x033b01 thru 0x03dab7 (0x9fb7 bytes) 000643 1787782617.831513 mapread 0x0198e8 thru 0x025796 (0xbeaf bytes) 000644 1787782617.868345 MIRROR_RESYNC 10 mirrors 000645 1787782620.245424 mapwrite 0x036ca4 thru 0x03ffff (0x935c bytes) 000646 1787782620.406024 write 0x009819 thru 0x011233 (0x7a1b bytes) 000647 1787782620.425349 trunc from 0x040000 to 0x0242e3 (0x1bd1c bytes) 000648 1787782620.456754 mapread 0x01f49a thru 0x0242e3 (0x4e4a bytes) 000649 1787782620.470203 trunc from 0x009e5e to 0x00f9b1 (0x5b54 bytes) 000651 1787782620.492562 write 0x03d68c thru 0x03ffff (0x2974 bytes) 000652 1787782620.496103 punch from 0x0287db to 0x030c40 (0x8466 bytes) 000655 1787782620.503795 trunc from 0x013078 to 0x0226bb (0xf644 bytes) 000656 1787782620.510543 mapwrite 0x01c8b9 thru 0x01e1ba (0x1902 bytes) 000657 1787782620.548803 read 0x03cc32 thru 0x03e4b6 (0x1885 bytes) 000658 1787782620.556133 fallocate 0x00e6a0 thru 0x01e5dc (0xff3d bytes) 000659 1787782620.563988 trunc from 0x00cc34 to 0x0167c9 (0x9b96 bytes) 000661 1787782620.578565 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1a02513f (stale) Mirror 14: 0x1a02513f (stale) Mirror 19: 0x1a02513f (stale) Mirror 20: 0x1a02513f (stale) Mirror 22: 0x1a02513f (stale) Mirror 23: 0x1a02513f (stale) Mirror 24: 0x1a02513f (stale) Mirror 25: 0x1a02513f (stale) Mirror 26: 0x1a02513f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe69dcbdc Mirror 27: 0xe69dcbdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000663 1787782622.129635 read 0x02d3b5 thru 0x03c074 (0xecc0 bytes) 000664 1787782622.151429 trunc from 0x03c7f6 to 0x03ffff (0x380a bytes) 000665 1787782622.203425 trunc from 0x040000 to 0x024cdd (0x1b322 bytes) 000666 1787782622.239962 punch from 0x017d60 to 0x020e44 (0x90e5 bytes) 000667 1787782622.248742 punch from 0x0079ac to 0x007abe (0x113 bytes) 000668 1787782622.254262 fallocate 0x033b2a thru 0x037a0f (0x3ee6 bytes) 000669 1787782622.260883 write 0x03d8d8 thru 0x03ffff (0x2728 bytes) 000670 1787782622.271432 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1a02513f (stale) Mirror 14: 0x1a02513f (stale) Mirror 19: 0x1a02513f (stale) Mirror 20: 0x1a02513f (stale) Mirror 22: 0x1a02513f (stale) Mirror 23: 0x1a02513f (stale) Mirror 24: 0x1a02513f (stale) Mirror 25: 0x1a02513f (stale) Mirror 26: 0x1a02513f (stale) Mirror 27: 0xe69dcbdc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x165249e6 Mirror 28: 0x165249e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000671 1787782623.741830 punch from 0x019cca to 0x021abe (0x7df5 bytes) 000672 1787782623.794870 fallocate 0x0271b7 thru 0x027436 (0x280 bytes) 000673 1787782623.801408 punch from 0x034f46 to 0x038cbb (0x3d76 bytes) 000674 1787782623.807107 mapwrite 0x00f6fb thru 0x017c74 (0x857a bytes) 000675 1787782623.861763 punch from 0x01ab62 to 0x01dedc (0x337b bytes) 000676 1787782623.869014 write 0x019e9e thru 0x01eadd (0x4c40 bytes) 000677 1787782623.896882 write 0x021369 thru 0x03051e (0xf1b6 bytes) 000678 1787782623.929410 read 0x02151b thru 0x02a8b1 (0x9397 bytes) 000679 1787782623.930022 trunc from 0x040000 to 0x015806 (0x2a7f9 bytes) 000680 1787782623.973013 read 0x00c8bb thru 0x00d1f5 (0x93b bytes) 000681 1787782623.976795 write 0x01efb8 thru 0x01f85f (0x8a8 bytes) 000682 1787782623.980488 trunc from 0x01f860 to 0x0050b8 (0x1a7a7 bytes) 000683 1787782624.002879 trunc from 0x0035dd to 0x0050b8 (0x1adc bytes) 000684 1787782624.027430 read 0x004aaa thru 0x0050b8 (0x60f bytes) 000685 1787782624.060604 write 0x028d8b thru 0x029247 (0x4bd bytes) 000686 1787782624.061692 fallocate 0x034c39 thru 0x03bb15 (0x6edd bytes) 000687 1787782624.080028 punch from 0x02f9bd to 0x02fe98 (0x4dc bytes) 000688 1787782624.103110 mapread 0x02594e thru 0x028d37 (0x33ea bytes) 000689 1787782624.149532 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 9: 0xeae1f52d (stale) Mirror 14: 0xeae1f52d (stale) Mirror 19: 0xeae1f52d (stale) Mirror 20: 0xeae1f52d (stale) Mirror 22: 0xeae1f52d (stale) Mirror 23: 0xeae1f52d (stale) Mirror 24: 0xeae1f52d (stale) Mirror 25: 0xeae1f52d (stale) Mirror 26: 0xeae1f52d (stale) Mirror 27: 0x91cbc92c (stale) Mirror 28: 0x88256bc4 (stale) CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 1: 0x85a9f147 Mirror 29: 0x85a9f147 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb16: skipped 000690 1787782625.497772 fallocate 0x032a57 thru 0x03a7e7 (0x7d91 bytes) 000691 1787782625.610475 MIRROR_RESYNC 13 mirrors 000692 1787782628.746945 punch from 0x033dea to 0x03bb15 (0x7d2c bytes) 000693 1787782628.799813 read 0x025e04 thru 0x02eb8b (0x8d88 bytes) 000694 1787782628.813884 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 9: 0x85a9f147 (stale) Mirror 14: 0x85a9f147 (stale) Mirror 19: 0x85a9f147 (stale) Mirror 22: 0x85a9f147 (stale) Mirror 23: 0x85a9f147 (stale) Mirror 24: 0x85a9f147 (stale) Mirror 25: 0x85a9f147 (stale) Mirror 26: 0x85a9f147 (stale) Mirror 27: 0x85a9f147 (stale) Mirror 28: 0x85a9f147 (stale) Mirror 29: 0x85a9f147 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 9: 0x85a9f147 (stale) Mirror 14: 0x85a9f147 (stale) Mirror 19: 0x85a9f147 (stale) Mirror 22: 0x85a9f147 (stale) Mirror 23: 0x85a9f147 (stale) Mirror 24: 0x85a9f147 (stale) Mirror 25: 0x85a9f147 (stale) Mirror 26: 0x85a9f147 (stale) Mirror 27: 0x85a9f147 (stale) Mirror 28: 0x85a9f147 (stale) Mirror 29: 0x85a9f147 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb16: skipped 000695 1787782629.893740 mapread 0x0348df thru 0x03890e (0x4030 bytes) 000696 1787782629.914532 trunc from 0x02ed8d to 0x03bb15 (0xcd89 bytes) 000697 1787782629.924408 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 14: 0x85a9f147 (stale) Mirror 19: 0x85a9f147 (stale) Mirror 22: 0x85a9f147 (stale) Mirror 23: 0x85a9f147 (stale) Mirror 24: 0x85a9f147 (stale) Mirror 25: 0x85a9f147 (stale) Mirror 26: 0x85a9f147 (stale) Mirror 27: 0x85a9f147 (stale) Mirror 28: 0x85a9f147 (stale) Mirror 29: 0x85a9f147 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb16): Mirror 14: 0x85a9f147 (stale) Mirror 19: 0x85a9f147 (stale) Mirror 22: 0x85a9f147 (stale) Mirror 23: 0x85a9f147 (stale) Mirror 24: 0x85a9f147 (stale) Mirror 25: 0x85a9f147 (stale) Mirror 26: 0x85a9f147 (stale) Mirror 27: 0x85a9f147 (stale) Mirror 28: 0x85a9f147 (stale) Mirror 29: 0x85a9f147 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb16: skipped 000698 1787782630.813903 mapwrite 0x006530 thru 0x0094f5 (0x2fc6 bytes) 000699 1787782630.890305 trunc from 0x03bb16 to 0x03786e (0x42a7 bytes) 000700 1787782630.920547 trunc from 0x03786f to 0x000c09 (0x36c65 bytes) 000701 1787782630.942098 mapwrite 0x017b31 thru 0x022b3c (0xb00c bytes) 000702 1787782631.000721 read 0x0135fd thru 0x01e8b5 (0xb2b9 bytes) 000703 1787782631.001827 trunc from 0x022b3d to 0x01d8d5 (0x5267 bytes) 000705 1787782631.022559 trunc from 0x01336b to 0x01a61c (0x72b2 bytes) 000706 1787782631.029587 write 0x03525f thru 0x0378d1 (0x2673 bytes) 000707 1787782631.031338 read 0x02f073 thru 0x0315d3 (0x2561 bytes) 000708 1787782631.047052 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x378d2): Mirror 14: 0xd7e31b21 (stale) Mirror 19: 0xd7e31b21 (stale) Mirror 22: 0xd7e31b21 (stale) Mirror 24: 0xd7e31b21 (stale) Mirror 25: 0xd7e31b21 (stale) Mirror 26: 0xd7e31b21 (stale) Mirror 27: 0xd7e31b21 (stale) Mirror 28: 0xd7e31b21 (stale) Mirror 29: 0xd7e31b21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378d2): Mirror 14: 0xd7e31b21 (stale) Mirror 19: 0xd7e31b21 (stale) Mirror 22: 0xd7e31b21 (stale) Mirror 24: 0xd7e31b21 (stale) Mirror 25: 0xd7e31b21 (stale) Mirror 26: 0xd7e31b21 (stale) Mirror 27: 0xd7e31b21 (stale) Mirror 28: 0xd7e31b21 (stale) Mirror 29: 0xd7e31b21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378d2: skipped 000709 1787782631.727807 read 0x00df5b thru 0x0115b9 (0x365f bytes) 000710 1787782631.740700 trunc from 0x0378d2 to 0x02d8f1 (0x9fe0 bytes) 000711 1787782631.765671 mapwrite 0x015ef4 thru 0x024beb (0xecf8 bytes) 000713 1787782631.860911 trunc from 0x023e97 to 0x0287ea (0x4954 bytes) 000714 1787782631.867880 trunc from 0x02d8f2 to 0x02850b (0x53e6 bytes) 000715 1787782631.887814 trunc from 0x007d51 to 0x00c88b (0x4b3b bytes) 000716 1787782631.895280 read 0x008755 thru 0x008fef (0x89b bytes) 000717 1787782631.903879 read 0x01ea8d thru 0x02850b (0x9a7f bytes) 000718 1787782631.906463 trunc from 0x02850c to 0x0381ab (0xfca0 bytes) 000719 1787782631.940179 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x381ac): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381ac): Mirror 14: 0x1af2d94b (stale) Mirror 19: 0x1af2d94b (stale) Mirror 22: 0x1af2d94b (stale) Mirror 24: 0x1af2d94b (stale) Mirror 25: 0x1af2d94b (stale) Mirror 26: 0x1af2d94b (stale) Mirror 27: 0x1af2d94b (stale) Mirror 28: 0x1af2d94b (stale) Mirror 29: 0x1af2d94b (stale) CRC-32 checksum value for chunk [0, 0x381ac): Mirror 1: 0xf4d1a5ff Mirror 30: 0xf4d1a5ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381ac: skipped 000720 1787782633.891120 trunc from 0x00c64a to 0x01b352 (0xed09 bytes) 000721 1787782633.989302 punch from 0x004124 to 0x00ce85 (0x8d62 bytes) 000722 1787782633.996978 mapread 0x01974a thru 0x0230ab (0x9962 bytes) 000723 1787782634.034733 mapread 0x037b14 thru 0x0381ab (0x698 bytes) 000724 1787782634.048549 punch from 0x00fd14 to 0x01b6d1 (0xb9be bytes) 000725 1787782634.056515 mapread 0x001abd thru 0x0022ee (0x832 bytes) 000726 1787782634.073583 mapwrite 0x017a78 thru 0x01b8cb (0x3e54 bytes) 000727 1787782634.125905 fallocate 0x0256f6 thru 0x026741 (0x104c bytes) 000729 1787782634.132676 mapwrite 0x02f61f thru 0x034b0e (0x54f0 bytes) 000730 1787782634.185760 trunc from 0x01b626 to 0x0247b7 (0x9192 bytes) 000731 1787782634.193715 trunc from 0x016d05 to 0x0183c1 (0x16bd bytes) 000732 1787782634.203287 mapread 0x023c7e thru 0x033007 (0xf38a bytes) 000733 1787782634.214916 mapwrite 0x015bbd thru 0x022786 (0xcbca bytes) 000735 1787782634.250164 fallocate 0x010bb8 thru 0x013b9a (0x2fe3 bytes) 000736 1787782634.255819 fallocate 0x0028b4 thru 0x00a4a2 (0x7bef bytes) 000737 1787782634.266811 MIRROR_RESYNC 11 mirrors 000738 1787782637.572184 fallocate 0x02bd7b thru 0x03bc0e (0xfe94 bytes) 000739 1787782637.639151 trunc from 0x03bc0f to 0x03876e (0x34a0 bytes) 000740 1787782637.667030 trunc from 0x0119ab to 0x01635d (0x49b3 bytes) 000741 1787782637.674795 punch from 0x01e425 to 0x02d8ff (0xf4db bytes) 000742 1787782637.685588 punch from 0x01e41c to 0x02ae78 (0xca5d bytes) 000743 1787782637.692658 write 0x00d249 thru 0x01c4b3 (0xf26b bytes) 000744 1787782637.713169 read 0x01c030 thru 0x02886f (0xc840 bytes) 000745 1787782637.727135 mapread 0x0094c8 thru 0x016053 (0xcb8c bytes) 000746 1787782637.756741 read 0x01dba2 thru 0x022173 (0x45d2 bytes) 000747 1787782637.762937 MIRROR_RESYNC 11 mirrors 000748 1787782641.820157 trunc from 0x03876f to 0x00352c (0x35242 bytes) 000749 1787782641.910167 write 0x011f78 thru 0x01854b (0x65d4 bytes) 000750 1787782641.922009 trunc from 0x01854c to 0x0229fc (0xa4b1 bytes) 000752 1787782641.973247 trunc from 0x0229fd to 0x038bcf (0x161d3 bytes) 000753 1787782642.018562 trunc from 0x038bd0 to 0x004075 (0x34b5a bytes) 000754 1787782642.042612 punch from 0x0027bc to 0x004075 (0x18ba bytes) 000756 1787782642.047995 punch from 0x002e1c to 0x004075 (0x125a bytes) 000757 1787782642.054194 fallocate 0x018533 thru 0x0233b5 (0xae83 bytes) 000758 1787782642.059782 fallocate 0x02ecb6 thru 0x03c52a (0xd875 bytes) 000759 1787782642.069894 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x233b6): Mirror 14: 0x39b4512f (stale) Mirror 19: 0x39b4512f (stale) Mirror 24: 0x39b4512f (stale) Mirror 25: 0x39b4512f (stale) Mirror 26: 0x39b4512f (stale) Mirror 27: 0x39b4512f (stale) Mirror 28: 0x39b4512f (stale) Mirror 29: 0x39b4512f (stale) Mirror 30: 0x39b4512f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233b6): Mirror 14: 0x39b4512f (stale) Mirror 19: 0x39b4512f (stale) Mirror 24: 0x39b4512f (stale) Mirror 25: 0x39b4512f (stale) Mirror 26: 0x39b4512f (stale) Mirror 27: 0x39b4512f (stale) Mirror 28: 0x39b4512f (stale) Mirror 29: 0x39b4512f (stale) Mirror 30: 0x39b4512f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233b6: skipped 000760 1787782642.825624 read 0x01dab6 thru 0x021c86 (0x41d1 bytes) 000761 1787782642.836280 trunc from 0x000444 to 0x002a1e (0x25db bytes) 000762 1787782642.845898 write 0x00b21f thru 0x00b374 (0x156 bytes) 000763 1787782642.860375 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x233b6): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233b6): Mirror 14: 0x39b4512f (stale) Mirror 19: 0x39b4512f (stale) Mirror 24: 0x39b4512f (stale) Mirror 25: 0x39b4512f (stale) Mirror 26: 0x39b4512f (stale) Mirror 27: 0x39b4512f (stale) Mirror 28: 0x39b4512f (stale) Mirror 29: 0x39b4512f (stale) Mirror 30: 0x39b4512f (stale) CRC-32 checksum value for chunk [0, 0x233b6): Mirror 1: 0xff09b4ed Mirror 31: 0xff09b4ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233b6: skipped 000764 1787782644.367518 punch from 0x0079c9 to 0x0112ac (0x98e4 bytes) 000765 1787782644.433847 trunc from 0x018460 to 0x0233b5 (0xaf56 bytes) 000767 1787782644.450914 read 0x0163a4 thru 0x01e12b (0x7d88 bytes) 000768 1787782644.494341 fallocate 0x02b8aa thru 0x03778c (0xbee3 bytes) 000769 1787782644.510932 mapwrite 0x02b1cd thru 0x02fdeb (0x4c1f bytes) 000770 1787782644.597789 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2fdec): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdec): Mirror 14: 0xab907c9d (stale) Mirror 19: 0xab907c9d (stale) Mirror 24: 0xab907c9d (stale) Mirror 25: 0xab907c9d (stale) Mirror 26: 0xab907c9d (stale) Mirror 27: 0xab907c9d (stale) Mirror 28: 0xab907c9d (stale) Mirror 29: 0xab907c9d (stale) Mirror 30: 0xab907c9d (stale) Mirror 31: 0x3ac9f4b4 (stale) CRC-32 checksum value for chunk [0, 0x2fdec): Mirror 1: 0x8757ff8d Mirror 32: 0x8757ff8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdec: skipped 000771 1787782645.923362 mapwrite 0x01c1f5 thru 0x026f42 (0xad4e bytes) 000772 1787782646.121126 trunc from 0x013bbc to 0x01d634 (0x9a79 bytes) 000773 1787782646.133958 read 0x00158c thru 0x008992 (0x7407 bytes) 000774 1787782646.173419 mapwrite 0x00834e thru 0x008a07 (0x6ba bytes) 000776 1787782646.209485 write 0x030da6 thru 0x030f70 (0x1cb bytes) 000777 1787782646.217675 read 0x01bcdb thru 0x0250dd (0x9403 bytes) 000778 1787782646.250538 read 0x00bce2 thru 0x014a13 (0x8d32 bytes) 000779 1787782646.251482 mapwrite 0x013725 thru 0x016471 (0x2d4d bytes) 000780 1787782646.297515 fallocate 0x009353 thru 0x016324 (0xcfd2 bytes) 000781 1787782646.325062 trunc from 0x030f71 to 0x025a70 (0xb500 bytes) 000782 1787782646.384327 MIRROR_RESYNC 12 mirrors 000783 1787782649.892412 mapwrite 0x007813 thru 0x0123cf (0xabbd bytes) 000784 1787782650.037804 read 0x01c0a2 thru 0x01ecb3 (0x2c12 bytes) 000785 1787782650.050052 mapread 0x01657d thru 0x018646 (0x20ca bytes) 000786 1787782650.053250 fallocate 0x02c1a3 thru 0x02e966 (0x27c4 bytes) 000787 1787782650.059464 mapread 0x00975b thru 0x01078e (0x7034 bytes) 000788 1787782650.060491 punch from 0x02276f to 0x026e0c (0x469e bytes) 000789 1787782650.074788 write 0x01f941 thru 0x024518 (0x4bd8 bytes) 000790 1787782650.092192 trunc from 0x02e967 to 0x011b31 (0x1ce35 bytes) 000791 1787782650.116873 trunc from 0x0024eb to 0x00d0af (0xabc5 bytes) 000792 1787782650.122972 trunc from 0x011b32 to 0x0076c0 (0xa471 bytes) 000793 1787782650.150546 fallocate 0x002ab8 thru 0x009433 (0x697c bytes) 000794 1787782650.162888 write 0x033063 thru 0x03ffff (0xcf9d bytes) 000797 1787782650.165779 write 0x00213c thru 0x003ed3 (0x1d98 bytes) 000798 1787782650.183542 trunc from 0x028aed to 0x03764e (0xeb62 bytes) 000799 1787782650.211466 punch from 0x0016d9 to 0x00bbf1 (0xa519 bytes) 000801 1787782650.230269 MIRROR_RESYNC 12 mirrors 000802 1787782652.689910 write 0x007abf thru 0x00ed9e (0x72e0 bytes) 000803 1787782652.779886 read 0x006f7f thru 0x0130c8 (0xc14a bytes) 000804 1787782652.799199 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x699b574d (stale) Mirror 24: 0x699b574d (stale) Mirror 25: 0x699b574d (stale) Mirror 26: 0x699b574d (stale) Mirror 27: 0x699b574d (stale) Mirror 28: 0x699b574d (stale) Mirror 29: 0x699b574d (stale) Mirror 30: 0x699b574d (stale) Mirror 31: 0x699b574d (stale) Mirror 32: 0x699b574d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x699b574d (stale) Mirror 24: 0x699b574d (stale) Mirror 25: 0x699b574d (stale) Mirror 26: 0x699b574d (stale) Mirror 27: 0x699b574d (stale) Mirror 28: 0x699b574d (stale) Mirror 29: 0x699b574d (stale) Mirror 30: 0x699b574d (stale) Mirror 31: 0x699b574d (stale) Mirror 32: 0x699b574d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000805 1787782653.626048 fallocate 0x00625f thru 0x012b9e (0xc940 bytes) 000806 1787782653.648280 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x699b574d (stale) Mirror 24: 0x699b574d (stale) Mirror 25: 0x699b574d (stale) Mirror 26: 0x699b574d (stale) Mirror 27: 0x699b574d (stale) Mirror 28: 0x699b574d (stale) Mirror 29: 0x699b574d (stale) Mirror 30: 0x699b574d (stale) Mirror 32: 0x699b574d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x699b574d (stale) Mirror 24: 0x699b574d (stale) Mirror 25: 0x699b574d (stale) Mirror 26: 0x699b574d (stale) Mirror 27: 0x699b574d (stale) Mirror 28: 0x699b574d (stale) Mirror 29: 0x699b574d (stale) Mirror 30: 0x699b574d (stale) Mirror 32: 0x699b574d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000808 1787782654.650277 mapwrite 0x016585 thru 0x01fe6d (0x98e9 bytes) 000809 1787782654.719491 punch from 0x030312 to 0x03e612 (0xe301 bytes) 000810 1787782654.747890 MIRROR_RESYNC 10 mirrors 000812 1787782657.842775 read 0x004f07 thru 0x00ca3d (0x7b37 bytes) 000813 1787782657.855275 write 0x002b6c thru 0x00d348 (0xa7dd bytes) 000814 1787782657.930563 write 0x02bb02 thru 0x039865 (0xdd64 bytes) 000815 1787782657.948517 write 0x02ffe6 thru 0x03fe5d (0xfe78 bytes) 000816 1787782657.970639 write 0x015de8 thru 0x020645 (0xa85e bytes) 000817 1787782657.995479 punch from 0x00ff5f to 0x011cbc (0x1d5e bytes) 000818 1787782658.000574 read 0x01f609 thru 0x028406 (0x8dfe bytes) 000819 1787782658.010968 trunc from 0x00da9e to 0x010fb9 (0x351c bytes) 000820 1787782658.065388 write 0x0263f4 thru 0x02f4b0 (0x90bd bytes) 000821 1787782658.067472 fallocate 0x0225ad thru 0x02c0e0 (0x9b34 bytes) 000822 1787782658.076878 read 0x00a123 thru 0x00bda6 (0x1c84 bytes) 000823 1787782658.077729 read 0x03edab thru 0x03ffff (0x1255 bytes) 000824 1787782658.083164 trunc from 0x00f0b8 to 0x00f406 (0x34f bytes) 000825 1787782658.099029 mapread 0x03a444 thru 0x03ffff (0x5bbc bytes) 000826 1787782658.101148 read 0x008671 thru 0x016645 (0xdfd5 bytes) 000827 1787782658.154091 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xf54f7b4f (stale) Mirror 24: 0xf54f7b4f (stale) Mirror 25: 0xf54f7b4f (stale) Mirror 26: 0xf54f7b4f (stale) Mirror 27: 0xf54f7b4f (stale) Mirror 28: 0xf54f7b4f (stale) Mirror 29: 0xf54f7b4f (stale) Mirror 30: 0xf54f7b4f (stale) Mirror 32: 0xf54f7b4f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5752d680 Mirror 33: 0x5752d680 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000828 1787782659.857286 trunc from 0x040000 to 0x018113 (0x27eec bytes) 000829 1787782660.009261 trunc from 0x007e3c to 0x0130f6 (0xb2bb bytes) 000830 1787782660.017954 read 0x01085b thru 0x018113 (0x78b9 bytes) 000831 1787782660.037304 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18114): Mirror 19: 0xf794a550 (stale) Mirror 24: 0xf794a550 (stale) Mirror 26: 0xf794a550 (stale) Mirror 27: 0xf794a550 (stale) Mirror 28: 0xf794a550 (stale) Mirror 29: 0xf794a550 (stale) Mirror 30: 0xf794a550 (stale) Mirror 32: 0xf794a550 (stale) Mirror 33: 0xae0daac5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18114): Mirror 19: 0xf794a550 (stale) Mirror 24: 0xf794a550 (stale) Mirror 26: 0xf794a550 (stale) Mirror 27: 0xf794a550 (stale) Mirror 28: 0xf794a550 (stale) Mirror 29: 0xf794a550 (stale) Mirror 30: 0xf794a550 (stale) Mirror 32: 0xf794a550 (stale) Mirror 33: 0xae0daac5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18114: skipped 000833 1787782660.730336 trunc from 0x018114 to 0x02158c (0x9479 bytes) 000834 1787782660.779700 punch from 0x020e9c to 0x02158c (0x6f1 bytes) 000835 1787782660.785245 read 0x010e10 thru 0x01bf46 (0xb137 bytes) 000836 1787782660.794310 mapwrite 0x0027e5 thru 0x00a4df (0x7cfb bytes) 000837 1787782660.883623 punch from 0x01af8d to 0x02158c (0x6600 bytes) 000838 1787782660.889202 read 0x0079a5 thru 0x00a1a1 (0x27fd bytes) 000839 1787782660.891163 trunc from 0x0012ec to 0x00a643 (0x9358 bytes) 000840 1787782660.920864 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2158d): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2158d): Mirror 19: 0xc99b44a2 (stale) Mirror 24: 0xc99b44a2 (stale) Mirror 26: 0xc99b44a2 (stale) Mirror 27: 0xc99b44a2 (stale) Mirror 28: 0xc99b44a2 (stale) Mirror 29: 0xc99b44a2 (stale) Mirror 30: 0xc99b44a2 (stale) Mirror 32: 0xc99b44a2 (stale) Mirror 33: 0xd82e9971 (stale) CRC-32 checksum value for chunk [0, 0x2158d): Mirror 1: 0x269a1277 Mirror 34: 0x269a1277 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2158d: skipped 000841 1787782662.523641 write 0x019f98 thru 0x01ec45 (0x4cae bytes) 000842 1787782662.608932 trunc from 0x008b65 to 0x00b3b0 (0x284c bytes) 000843 1787782662.623924 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2158d): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2158d): Mirror 19: 0xc99b44a2 (stale) Mirror 24: 0xc99b44a2 (stale) Mirror 26: 0xc99b44a2 (stale) Mirror 27: 0xc99b44a2 (stale) Mirror 28: 0xc99b44a2 (stale) Mirror 29: 0xc99b44a2 (stale) Mirror 30: 0xc99b44a2 (stale) Mirror 32: 0xc99b44a2 (stale) Mirror 33: 0xd82e9971 (stale) Mirror 34: 0x269a1277 (stale) CRC-32 checksum value for chunk [0, 0x2158d): Mirror 1: 0x3c173426 Mirror 35: 0x3c173426 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2158d: skipped 000844 1787782663.851870 punch from 0x01f995 to 0x02158c (0x1bf8 bytes) 000845 1787782663.941326 punch from 0x001083 to 0x003757 (0x26d5 bytes) 000846 1787782663.950021 read 0x01fd84 thru 0x02158c (0x1809 bytes) 000847 1787782663.965351 mapread 0x0090bc thru 0x011dd5 (0x8d1a bytes) 000848 1787782664.018923 trunc from 0x02158d to 0x023976 (0x23ea bytes) 000849 1787782664.070924 mapwrite 0x03037a thru 0x03f609 (0xf290 bytes) 000850 1787782664.132125 mapwrite 0x01bb3c thru 0x020b92 (0x5057 bytes) 000851 1787782664.167102 MIRROR_RESYNC 12 mirrors 000852 1787782667.644106 read 0x00ac11 thru 0x014ca7 (0xa097 bytes) 000853 1787782667.657859 fallocate 0x039c64 thru 0x03ffff (0x639c bytes) 000854 1787782667.735195 write 0x0326fe thru 0x032ad9 (0x3dc bytes) 000855 1787782667.741238 read 0x02f74d thru 0x03c2b0 (0xcb64 bytes) 000856 1787782667.753038 punch from 0x03935e to 0x03f609 (0x62ac bytes) 000857 1787782667.760378 read 0x01b326 thru 0x0205c7 (0x52a2 bytes) 000858 1787782667.761088 read 0x0288d4 thru 0x02c9f1 (0x411e bytes) 000860 1787782667.768590 MIRROR_RESYNC 12 mirrors 000861 1787782671.106582 fallocate 0x03d10d thru 0x03d78d (0x681 bytes) 000862 1787782671.146818 trunc from 0x03f60a to 0x03a27b (0x538e bytes) 000863 1787782671.201433 trunc from 0x03a27c to 0x015006 (0x25275 bytes) 000864 1787782671.235573 mapwrite 0x027b2b thru 0x02ae50 (0x3326 bytes) 000865 1787782671.301024 trunc from 0x01ffba to 0x026431 (0x6478 bytes) 000866 1787782671.307298 mapread 0x0206d1 thru 0x0257e5 (0x5115 bytes) 000867 1787782671.318742 trunc from 0x02ae51 to 0x005550 (0x25900 bytes) 000868 1787782671.339312 write 0x03d72a thru 0x03ffff (0x28d6 bytes) 000869 1787782671.342038 trunc from 0x03f5f6 to 0x03ffff (0xa0a bytes) 000870 1787782671.361166 mapwrite 0x013aab thru 0x022e0f (0xf365 bytes) 000871 1787782671.406131 fallocate 0x01153f thru 0x01b6d9 (0xa19b bytes) 000873 1787782671.412480 mapread 0x029edf thru 0x034006 (0xa128 bytes) 000874 1787782671.433883 read 0x02bcb5 thru 0x02c9db (0xd27 bytes) 000875 1787782671.434978 trunc from 0x040000 to 0x0209e6 (0x1f619 bytes) 000876 1787782671.483789 mapread 0x00fb0a thru 0x015cd5 (0x61cc bytes) 000877 1787782671.493989 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x209e7): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209e7): Mirror 19: 0xd9940055 (stale) Mirror 24: 0xd9940055 (stale) Mirror 26: 0xd9940055 (stale) Mirror 27: 0xd9940055 (stale) Mirror 28: 0xd9940055 (stale) Mirror 29: 0xd9940055 (stale) Mirror 30: 0xd9940055 (stale) Mirror 32: 0xd9940055 (stale) Mirror 33: 0xd9940055 (stale) Mirror 34: 0xd9940055 (stale) Mirror 35: 0xd9940055 (stale) CRC-32 checksum value for chunk [0, 0x209e7): Mirror 1: 0x53395e08 Mirror 36: 0x53395e08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209e7: skipped 000878 1787782673.187716 mapread 0x011e7e thru 0x0200ff (0xe282 bytes) 000879 1787782673.217729 fallocate 0x014d44 thru 0x01cdb3 (0x8070 bytes) 000880 1787782673.291794 mapwrite 0x022066 thru 0x022f39 (0xed4 bytes) 000881 1787782673.395005 mapread 0x01f2cb thru 0x022f39 (0x3c6f bytes) 000882 1787782673.405644 punch from 0x01fc3b to 0x022f39 (0x32ff bytes) 000884 1787782673.421259 mapread 0x01d963 thru 0x022f39 (0x55d7 bytes) 000885 1787782673.453950 mapwrite 0x01a75d thru 0x01b474 (0xd18 bytes) 000886 1787782673.502522 mapread 0x021bd4 thru 0x022f39 (0x1366 bytes) 000887 1787782673.503418 punch from 0x0191a3 to 0x022f39 (0x9d97 bytes) 000888 1787782673.511995 mapwrite 0x0394f4 thru 0x03ffff (0x6b0c bytes) 000889 1787782673.590723 mapwrite 0x0297db thru 0x02f84f (0x6075 bytes) 000890 1787782673.630846 fallocate 0x013c48 thru 0x017cb0 (0x4069 bytes) 000891 1787782673.647741 trunc from 0x040000 to 0x039e0a (0x61f5 bytes) 000892 1787782673.681027 punch from 0x00b8e5 to 0x00cd3b (0x1457 bytes) 000894 1787782673.700163 mapwrite 0x017258 thru 0x01f322 (0x80cb bytes) 000897 1787782673.767188 mapread 0x0224d9 thru 0x02d138 (0xac60 bytes) 000898 1787782673.770392 trunc from 0x02a1b3 to 0x02d765 (0x35b3 bytes) 000899 1787782673.778977 mapread 0x02415d thru 0x02f927 (0xb7cb bytes) 000900 1787782673.791822 punch from 0x026cdf to 0x02891e (0x1c40 bytes) 000901 1787782673.807405 fallocate 0x01c83e thru 0x021122 (0x48e5 bytes) 000902 1787782673.829036 read 0x021d3b thru 0x025760 (0x3a26 bytes) 000903 1787782673.843561 mapread 0x022412 thru 0x02c4d3 (0xa0c2 bytes) 000904 1787782673.860359 mapread 0x00bb9e thru 0x01019c (0x45ff bytes) 000905 1787782673.886609 fallocate 0x016ad8 thru 0x01f9f8 (0x8f21 bytes) 000906 1787782673.894483 trunc from 0x038fef to 0x039e0a (0xe1c bytes) 000907 1787782673.904366 punch from 0x024a54 to 0x02e390 (0x993d bytes) 000909 1787782673.923683 trunc from 0x039e0b to 0x0183ad (0x21a5d bytes) 000910 1787782673.960499 read 0x000226 thru 0x004713 (0x44ee bytes) 000911 1787782673.975425 punch from 0x0010dd to 0x00c73c (0xb660 bytes) 000912 1787782673.981891 mapread 0x018258 thru 0x0183ad (0x156 bytes) 000913 1787782673.982960 mapwrite 0x0290b1 thru 0x036046 (0xcf96 bytes) 000914 1787782674.062642 trunc from 0x036047 to 0x01ae89 (0x1b1bd bytes) 000915 1787782674.087081 read 0x010df3 thru 0x014234 (0x3442 bytes) 000916 1787782674.087927 punch from 0x0040fc to 0x012ad9 (0xe9de bytes) 000917 1787782674.101269 mapread 0x00bf54 thru 0x01710f (0xb1bc bytes) 000918 1787782674.117838 mapread 0x005c22 thru 0x00f670 (0x9a4f bytes) 000919 1787782674.118999 read 0x0096ee thru 0x00d1af (0x3ac2 bytes) 000920 1787782674.120514 trunc from 0x005664 to 0x0097cb (0x4168 bytes) 000921 1787782674.125969 mapwrite 0x0374a5 thru 0x039ef4 (0x2a50 bytes) 000922 1787782674.211949 trunc from 0x039ef5 to 0x00649a (0x33a5a bytes) 000923 1787782674.242723 mapread 0x002704 thru 0x003af3 (0x13f0 bytes) 000924 1787782674.244026 punch from 0x001017 to 0x00649a (0x5484 bytes) 000926 1787782674.256142 mapwrite 0x028696 thru 0x02ea35 (0x63a0 bytes) 000927 1787782674.344525 mapwrite 0x0304e1 thru 0x0374ec (0x700c bytes) 000928 1787782674.497400 trunc from 0x035a47 to 0x0374ec (0x1aa6 bytes) 000929 1787782674.512283 mapwrite 0x004247 thru 0x012310 (0xe0ca bytes) 000930 1787782674.751024 trunc from 0x0374ed to 0x03b3a5 (0x3eb9 bytes) 000931 1787782674.797749 read 0x01196a thru 0x01d1f2 (0xb889 bytes) 000932 1787782674.817114 mapread 0x011fae thru 0x012eb1 (0xf04 bytes) 000933 1787782674.817998 punch from 0x0283c4 to 0x034025 (0xbc62 bytes) 000934 1787782674.824883 fallocate 0x020a85 thru 0x0301ab (0xf727 bytes) 000935 1787782674.834563 mapwrite 0x02aaf5 thru 0x039c35 (0xf141 bytes) 000936 1787782674.968146 fallocate 0x01ce2d thru 0x02ce29 (0xfffd bytes) 000937 1787782674.985544 read 0x011bb9 thru 0x01b969 (0x9db1 bytes) 000940 1787782674.986951 trunc from 0x03b3a6 to 0x037efd (0x34a8 bytes) 000941 1787782675.014349 trunc from 0x037efe to 0x02b84f (0xc6ae bytes) 000942 1787782675.050420 punch from 0x006b5a to 0x010dc4 (0xa26b bytes) 000943 1787782675.059835 trunc from 0x02b850 to 0x00ef71 (0x1c8de bytes) 000944 1787782675.097351 mapread 0x002d15 thru 0x00652c (0x3818 bytes) 000945 1787782675.122847 mapread 0x00e12f thru 0x00ef71 (0xe43 bytes) 000946 1787782675.126019 fallocate 0x03deab thru 0x03ffff (0x2155 bytes) 000947 1787782675.134815 mapread 0x01561a thru 0x023ed2 (0xe8b9 bytes) 000948 1787782675.177095 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x27ad9bde (stale) Mirror 24: 0x27ad9bde (stale) Mirror 26: 0x27ad9bde (stale) Mirror 27: 0x27ad9bde (stale) Mirror 28: 0x27ad9bde (stale) Mirror 29: 0x27ad9bde (stale) Mirror 30: 0x27ad9bde (stale) Mirror 32: 0x27ad9bde (stale) Mirror 34: 0x27ad9bde (stale) Mirror 35: 0x27ad9bde (stale) Mirror 36: 0x2de6eafd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x27ad9bde (stale) Mirror 24: 0x27ad9bde (stale) Mirror 26: 0x27ad9bde (stale) Mirror 27: 0x27ad9bde (stale) Mirror 28: 0x27ad9bde (stale) Mirror 29: 0x27ad9bde (stale) Mirror 30: 0x27ad9bde (stale) Mirror 32: 0x27ad9bde (stale) Mirror 34: 0x27ad9bde (stale) Mirror 35: 0x27ad9bde (stale) Mirror 36: 0x2de6eafd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1787782676.130796 read 0x00ec80 thru 0x014413 (0x5794 bytes) 000950 1787782676.145991 trunc from 0x040000 to 0x0291f6 (0x16e09 bytes) 000951 1787782676.176840 trunc from 0x014488 to 0x02263f (0xe1b8 bytes) 000952 1787782676.187833 read 0x002d6c thru 0x00e6cb (0xb960 bytes) 000953 1787782676.213751 fallocate 0x03092f thru 0x031d8a (0x145c bytes) 000954 1787782676.224848 mapwrite 0x033c56 thru 0x03e173 (0xa51e bytes) 000955 1787782676.276497 mapwrite 0x03385a thru 0x03b369 (0x7b10 bytes) 000956 1787782676.320768 write 0x02d573 thru 0x03b54d (0xdfdb bytes) 000958 1787782676.336492 fallocate 0x0176c2 thru 0x01852d (0xe6c bytes) 000959 1787782676.349189 write 0x026259 thru 0x02c8ee (0x6696 bytes) 000960 1787782676.353794 trunc from 0x03e174 to 0x036990 (0x77e3 bytes) 000961 1787782676.403129 write 0x012de0 thru 0x014e6d (0x208e bytes) 000962 1787782676.404390 mapwrite 0x02cf59 thru 0x03814b (0xb1f3 bytes) 000963 1787782676.505866 MIRROR_RESYNC 12 mirrors 000964 1787782680.727412 fallocate 0x02727c thru 0x02944a (0x21cf bytes) 000965 1787782680.768083 write 0x0189da thru 0x01c96e (0x3f95 bytes) 000966 1787782680.783284 MIRROR_RESYNC 12 mirrors 000967 1787782685.515054 MIRROR_RESYNC 12 mirrors 000968 1787782686.473887 mapread 0x005ac9 thru 0x00bac5 (0x5ffd bytes) 000969 1787782686.516578 trunc from 0x027bb0 to 0x028f24 (0x1375 bytes) 000970 1787782686.607513 fallocate 0x0214e3 thru 0x02f9a3 (0xe4c1 bytes) 000971 1787782686.624356 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3814c): Mirror 19: 0xe936320f (stale) Mirror 24: 0xe936320f (stale) Mirror 26: 0xe936320f (stale) Mirror 28: 0xe936320f (stale) Mirror 29: 0xe936320f (stale) Mirror 30: 0xe936320f (stale) Mirror 32: 0xe936320f (stale) Mirror 34: 0xe936320f (stale) Mirror 35: 0xe936320f (stale) Mirror 36: 0xe936320f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3814c): Mirror 19: 0xe936320f (stale) Mirror 24: 0xe936320f (stale) Mirror 26: 0xe936320f (stale) Mirror 28: 0xe936320f (stale) Mirror 29: 0xe936320f (stale) Mirror 30: 0xe936320f (stale) Mirror 32: 0xe936320f (stale) Mirror 34: 0xe936320f (stale) Mirror 35: 0xe936320f (stale) Mirror 36: 0xe936320f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3814c: skipped 000972 1787782687.476687 read 0x00850f thru 0x011f63 (0x9a55 bytes) 000973 1787782687.488867 read 0x030a86 thru 0x036821 (0x5d9c bytes) 000974 1787782687.522167 mapwrite 0x02c2f9 thru 0x039726 (0xd42e bytes) 000975 1787782687.617487 mapread 0x031a7f thru 0x039726 (0x7ca8 bytes) 000976 1787782687.619467 trunc from 0x039727 to 0x03272a (0x6ffc bytes) 000977 1787782687.658626 trunc from 0x024155 to 0x02b134 (0x6fe0 bytes) 000978 1787782687.669014 fallocate 0x032f70 thru 0x03e905 (0xb996 bytes) 000979 1787782687.684993 write 0x02126b thru 0x027766 (0x64fc bytes) 000980 1787782687.715914 trunc from 0x0074b7 to 0x015a39 (0xe583 bytes) 000981 1787782687.721103 trunc from 0x01937c to 0x01f916 (0x659b bytes) 000982 1787782687.729078 read 0x02ed27 thru 0x02f25b (0x535 bytes) 000983 1787782687.730634 write 0x01ccaf thru 0x02a30c (0xd65e bytes) 000984 1787782687.757021 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x177:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e906): Mirror 19: 0x8b5ffa7 (stale) Mirror 24: 0x8b5ffa7 (stale) Mirror 26: 0x8b5ffa7 (stale) Mirror 28: 0x8b5ffa7 (stale) Mirror 30: 0x8b5ffa7 (stale) Mirror 32: 0x8b5ffa7 (stale) Mirror 34: 0x8b5ffa7 (stale) Mirror 35: 0x8b5ffa7 (stale) Mirror 36: 0x8b5ffa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e906): Mirror 19: 0x8b5ffa7 (stale) Mirror 24: 0x8b5ffa7 (stale) Mirror 26: 0x8b5ffa7 (stale) Mirror 28: 0x8b5ffa7 (stale) Mirror 30: 0x8b5ffa7 (stale) Mirror 32: 0x8b5ffa7 (stale) Mirror 34: 0x8b5ffa7 (stale) Mirror 35: 0x8b5ffa7 (stale) Mirror 36: 0x8b5ffa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e906: skipped 000985 1787782688.535746 punch from 0x032cd5 to 0x039d3f (0x706b bytes) 000986 1787782688.546848 fallocate 0x016172 thru 0x0197ba (0x3649 bytes) 000987 1787782688.555684 trunc from 0x0284ce to 0x035d68 (0xd89b bytes) 000988 1787782688.566698 trunc from 0x00ccf7 to 0x00e67c (0x1986 bytes) 000989 1787782688.571657 read 0x03e6b3 thru 0x03e905 (0x253 bytes) 000990 1787782688.582851 trunc from 0x03e906 to 0x00cd6c (0x31b99 bytes) 000992 1787782688.635376 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xcd6d): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcd6d): Mirror 19: 0xc85a8b4a (stale) Mirror 24: 0xc85a8b4a (stale) Mirror 26: 0xc85a8b4a (stale) Mirror 28: 0xc85a8b4a (stale) Mirror 30: 0xc85a8b4a (stale) Mirror 32: 0xc85a8b4a (stale) Mirror 34: 0xc85a8b4a (stale) Mirror 35: 0xc85a8b4a (stale) Mirror 36: 0xc85a8b4a (stale) CRC-32 checksum value for chunk [0, 0xcd6d): Mirror 1: 0xc85a8b4a Mirror 37: 0xc85a8b4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcd6d: skipped 000993 1787782690.252912 mapwrite 0x016a3e thru 0x01fe8d (0x9450 bytes) 000994 1787782690.430190 punch from 0x01eef9 to 0x01fe8d (0xf95 bytes) 000995 1787782690.437022 trunc from 0x0166ce to 0x01b447 (0x4d7a bytes) 000997 1787782690.446387 trunc from 0x00f383 to 0x00f6e7 (0x365 bytes) 000998 1787782690.451362 write 0x001b57 thru 0x00851c (0x69c6 bytes) 000999 1787782690.464595 write 0x030759 thru 0x037a55 (0x72fd bytes) 001000 1787782690.465979 mapread 0x030a54 thru 0x037a55 (0x7002 bytes) All operations completed A-OK! PASS 70a (191s) == sanity-flr test 71: check mirror extend parent fid ==== 18:18:18 (1787782698) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 18:18:31 (1787782711) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351433 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209017 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 18:18:42 (1787782722) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 9229632 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9031456 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9393952 .. Waiting 62623 62624 62625 62626 62629 done 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:20:06 (1787782806) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8313056 .. Extending file size to 8802528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8341056 .. Extending file size to 8438528 .. Extending file size to 9314752 .. Extending file size to 9055456 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 64295 64297 64298 64300 64301 failed 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:21:33 (1787782893) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2372 1268932 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24884 3582136 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19000 3588020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43884 7170156 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2372 1268932 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24884 3582136 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19000 3588020 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43884 7170156 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162525 s, 12.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222052 s, 9.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x598808aa (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:21:58 (1787782918) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940148 s, 22.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 18:22:05 (1787782925) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:22:13 (1787782933) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123325 s, 17.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x216:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x217:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:22:22 (1787782942) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166524 s, 12.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:22:33 (1787782953) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122258 s, 17.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:23:08 (1787782988) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455791 s, 23.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x218:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x218:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x218:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:24:04 (1787783044) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145038 s, 14.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:25:00 (1787783100) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg450-server: Pool lustre.test_204f created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0715929 s, 29.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg450-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:26:17 (1787783177) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:26:25 (1787783185) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:26:34 (1787783194) oleg450-server: Pool lustre.test_206 created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg450-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:27:01 (1787783221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545572 s, 19.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746343 s, 14.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x50c5c894 Mirror 3: 0x50c5c894 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:27:14 (1787783234) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:27:18 (1787783238) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:27:23 (1787783243) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0616061 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xec0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x930000 Read this (4 bytes): BBBB 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:28:06 (1787783286) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.015647 s, 524 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:28:14 (1787783294) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748316 s, 14.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 18:28:17.000000000 -0400 Modify: 2026-08-26 18:28:17.000000000 -0400 Change: 2026-08-26 18:28:18.000000000 -0400 Birth: 2026-08-26 18:28:17.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:28:25 (1787783305) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366951 s, 28.6 MB/s oleg450-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg450-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg450-server: O/280000401/d3/143: File not found by ext2_lookup PASS 210b (29s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:28:54 (1787783334) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.256628 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0232605 s, 176 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 18:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 18:28 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:29:04 (1787783344) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:29:13 (1787783353) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2032 sec =========== 18:29:21 (1787783361) === sanity-flr: start cleanup 18:29:23 (1787783363) === === sanity-flr: finish cleanup 18:29:33 (1787783373) ===