-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 09:43:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg229-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:44:03 (1787838243) === oleg229-client.virtnet: executing check_config_client /mnt/lustre oleg229-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg229-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff956143ccc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff956143ccc000.idle_timeout=debug disable quota as required oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:44:50 (1787838290) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8215] == sanity-flr test 0a: lfs mirror create with -N option == 09:44:52 (1787838292) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:45:09 (1787838309) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:45:13 (1787838313) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:45:16 (1787838316) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:45:42 (1787838342) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:45:45 (1787838345) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:45:48 (1787838348) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff956143ccc000.stats=clear osc.lustre-OST0001-osc-ffff956143ccc000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:45:57 (1787838357) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:46:08 (1787838368) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:46:17 (1787838377) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:46:20 (1787838380) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:46:29 (1787838389) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 09:46:39 (1787838399) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:46:48 (1787838408) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:46:57 (1787838417) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:47:06 (1787838426) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:47:21 (1787838441) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:47:28 (1787838448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703346 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501488 s, 20.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:47:35 (1787838455) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237041 s, 44.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:47:44 (1787838464) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:47:56 (1787838476) Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:49:12 (1787838552) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.616694 s, 11.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:49:55 (1787838595) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:51:17 (1787838677) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.04854 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 09:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.248s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.221s user 0m0.000s sys 0m0.186s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.295s user 0m0.001s sys 0m0.185s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956143ccc000.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:52:58 (1787838778) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:53:02 (1787838782) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:53:06 (1787838786) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:53:09 (1787838789) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:53:17 (1787838797) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.610615 s, 20.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:53:29 (1787838809) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509552 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412209 s, 25.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:53:36 (1787838816) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:53:45 (1787838825) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128814 s, 795 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115442 s, 887 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102264 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00337572 s, 303 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:53:57 (1787838837) create test files with size 15746507 .. c3019679003adda3a18a4430a41c9ac7 - 45dfd241b53300a88aa92b14ab6607cf - 5526cb4be1fe9dd5a060aa3cd96883db - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 09:54:24 (1787838864) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (21s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:54:45 (1787838885) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:54:53 (1787838893) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792648 s, 13.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03383 s, 31.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 09:55:04 (1787838904) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0680031 s, 30.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.321025 s, 13.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254549 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 09:55:22 (1787838922) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:55:25 (1787838925) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:55:29 (1787838929) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425076 s, 24.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 09:55:38 (1787838938) llite.lustre-ffff95614a2fa000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 09:55:47 (1787838947) llite.lustre-ffff95614a2fa000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287349 s, 36.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462759 s, 22.7 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 09:55:55 (1787838955) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.410156 s, 7.7 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:56:05 (1787838965) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:56:17 (1787838977) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.477178 s, 22.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:56:30 (1787838990) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9561429d2000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.512664 s, 20.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff95614c205000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:56:46 (1787839006) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:57:30 (1787839050) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:58:07 (1787839087) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:58:11 (1787839091) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.208302 s, 15.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.235114 s, 13.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xffebeffb (nosync,stale) Mirror 2: 0xffebeffb (nosync,stale) ** verify mirror 2 doesn't change original checksum: d1d66818b2fe6e346810873a5c9dfe09 - mirror 1 checksum: 6b1a2af8ed1abe941796370362aac2f2 - mirror 2 checksum: d1d66818b2fe6e346810873a5c9dfe09 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T09:58:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T09:58:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d1d66818b2fe6e346810873a5c9dfe09 - mirror 1 checksum: 6b1a2af8ed1abe941796370362aac2f2 - mirror 2 checksum: 6b1a2af8ed1abe941796370362aac2f2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T09:58:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T09:58:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:58:23 (1787839103) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597218 s, 17.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:58:30 (1787839110) Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 09:58:42 (1787839122) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 09:59:18 (1787839158) targets are mounted 09:59:18 (1787839158) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501499 s, 20.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:59:38 (1787839178) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0572517 s, 358 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0131126 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:59:49 (1787839189) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0399876 s, 512 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00849529 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:00:00 (1787839200) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00989698 s, 1.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:00:07 (1787839207) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.069062 s, 297 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:00:15 (1787839215) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00722264 s, 567 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0239346 s, 171 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:00:24 (1787839224) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:00:46 (1787839246) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:01:15 (1787839275) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:01:41 (1787839301) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:02:08 (1787839328) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 10:02:16 (1787839336) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58851 mirror split pid 58852 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58851 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58852 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 10:03:27 (1787839407) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9410 000001 1787839410.290073 mapwrite 0x00af6e thru 0x01818e (0xd221 bytes) truncating to largest ever: 0x1e5e5 000002 1787839410.404109 trunc from 0x01818f to 0x01e5e4 (0x6456 bytes) 000003 1787839410.480348 punch from 0x014ce5 to 0x01e5e4 (0x9900 bytes) 000005 1787839410.499336 mapwrite 0x023454 thru 0x0279df (0x458c bytes) 000006 1787839410.550987 read 0x025333 thru 0x0279df (0x26ad bytes) fallocating to largest ever: 0x3558f 000007 1787839410.554369 fallocate 0x034ea7 thru 0x03558e (0x6e8 bytes) 000009 1787839410.567326 trunc from 0x0279e0 to 0x021a78 (0x5f67 bytes) 000011 1787839410.581226 punch from 0x00c21f to 0x0170eb (0xaecd bytes) 000012 1787839410.586569 read 0x012518 thru 0x015f0d (0x39f6 bytes) 000013 1787839410.597396 mapread 0x01c439 thru 0x021a78 (0x5640 bytes) fallocating to largest ever: 0x3eb0c 000014 1787839410.599454 fallocate 0x03e5c8 thru 0x03eb0b (0x544 bytes) 000015 1787839410.604685 punch from 0x019a17 to 0x021a78 (0x8062 bytes) 000016 1787839410.609566 read 0x015ac5 thru 0x021a78 (0xbfb4 bytes) 000017 1787839410.622031 trunc from 0x001d22 to 0x004ffa (0x32d9 bytes) 000018 1787839410.628045 punch from 0x01a29c to 0x01d84f (0x35b4 bytes) 000019 1787839410.634349 punch from 0x006d2a to 0x00e345 (0x761c bytes) 000020 1787839410.640271 trunc from 0x021a79 to 0x02efbe (0xd546 bytes) 000021 1787839410.657331 trunc from 0x02efbf to 0x03694f (0x7991 bytes) 000022 1787839410.674383 read 0x00683f thru 0x007d97 (0x1559 bytes) 000023 1787839410.687308 write 0x006d8d thru 0x00c981 (0x5bf5 bytes) 000024 1787839410.690493 fallocate 0x012960 thru 0x019346 (0x69e7 bytes) 000025 1787839410.698251 mapread 0x02a961 thru 0x02f36f (0x4a0f bytes) 000026 1787839410.700041 read 0x0250a5 thru 0x02ad3b (0x5c97 bytes) 000027 1787839410.701332 read 0x0253c9 thru 0x026b73 (0x17ab bytes) 000029 1787839410.704046 mapwrite 0x026015 thru 0x02800c (0x1ff8 bytes) 000030 1787839410.727814 mapwrite 0x0042fe thru 0x004e88 (0xb8b bytes) 000032 1787839410.741016 trunc from 0x01f8de to 0x029767 (0x9e8a bytes) 000033 1787839410.745168 trunc from 0x036950 to 0x029d69 (0xcbe6 bytes) 000034 1787839410.755614 trunc from 0x029d6a to 0x02ef0f (0x51a6 bytes) 000035 1787839410.776678 fallocate 0x0166b6 thru 0x020197 (0x9ae2 bytes) 000036 1787839410.783238 trunc from 0x012d69 to 0x01676f (0x3a07 bytes) 000037 1787839410.787632 fallocate 0x01efcc thru 0x02c974 (0xd9a9 bytes) 000038 1787839410.793037 mapwrite 0x00e1f4 thru 0x015cfc (0x7b09 bytes) 000039 1787839410.824481 read 0x012318 thru 0x02188f (0xf578 bytes) 000040 1787839410.825328 trunc from 0x02ef10 to 0x00d46b (0x21aa4 bytes) 000041 1787839410.870713 trunc from 0x00d46c to 0x01dbb3 (0x10748 bytes) 000042 1787839410.894125 mapread 0x01464f thru 0x01d107 (0x8ab9 bytes) 000043 1787839410.925345 write 0x035f4e thru 0x03ffff (0xa0b2 bytes) 000044 1787839410.926672 write 0x00fcf6 thru 0x01d358 (0xd663 bytes) 000046 1787839410.930390 mapwrite 0x0209ad thru 0x02bc9a (0xb2ee bytes) 000047 1787839410.976793 fallocate 0x026c96 thru 0x03204e (0xb3b9 bytes) 000048 1787839410.981977 mapread 0x031ef5 thru 0x03301b (0x1127 bytes) zero_range to largest ever: 0x40000 000049 1787839410.983028 trunc from 0x03b6b0 to 0x03ffff (0x4950 bytes) 000050 1787839410.988128 mapread 0x01f48e thru 0x027667 (0x81da bytes) 000052 1787839410.990082 write 0x02d427 thru 0x037c33 (0xa80d bytes) 000054 1787839410.993995 punch from 0x033055 to 0x0339a5 (0x951 bytes) 000056 1787839411.012328 MIRROR_EXTEND to 2 mirrors 000057 1787839412.113745 fallocate 0x01848b thru 0x01933e (0xeb4 bytes) 000058 1787839412.203666 punch from 0x021e13 to 0x02c550 (0xa73e bytes) 000059 1787839412.226838 fallocate 0x03cb2d thru 0x03ffff (0x34d3 bytes) 000060 1787839412.241672 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x327d8d41 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5dd0462a Mirror 3: 0x5dd0462a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000061 1787839413.536714 mapwrite 0x0070bf thru 0x009bdb (0x2b1d bytes) 000062 1787839413.648626 mapread 0x018b5b thru 0x02186e (0x8d14 bytes) 000063 1787839413.684646 trunc from 0x040000 to 0x01b9aa (0x24655 bytes) 000064 1787839413.722868 punch from 0x006e1b to 0x011b19 (0xacff bytes) 000065 1787839413.735671 read 0x012130 thru 0x01b44b (0x931c bytes) 000066 1787839413.736476 trunc from 0x00d990 to 0x018024 (0xa695 bytes) 000067 1787839413.760870 punch from 0x009350 to 0x014121 (0xadd2 bytes) 000068 1787839413.779072 read 0x014344 thru 0x01b9aa (0x7667 bytes) 000069 1787839413.807146 fallocate 0x0398ea thru 0x03ffff (0x6716 bytes) 000070 1787839413.824456 mapwrite 0x02930b thru 0x0375c0 (0xe2b6 bytes) 000071 1787839413.935303 write 0x03d0d3 thru 0x03ffff (0x2f2d bytes) 000072 1787839413.936631 read 0x004645 thru 0x0053ef (0xdab bytes) 000073 1787839413.937218 fallocate 0x014806 thru 0x0242da (0xfad5 bytes) 000074 1787839413.945092 mapwrite 0x01baca thru 0x0276b6 (0xbbed bytes) 000075 1787839413.999414 trunc from 0x03c1f9 to 0x03ffff (0x3e07 bytes) 000076 1787839414.015234 read 0x023734 thru 0x02e5b0 (0xae7d bytes) 000077 1787839414.018254 mapwrite 0x03cf7e thru 0x03ef61 (0x1fe4 bytes) 000078 1787839414.054320 trunc from 0x01cf0a to 0x028bc1 (0xbcb8 bytes) 000079 1787839414.063562 read 0x00f6d4 thru 0x019433 (0x9d60 bytes) 000080 1787839414.064346 punch from 0x01801f to 0x024afd (0xcadf bytes) 000081 1787839414.075137 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x327d8d41 (stale) Mirror 3: 0x5dd0462a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6176f4f8 Mirror 4: 0x6176f4f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000082 1787839415.287550 fallocate 0x016efa thru 0x023ed0 (0xcfd7 bytes) 000083 1787839415.321636 fallocate 0x00b49b thru 0x016784 (0xb2ea bytes) 000084 1787839415.327429 read 0x0152e8 thru 0x02314f (0xde68 bytes) 000085 1787839415.335457 write 0x030011 thru 0x0381ce (0x81be bytes) 000086 1787839415.348173 write 0x02b3f1 thru 0x02ba22 (0x632 bytes) 000088 1787839415.354159 fallocate 0x0018db thru 0x0113eb (0xfb11 bytes) 000089 1787839415.358840 trunc from 0x040000 to 0x00fa95 (0x3056a bytes) 000090 1787839415.422588 trunc from 0x00a5be to 0x00fa95 (0x54d8 bytes) 000091 1787839415.429168 trunc from 0x00fa96 to 0x00ab93 (0x4f02 bytes) 000092 1787839415.462280 fallocate 0x03c723 thru 0x03ffff (0x38dd bytes) 000093 1787839415.469358 write 0x036511 thru 0x03ffff (0x9aef bytes) 000094 1787839415.472464 trunc from 0x023f93 to 0x024ef7 (0xf65 bytes) 000095 1787839415.478464 fallocate 0x027f1f thru 0x02f53d (0x761f bytes) 000096 1787839415.488296 read 0x01044e thru 0x01de7e (0xda31 bytes) 000097 1787839415.501878 write 0x03ecce thru 0x03ffff (0x1332 bytes) 000098 1787839415.503392 punch from 0x0252fd to 0x02c3fa (0x70fe bytes) 000100 1787839415.508739 mapwrite 0x03065f thru 0x03c622 (0xbfc4 bytes) 000101 1787839415.550859 trunc from 0x040000 to 0x01a569 (0x25a96 bytes) 000102 1787839415.566461 mapwrite 0x019425 thru 0x021a07 (0x85e3 bytes) 000103 1787839415.634149 trunc from 0x021a08 to 0x009551 (0x184b6 bytes) 000104 1787839415.655159 write 0x030ff8 thru 0x037afb (0x6b04 bytes) 000106 1787839415.658826 trunc from 0x02223c to 0x026810 (0x45d5 bytes) 000107 1787839415.667326 read 0x007085 thru 0x01128f (0xa20b bytes) 000108 1787839415.677654 write 0x019bb8 thru 0x026b92 (0xcfdb bytes) 000109 1787839415.685514 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37afc): Mirror 2: 0x6b838ad8 (stale) Mirror 4: 0x4459290 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37afc): Mirror 2: 0x6b838ad8 (stale) Mirror 4: 0x4459290 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37afc: skipped 000111 1787839416.121187 mapwrite 0x017924 thru 0x01a6cf (0x2dac bytes) 000112 1787839416.199109 mapwrite 0x012ce4 thru 0x014c8d (0x1faa bytes) 000113 1787839416.229478 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37afc): Mirror 4: 0x4459290 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37afc): Mirror 4: 0x4459290 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37afc: skipped 000114 1787839416.547533 mapwrite 0x03c292 thru 0x03ffff (0x3d6e bytes) 000115 1787839416.627395 trunc from 0x005d7e to 0x012f77 (0xd1fa bytes) 000116 1787839416.646697 read 0x03b1a9 thru 0x03ffff (0x4e57 bytes) 000119 1787839416.653884 trunc from 0x01759c to 0x01bddb (0x4840 bytes) 000120 1787839416.670870 trunc from 0x02607e to 0x02eb41 (0x8ac4 bytes) 000121 1787839416.677195 trunc from 0x02762c to 0x036aea (0xf4bf bytes) 000122 1787839416.718567 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) 000123 1787839416.893278 trunc from 0x037b92 to 0x03f58e (0x79fd bytes) 000124 1787839416.897176 write 0x00c05a thru 0x01af5c (0xef03 bytes) 000125 1787839416.912561 fallocate 0x01b81a thru 0x01c6d5 (0xebc bytes) 000126 1787839416.916170 mapwrite 0x03ec4a thru 0x03ffff (0x13b6 bytes) 000127 1787839416.942129 fallocate 0x00c377 thru 0x0123b6 (0x6040 bytes) 000128 1787839416.945354 mapwrite 0x0187da thru 0x024aa9 (0xc2d0 bytes) 000129 1787839416.991238 mapread 0x01bbc7 thru 0x024ca7 (0x90e1 bytes) 000130 1787839416.992752 punch from 0x0392c5 to 0x03d3db (0x4117 bytes) 000131 1787839417.002094 mapwrite 0x017a3c thru 0x024851 (0xce16 bytes) 000132 1787839417.033629 read 0x033ae3 thru 0x0396e6 (0x5c04 bytes) 000133 1787839417.047300 trunc from 0x040000 to 0x01e480 (0x21b7f bytes) 000134 1787839417.082858 trunc from 0x002fa1 to 0x00eeb6 (0xbf16 bytes) 000135 1787839417.090644 read 0x00841a thru 0x008e8c (0xa73 bytes) 000136 1787839417.110676 mapwrite 0x028f86 thru 0x030ef2 (0x7f6d bytes) 000137 1787839417.249430 mapwrite 0x02ba65 thru 0x0304ba (0x4a56 bytes) 000138 1787839417.286228 fallocate 0x02a8f3 thru 0x031757 (0x6e65 bytes) 000139 1787839417.300477 fallocate 0x02a993 thru 0x037082 (0xc6f0 bytes) 000140 1787839417.331040 read 0x026598 thru 0x030c27 (0xa690 bytes) 000142 1787839417.337383 mapread 0x011a2f thru 0x01f8f7 (0xdec9 bytes) 000143 1787839417.352375 punch from 0x01a4c4 to 0x01db03 (0x3640 bytes) 000144 1787839417.369498 trunc from 0x025939 to 0x0280f5 (0x27bd bytes) 000145 1787839417.381092 trunc from 0x018de3 to 0x01efbf (0x61dd bytes) 000146 1787839417.397602 write 0x00934d thru 0x012c35 (0x98e9 bytes) 000147 1787839417.399978 punch from 0x030926 to 0x031757 (0xe32 bytes) 000148 1787839417.412131 fallocate 0x000ece thru 0x00aba8 (0x9cdb bytes) 000149 1787839417.426134 MIRROR_EXTEND to 2 mirrors 000151 1787839418.692281 mapwrite 0x022a8b thru 0x026e9f (0x4415 bytes) 000152 1787839418.903401 mapwrite 0x03195a thru 0x03fbca (0xe271 bytes) 000153 1787839419.063508 punch from 0x02a89f to 0x02b94c (0x10ae bytes) 000154 1787839419.078494 trunc from 0x03fbcb to 0x03a173 (0x5a57 bytes) 000155 1787839419.099384 trunc from 0x03a174 to 0x00353a (0x36c39 bytes) 000156 1787839419.140544 trunc from 0x000255 to 0x00353a (0x32e6 bytes) 000157 1787839419.147346 fallocate 0x021fa2 thru 0x02e093 (0xc0f2 bytes) 000158 1787839419.155133 punch from 0x001805 to 0x001e86 (0x682 bytes) 000159 1787839419.167894 MIRROR_RESYNC 2 mirrors 000160 1787839419.845616 mapwrite 0x03b3f2 thru 0x03ffff (0x4c0e bytes) 000161 1787839419.941493 trunc from 0x040000 to 0x0388ba (0x7745 bytes) 000164 1787839420.019129 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x388bb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388bb): Mirror 2: 0x8f45004f (stale) CRC-32 checksum value for chunk [0, 0x388bb): Mirror 1: 0x8f45004f Mirror 3: 0x8f45004f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388bb: skipped 000165 1787839421.270443 punch from 0x0254e8 to 0x02c1ea (0x6d03 bytes) 000166 1787839421.299663 mapread 0x011ab9 thru 0x018c68 (0x71b0 bytes) 000167 1787839421.322385 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x388bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388bb): Mirror 2: 0x8f45004f (stale) Mirror 3: 0x8f45004f (stale) CRC-32 checksum value for chunk [0, 0x388bb): Mirror 1: 0x8f45004f Mirror 4: 0x8f45004f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388bb: skipped 000168 1787839422.612594 punch from 0x0137b3 to 0x01b497 (0x7ce5 bytes) 000169 1787839422.664978 mapread 0x0348d7 thru 0x0388ba (0x3fe4 bytes) 000170 1787839422.701169 write 0x023638 thru 0x023d05 (0x6ce bytes) 000171 1787839422.722724 fallocate 0x008183 thru 0x009998 (0x1816 bytes) 000172 1787839422.733609 read 0x01699b thru 0x024acb (0xe131 bytes) 000173 1787839422.753119 punch from 0x000f40 to 0x002d81 (0x1e42 bytes) 000174 1787839422.781113 mapread 0x0005aa thru 0x00711f (0x6b76 bytes) 000175 1787839422.825093 fallocate 0x01b9a4 thru 0x01d7e0 (0x1e3d bytes) 000176 1787839422.854335 MIRROR_RESYNC 4 mirrors 000177 1787839424.468795 MIRROR_EXTEND to 5 mirrors 000179 1787839425.480793 punch from 0x021b8c to 0x0220f4 (0x569 bytes) 000180 1787839425.521245 trunc from 0x0388bb to 0x007c92 (0x30c28 bytes) 000181 1787839425.565844 punch from 0x000dff to 0x001809 (0xa0b bytes) 000183 1787839425.573259 trunc from 0x007c93 to 0x01acef (0x1305d bytes) 000184 1787839425.602013 read 0x0046ba thru 0x00e338 (0x9c7f bytes) 000185 1787839425.627710 fallocate 0x028c2e thru 0x02ae45 (0x2218 bytes) 000186 1787839425.645935 mapread 0x004e6e thru 0x00ac07 (0x5d9a bytes) 000187 1787839425.656461 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1acf0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1acf0): Mirror 2: 0xc1a7de89 (stale) Mirror 3: 0xc1a7de89 (stale) Mirror 4: 0xc1a7de89 (stale) Mirror 5: 0xc1a7de89 (stale) CRC-32 checksum value for chunk [0, 0x1acf0): Mirror 1: 0xc1a7de89 Mirror 6: 0xc1a7de89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1acf0: skipped 000188 1787839427.397257 write 0x037b72 thru 0x03d9be (0x5e4d bytes) 000189 1787839427.473446 fallocate 0x028406 thru 0x03737d (0xef78 bytes) 000190 1787839427.481465 mapread 0x00f4a5 thru 0x01e4c3 (0xf01f bytes) 000191 1787839427.513470 trunc from 0x03d9bf to 0x022fa8 (0x1aa16 bytes) 000192 1787839427.552699 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 2: 0xf74b3529 (stale) Mirror 3: 0xf74b3529 (stale) Mirror 4: 0xf74b3529 (stale) Mirror 5: 0xf74b3529 (stale) Mirror 6: 0xf74b3529 (stale) CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 1: 0xf74b3529 Mirror 7: 0xf74b3529 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fa9: skipped 000193 1787839429.273961 read 0x018621 thru 0x022fa8 (0xa988 bytes) 000195 1787839429.288052 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 2: 0xf74b3529 (stale) Mirror 3: 0xf74b3529 (stale) Mirror 5: 0xf74b3529 (stale) Mirror 6: 0xf74b3529 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 2: 0xf74b3529 (stale) Mirror 3: 0xf74b3529 (stale) Mirror 5: 0xf74b3529 (stale) Mirror 6: 0xf74b3529 (stale) CRC-32 checksum value for chunk [0, 0x22fa9): Mirror 1: 0xf74b3529 Mirror 7: 0xf74b3529 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fa9: skipped 000196 1787839429.969829 write 0x03101d thru 0x03c1ff (0xb1e3 bytes) 000197 1787839430.061669 mapwrite 0x00f8d2 thru 0x010b34 (0x1263 bytes) 000198 1787839430.191720 mapread 0x0207e2 thru 0x0256ad (0x4ecc bytes) 000199 1787839430.223693 trunc from 0x03c200 to 0x035dcf (0x6430 bytes) 000200 1787839430.275158 mapread 0x02d8c2 thru 0x035dcf (0x850e bytes) 000201 1787839430.295032 mapread 0x017017 thru 0x01efe4 (0x7fce bytes) 000202 1787839430.352411 fallocate 0x021185 thru 0x02b08f (0x9f0b bytes) 000203 1787839430.385886 trunc from 0x01101f to 0x014218 (0x31fa bytes) 000204 1787839430.396377 fallocate 0x008a27 thru 0x014e0f (0xc3e9 bytes) 000205 1787839430.410159 read 0x027d05 thru 0x02f503 (0x77ff bytes) 000206 1787839430.410698 punch from 0x01a352 to 0x028c78 (0xe927 bytes) 000207 1787839430.451042 punch from 0x019760 to 0x02236f (0x8c10 bytes) 000208 1787839430.503249 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35dd0): Mirror 2: 0xdc18fa69 (stale) Mirror 3: 0xdc18fa69 (stale) Mirror 6: 0x42ad4d3e (stale) Mirror 7: 0x42ad4d3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35dd0): Mirror 2: 0xdc18fa69 (stale) Mirror 3: 0xdc18fa69 (stale) Mirror 6: 0x42ad4d3e (stale) Mirror 7: 0x42ad4d3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35dd0: skipped 000210 1787839431.141568 trunc from 0x035dd0 to 0x026df7 (0xefd8 bytes) 000211 1787839431.162522 punch from 0x006c9c to 0x00d252 (0x65b7 bytes) 000212 1787839431.172712 trunc from 0x026df8 to 0x02e1b4 (0x73bd bytes) 000213 1787839431.191300 write 0x000499 thru 0x00794d (0x74b5 bytes) 000215 1787839431.215066 MIRROR_RESYNC 5 mirrors 000216 1787839433.129862 mapwrite 0x01547d thru 0x0205f0 (0xb174 bytes) 000217 1787839433.275174 trunc from 0x02e1b5 to 0x00f9df (0x1e7d5 bytes) 000218 1787839433.320041 punch from 0x008aa9 to 0x00f9df (0x6f37 bytes) 000219 1787839433.331813 fallocate 0x01d3e2 thru 0x0278a3 (0xa4c2 bytes) 000221 1787839433.360762 trunc from 0x0278a4 to 0x016884 (0x1101f bytes) 000222 1787839433.408741 punch from 0x00940b to 0x00dcab (0x48a1 bytes) 000223 1787839433.425852 mapwrite 0x0272e2 thru 0x032379 (0xb098 bytes) 000224 1787839433.579992 read 0x00c0d8 thru 0x019482 (0xd3ab bytes) 000225 1787839433.599046 read 0x00bebc thru 0x016666 (0xa7ab bytes) 000226 1787839433.637325 MIRROR_RESYNC 5 mirrors 000227 1787839435.538846 mapwrite 0x02c59a thru 0x0353b9 (0x8e20 bytes) 000228 1787839435.764772 fallocate 0x0188de thru 0x021710 (0x8e33 bytes) 000230 1787839435.774416 trunc from 0x017600 to 0x018bb4 (0x15b5 bytes) 000231 1787839435.781324 write 0x02914c thru 0x02efa3 (0x5e58 bytes) 000232 1787839435.782691 mapwrite 0x017716 thru 0x01c406 (0x4cf1 bytes) 000233 1787839435.825207 write 0x030964 thru 0x033297 (0x2934 bytes) 000235 1787839435.826838 mapwrite 0x0363fa thru 0x03bf9f (0x5ba6 bytes) 000236 1787839435.907403 punch from 0x037ee6 to 0x03bf9f (0x40ba bytes) 000237 1787839435.915104 trunc from 0x03bfa0 to 0x014e6d (0x27132 bytes) 000238 1787839435.944264 punch from 0x0040da to 0x01271c (0xe643 bytes) 000240 1787839435.959214 trunc from 0x010c4c to 0x013d2b (0x30e0 bytes) 000242 1787839435.977988 mapwrite 0x01acbb thru 0x0262f4 (0xb63a bytes) 000243 1787839436.094623 read 0x016fe9 thru 0x02481f (0xd837 bytes) 000244 1787839436.096023 read 0x023555 thru 0x0262f4 (0x2da0 bytes) 000246 1787839436.104386 punch from 0x00214a to 0x00cf51 (0xae08 bytes) 000247 1787839436.116514 trunc from 0x00c0d3 to 0x010c99 (0x4bc7 bytes) 000248 1787839436.128294 mapwrite 0x038c14 thru 0x03b8f0 (0x2cdd bytes) 000250 1787839436.200603 read 0x020750 thru 0x02e619 (0xdeca bytes) 000251 1787839436.201594 punch from 0x01dc33 to 0x029f90 (0xc35e bytes) 000252 1787839436.214866 read 0x029259 thru 0x02a665 (0x140d bytes) 000253 1787839436.231888 trunc from 0x03b8f1 to 0x02bcb3 (0xfc3d bytes) 000254 1787839436.263649 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2bcb4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcb4): Mirror 2: 0xfc23f236 (stale) Mirror 3: 0xfc23f236 (stale) Mirror 6: 0xfc23f236 (stale) Mirror 7: 0xfc23f236 (stale) CRC-32 checksum value for chunk [0, 0x2bcb4): Mirror 1: 0xcde9f235 Mirror 8: 0xcde9f235 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcb4: skipped 000255 1787839437.855239 punch from 0x013280 to 0x0217a0 (0xe521 bytes) 000256 1787839437.929491 read 0x01077e thru 0x01234f (0x1bd2 bytes) 000257 1787839437.937279 trunc from 0x02bcb4 to 0x021064 (0xac4f bytes) 000258 1787839437.999361 fallocate 0x01ddd2 thru 0x02597d (0x7bac bytes) 000259 1787839438.006396 mapread 0x0215ca thru 0x02597d (0x43b4 bytes) 000260 1787839438.037093 trunc from 0x008683 to 0x010421 (0x7d9f bytes) 000261 1787839438.043025 read 0x00ec83 thru 0x0100ef (0x146d bytes) 000262 1787839438.056630 trunc from 0x02597e to 0x039b27 (0x141aa bytes) 000263 1787839438.088399 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b28): Mirror 2: 0xffae691c (stale) Mirror 3: 0xffae691c (stale) Mirror 7: 0xffae691c (stale) Mirror 8: 0xeacf5c02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b28): Mirror 2: 0xffae691c (stale) Mirror 3: 0xffae691c (stale) Mirror 7: 0xffae691c (stale) Mirror 8: 0xeacf5c02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b28: skipped 000264 1787839438.744668 MIRROR_RESYNC 5 mirrors 000265 1787839439.824031 punch from 0x0214fd to 0x0279bc (0x64c0 bytes) 000266 1787839439.866240 mapread 0x013ab0 thru 0x0231a2 (0xf6f3 bytes) 000267 1787839439.902335 punch from 0x02f98b to 0x039b27 (0xa19d bytes) 000268 1787839439.908030 mapread 0x00fd5d thru 0x014210 (0x44b4 bytes) 000269 1787839439.913036 trunc from 0x039b28 to 0x00e366 (0x2b7c1 bytes) 000270 1787839439.953148 fallocate 0x01fcc0 thru 0x027bdf (0x7f20 bytes) 000271 1787839439.961222 trunc from 0x006cc5 to 0x00e366 (0x76a2 bytes) 000272 1787839439.973218 MIRROR_RESYNC 5 mirrors 000274 1787839441.160593 trunc from 0x00e367 to 0x03ac08 (0x2c8a2 bytes) 000276 1787839441.247152 mapread 0x01c903 thru 0x01f118 (0x2816 bytes) 000277 1787839441.301654 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 2: 0x9392d32f (stale) Mirror 7: 0x9392d32f (stale) Mirror 8: 0x9392d32f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 2: 0x9392d32f (stale) Mirror 7: 0x9392d32f (stale) Mirror 8: 0x9392d32f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac09: skipped 000278 1787839441.876997 write 0x005f45 thru 0x00c22e (0x62ea bytes) 000279 1787839441.898283 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 7: 0x9392d32f (stale) Mirror 8: 0x9392d32f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 7: 0x9392d32f (stale) Mirror 8: 0x9392d32f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac09: skipped 000280 1787839442.352658 mapread 0x009f9d thru 0x00f791 (0x57f5 bytes) 000281 1787839442.379573 punch from 0x01333b to 0x015b25 (0x27eb bytes) 000282 1787839442.395006 mapread 0x01b3cd thru 0x025ee4 (0xab18 bytes) 000283 1787839442.431399 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 7: 0x9392d32f (stale) Mirror 8: 0x9392d32f (stale) CRC-32 checksum value for chunk [0, 0x3ac09): Mirror 1: 0xe5393334 Mirror 9: 0xe5393334 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac09: skipped 000284 1787839443.611048 punch from 0x030576 to 0x03ac08 (0xa693 bytes) 000285 1787839443.678744 punch from 0x006fdb to 0x00c4ea (0x5510 bytes) 000287 1787839443.698549 MIRROR_RESYNC 4 mirrors 000288 1787839445.000309 mapwrite 0x035a0c thru 0x03ffff (0xa5f4 bytes) 000289 1787839445.135190 trunc from 0x01b37c to 0x028404 (0xd089 bytes) 000290 1787839445.139150 punch from 0x030f01 to 0x0361a6 (0x52a6 bytes) 000291 1787839445.146823 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9975306 (stale) Mirror 9: 0x9975306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9975306 (stale) Mirror 9: 0x9975306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000292 1787839445.509610 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9975306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9975306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1787839445.864145 mapwrite 0x009a71 thru 0x00a209 (0x799 bytes) 000294 1787839445.890364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9975306 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa0a0a105 Mirror 10: 0xa0a0a105 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1787839447.587615 mapwrite 0x03db03 thru 0x03ffff (0x24fd bytes) 000296 1787839447.738104 fallocate 0x02a58e thru 0x030700 (0x6173 bytes) 000297 1787839447.745232 trunc from 0x040000 to 0x02df86 (0x12079 bytes) 000298 1787839447.799275 read 0x0130e6 thru 0x018e2e (0x5d49 bytes) 000299 1787839447.815953 mapread 0x01058c thru 0x015ed0 (0x5945 bytes) 000301 1787839447.833292 fallocate 0x02471a thru 0x034693 (0xff7a bytes) 000302 1787839447.840507 trunc from 0x02df87 to 0x013132 (0x1ae54 bytes) 000303 1787839447.865054 read 0x00ff86 thru 0x013132 (0x31ad bytes) 000304 1787839447.866163 read 0x004069 thru 0x008aa4 (0x4a3c bytes) 000305 1787839447.884189 punch from 0x003cd6 to 0x00b527 (0x7852 bytes) 000306 1787839447.892811 write 0x0381d3 thru 0x0385b9 (0x3e7 bytes) 000307 1787839447.894846 trunc from 0x0385ba to 0x02ef25 (0x9694 bytes) 000308 1787839447.925339 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ef26): Mirror 10: 0xa981240a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef26): Mirror 10: 0xa981240a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef26: skipped 000309 1787839448.277035 write 0x01e984 thru 0x01e9c8 (0x45 bytes) 000310 1787839448.288942 write 0x00f939 thru 0x01b1d8 (0xb8a0 bytes) 000311 1787839448.300549 write 0x00d9b4 thru 0x00fd20 (0x236d bytes) 000312 1787839448.306243 mapread 0x020576 thru 0x029929 (0x93b4 bytes) 000313 1787839448.326849 read 0x005709 thru 0x013e04 (0xe6fc bytes) 000314 1787839448.344780 fallocate 0x018c71 thru 0x020567 (0x78f7 bytes) 000315 1787839448.350771 trunc from 0x00d1c7 to 0x013b8e (0x69c8 bytes) 000316 1787839448.374153 trunc from 0x02ef26 to 0x00236d (0x2cbb8 bytes) 000317 1787839448.411582 MIRROR_RESYNC 2 mirrors 000318 1787839449.111554 punch from 0x0014a7 to 0x00236d (0xec7 bytes) 000319 1787839449.194834 trunc from 0x000837 to 0x00236d (0x1b37 bytes) 000320 1787839449.200036 fallocate 0x01ee5c thru 0x02746a (0x860f bytes) 000321 1787839449.213038 punch from 0x001b49 to 0x00236d (0x825 bytes) 000322 1787839449.225788 mapread 0x000d21 thru 0x00236d (0x164d bytes) 000323 1787839449.267249 read 0x00217b thru 0x00236d (0x1f3 bytes) 000324 1787839449.268181 punch from 0x000897 to 0x00236d (0x1ad7 bytes) 000325 1787839449.282618 fallocate 0x01893f thru 0x0201aa (0x786c bytes) 000326 1787839449.287016 punch from 0x001783 to 0x00236d (0xbeb bytes) 000327 1787839449.295170 mapwrite 0x020299 thru 0x02c8e8 (0xc650 bytes) 000328 1787839449.470933 read 0x01e3f1 thru 0x02881e (0xa42e bytes) 000330 1787839449.471940 trunc from 0x01db53 to 0x024396 (0x6844 bytes) 000331 1787839449.479864 mapread 0x008042 thru 0x01782c (0xf7eb bytes) 000332 1787839449.493499 punch from 0x0287ec to 0x02c7dc (0x3ff1 bytes) 000333 1787839449.500502 punch from 0x023c7f to 0x029cfc (0x607e bytes) 000334 1787839449.507804 punch from 0x001d60 to 0x009b42 (0x7de3 bytes) 000336 1787839449.515522 mapwrite 0x00443c thru 0x006369 (0x1f2e bytes) 000337 1787839449.550547 trunc from 0x0138a9 to 0x0192d0 (0x5a28 bytes) 000338 1787839449.557691 punch from 0x01b119 to 0x027c07 (0xcaef bytes) 000339 1787839449.563566 punch from 0x0053ed to 0x009ee1 (0x4af5 bytes) 000340 1787839449.571892 fallocate 0x01826b thru 0x01b6e5 (0x347b bytes) 000341 1787839449.577847 punch from 0x025404 to 0x02c8e8 (0x74e5 bytes) 000342 1787839449.587757 punch from 0x018c46 to 0x02755f (0xe91a bytes) 000343 1787839449.593304 trunc from 0x02c8e9 to 0x00f129 (0x1d7bf bytes) 000344 1787839449.625895 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf12a): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf12a): Mirror 10: 0xbe300f6a (stale) CRC-32 checksum value for chunk [0, 0xf12a): Mirror 1: 0xa2510ca3 Mirror 11: 0xa2510ca3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf12a: skipped 000345 1787839450.451329 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf12a): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf12a): Mirror 10: 0xbe300f6a (stale) CRC-32 checksum value for chunk [0, 0xf12a): Mirror 1: 0xa2510ca3 Mirror 11: 0xa2510ca3 Mirror 12: 0xa2510ca3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf12a: skipped 000346 1787839451.647623 fallocate 0x012370 thru 0x01e996 (0xc627 bytes) 000347 1787839451.714688 mapwrite 0x002bd0 thru 0x00fe91 (0xd2c2 bytes) 000348 1787839451.870539 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfe92): Mirror 11: 0xd533dfe0 (stale) Mirror 12: 0xd533dfe0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfe92): Mirror 11: 0xd533dfe0 (stale) Mirror 12: 0xd533dfe0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfe92: skipped 000349 1787839452.241079 write 0x0279db thru 0x02c727 (0x4d4d bytes) 000350 1787839452.250774 punch from 0x00fd20 to 0x01ef69 (0xf24a bytes) 000351 1787839452.258723 mapwrite 0x012d91 thru 0x01b4a9 (0x8719 bytes) 000352 1787839452.301468 MIRROR_RESYNC 3 mirrors 000353 1787839453.162274 write 0x016c67 thru 0x01f4af (0x8849 bytes) 000354 1787839453.203961 write 0x02bf23 thru 0x02c4c4 (0x5a2 bytes) 000355 1787839453.213944 trunc from 0x011189 to 0x013dba (0x2c32 bytes) 000356 1787839453.219097 read 0x01a94b thru 0x01d857 (0x2f0d bytes) 000358 1787839453.220378 mapwrite 0x001bd2 thru 0x00adca (0x91f9 bytes) 000359 1787839453.263289 mapread 0x006230 thru 0x007946 (0x1717 bytes) 000360 1787839453.264301 trunc from 0x02c728 to 0x005820 (0x26f07 bytes) 000361 1787839453.304559 fallocate 0x028442 thru 0x02f753 (0x7312 bytes) 000362 1787839453.310435 mapread 0x00196b thru 0x005820 (0x3eb6 bytes) 000363 1787839453.316266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5821): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5821): Mirror 11: 0x5b2c0c1d (stale) Mirror 12: 0x5b2c0c1d (stale) CRC-32 checksum value for chunk [0, 0x5821): Mirror 1: 0xcfa2cdcd Mirror 13: 0xcfa2cdcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5821: skipped 000364 1787839453.955702 fallocate 0x01c57c thru 0x025c15 (0x969a bytes) 000365 1787839454.002636 MIRROR_RESYNC 4 mirrors 000366 1787839454.834792 punch from 0x0030da to 0x0058bd (0x27e4 bytes) 000367 1787839454.894207 mapwrite 0x01b02d thru 0x021164 (0x6138 bytes) 000368 1787839454.956880 mapwrite 0x035b50 thru 0x03dd8f (0x8240 bytes) 000369 1787839455.078993 fallocate 0x001359 thru 0x00de37 (0xcadf bytes) 000370 1787839455.089086 write 0x020b37 thru 0x021082 (0x54c bytes) 000371 1787839455.092178 trunc from 0x02cef8 to 0x03535a (0x8463 bytes) 000373 1787839455.102497 mapread 0x003721 thru 0x003c2f (0x50f bytes) 000374 1787839455.120824 fallocate 0x032391 thru 0x03b487 (0x90f7 bytes) 000375 1787839455.128865 trunc from 0x02146f to 0x02b6a5 (0xa237 bytes) 000376 1787839455.154444 punch from 0x018094 to 0x0270ff (0xf06c bytes) 000377 1787839455.168163 read 0x034079 thru 0x03dd8f (0x9d17 bytes) 000378 1787839455.185031 punch from 0x0381a0 to 0x03dd8f (0x5bf0 bytes) 000379 1787839455.194543 trunc from 0x0385b2 to 0x03908f (0xade bytes) 000380 1787839455.218297 mapread 0x0241ce thru 0x03111b (0xcf4e bytes) 000381 1787839455.247003 read 0x016531 thru 0x0167d4 (0x2a4 bytes) 000383 1787839455.250731 read 0x035950 thru 0x03cb61 (0x7212 bytes) 000384 1787839455.274217 trunc from 0x03dd90 to 0x024df5 (0x18f9a bytes) 000385 1787839455.301892 write 0x00f9c9 thru 0x01e9a2 (0xefda bytes) 000386 1787839455.317459 punch from 0x021843 to 0x024731 (0x2eef bytes) 000387 1787839455.331220 punch from 0x021334 to 0x024df5 (0x3ac2 bytes) 000388 1787839455.340514 punch from 0x013469 to 0x019c3b (0x67d3 bytes) 000389 1787839455.372510 trunc from 0x024df6 to 0x01fbd4 (0x5221 bytes) 000390 1787839455.413843 trunc from 0x01c3d5 to 0x01fbd4 (0x3800 bytes) 000391 1787839455.426024 write 0x036c2e thru 0x03f502 (0x88d5 bytes) 000392 1787839455.432734 write 0x012df5 thru 0x0140cf (0x12db bytes) 000393 1787839455.454397 fallocate 0x013f62 thru 0x0233fd (0xf49c bytes) 000394 1787839455.461100 trunc from 0x03f503 to 0x003761 (0x3bda1 bytes) 000396 1787839455.504383 fallocate 0x01f40e thru 0x02ef62 (0xfb55 bytes) 000397 1787839455.527586 trunc from 0x02ef63 to 0x025089 (0x9ed9 bytes) 000398 1787839455.551074 fallocate 0x027827 thru 0x032bbe (0xb398 bytes) 000399 1787839455.557174 fallocate 0x03c913 thru 0x03ffff (0x36ed bytes) 000400 1787839455.563200 read 0x02f951 thru 0x038716 (0x8dc6 bytes) 000401 1787839455.575872 trunc from 0x040000 to 0x01573d (0x2a8c2 bytes) 000402 1787839455.595553 mapwrite 0x00480d thru 0x006169 (0x195d bytes) 000403 1787839455.609481 punch from 0x00d534 to 0x0115cc (0x4099 bytes) 000404 1787839455.618296 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1573e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1573e): Mirror 11: 0x696a1dee (stale) Mirror 12: 0x696a1dee (stale) Mirror 13: 0x696a1dee (stale) CRC-32 checksum value for chunk [0, 0x1573e): Mirror 1: 0x89ab05c9 Mirror 14: 0x89ab05c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1573e: skipped 000405 1787839456.655702 fallocate 0x02833c thru 0x034175 (0xbe3a bytes) 000406 1787839456.702819 read 0x0136ae thru 0x01573d (0x2090 bytes) 000407 1787839456.709755 read 0x00e470 thru 0x01573d (0x72ce bytes) 000409 1787839456.721322 fallocate 0x019d76 thru 0x01d012 (0x329d bytes) 000410 1787839456.727793 write 0x01a5e7 thru 0x02180b (0x7225 bytes) 000411 1787839456.744700 punch from 0x001d63 to 0x00d177 (0xb415 bytes) 000412 1787839456.749997 fallocate 0x008f3f thru 0x017aa4 (0xeb66 bytes) 000414 1787839456.758293 trunc from 0x02180c to 0x02b54d (0x9d42 bytes) 000415 1787839456.809315 read 0x00709f thru 0x014f60 (0xdec2 bytes) 000416 1787839456.831255 trunc from 0x02b54e to 0x031ba1 (0x6654 bytes) 000417 1787839456.846402 mapread 0x02c6d8 thru 0x031ba1 (0x54ca bytes) 000418 1787839456.855762 read 0x003396 thru 0x00eec6 (0xbb31 bytes) 000419 1787839456.860809 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31ba2): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ba2): Mirror 11: 0x573e6b82 (stale) Mirror 12: 0x573e6b82 (stale) Mirror 13: 0x573e6b82 (stale) Mirror 14: 0x35506a51 (stale) CRC-32 checksum value for chunk [0, 0x31ba2): Mirror 1: 0xf9b36836 Mirror 15: 0xf9b36836 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ba2: skipped 000420 1787839457.953844 trunc from 0x02b733 to 0x031ba1 (0x646f bytes) 000421 1787839458.007636 write 0x013f94 thru 0x0191d6 (0x5243 bytes) 000422 1787839458.028856 mapwrite 0x032eee thru 0x03d961 (0xaa74 bytes) 000423 1787839458.158057 mapread 0x02e0b7 thru 0x0348fb (0x6845 bytes) 000424 1787839458.162181 mapwrite 0x013c39 thru 0x01f7cf (0xbb97 bytes) 000425 1787839458.235140 trunc from 0x03a09e to 0x03bbbb (0x1b1e bytes) 000426 1787839458.263301 punch from 0x0162a1 to 0x01ec5f (0x89bf bytes) 000427 1787839458.273851 mapwrite 0x02e1ce thru 0x03b0de (0xcf11 bytes) 000428 1787839458.347591 mapwrite 0x011e1a thru 0x01d6f3 (0xb8da bytes) 000429 1787839458.436033 trunc from 0x03d962 to 0x02898e (0x14fd3 bytes) 000430 1787839458.466819 mapread 0x0237f0 thru 0x02424f (0xa60 bytes) 000431 1787839458.487420 write 0x00129f thru 0x00f8bd (0xe61f bytes) 000432 1787839458.503552 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2898f): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2898f): Mirror 11: 0xb9a13a78 (stale) Mirror 12: 0xb9a13a78 (stale) Mirror 13: 0xb9a13a78 (stale) Mirror 14: 0xfe0fb928 (stale) Mirror 15: 0x83166ca1 (stale) CRC-32 checksum value for chunk [0, 0x2898f): Mirror 1: 0x3850f558 Mirror 16: 0x3850f558 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2898f: skipped 000433 1787839459.981820 mapread 0x01f83f thru 0x02898e (0x9150 bytes) 000434 1787839460.010238 write 0x02b052 thru 0x02f7ba (0x4769 bytes) 000435 1787839460.083663 mapwrite 0x018a02 thru 0x01fe46 (0x7445 bytes) 000436 1787839460.129144 mapread 0x007171 thru 0x00cc41 (0x5ad1 bytes) 000437 1787839460.150651 punch from 0x0259be to 0x02872c (0x2d6f bytes) 000438 1787839460.158855 punch from 0x01b7c2 to 0x01f022 (0x3861 bytes) 000439 1787839460.173657 MIRROR_RESYNC 7 mirrors 000440 1787839462.513529 MIRROR_EXTEND to 8 mirrors 000441 1787839463.626110 write 0x032b54 thru 0x03ffff (0xd4ac bytes) 000442 1787839463.703030 mapread 0x00abd0 thru 0x00eb39 (0x3f6a bytes) 000443 1787839463.740773 write 0x011b72 thru 0x0145b7 (0x2a46 bytes) 000444 1787839463.741811 mapwrite 0x00ee12 thru 0x0131e9 (0x43d8 bytes) 000446 1787839463.774756 trunc from 0x010478 to 0x0149ab (0x4534 bytes) 000447 1787839463.781678 trunc from 0x0121d5 to 0x013c31 (0x1a5d bytes) 000448 1787839463.790355 mapwrite 0x002ffd thru 0x00cd84 (0x9d88 bytes) 000449 1787839463.850468 punch from 0x01daa7 to 0x021f0d (0x4467 bytes) 000450 1787839463.857747 punch from 0x0301b3 to 0x03dbb2 (0xda00 bytes) 000451 1787839463.870122 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x4463c7e6 (stale) Mirror 13: 0x4463c7e6 (stale) Mirror 14: 0x4463c7e6 (stale) Mirror 15: 0x4463c7e6 (stale) Mirror 16: 0x4463c7e6 (stale) Mirror 17: 0x4463c7e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x4463c7e6 (stale) Mirror 13: 0x4463c7e6 (stale) Mirror 14: 0x4463c7e6 (stale) Mirror 15: 0x4463c7e6 (stale) Mirror 16: 0x4463c7e6 (stale) Mirror 17: 0x4463c7e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000452 1787839464.415231 fallocate 0x02bb73 thru 0x02c605 (0xa93 bytes) 000453 1787839464.423358 trunc from 0x02f825 to 0x036319 (0x6af5 bytes) 000454 1787839464.430424 trunc from 0x040000 to 0x021f50 (0x1e0af bytes) 000455 1787839464.472274 trunc from 0x0109ee to 0x01423e (0x3851 bytes) 000456 1787839464.488052 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x21f51): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21f51): Mirror 11: 0x5965f26c (stale) Mirror 13: 0x5965f26c (stale) Mirror 14: 0x5965f26c (stale) Mirror 15: 0x5965f26c (stale) Mirror 16: 0x5965f26c (stale) Mirror 17: 0x5965f26c (stale) CRC-32 checksum value for chunk [0, 0x21f51): Mirror 1: 0xfe543606 Mirror 18: 0xfe543606 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21f51: skipped 000457 1787839465.744205 mapread 0x012cfb thru 0x014fa8 (0x22ae bytes) 000458 1787839465.779063 mapwrite 0x003964 thru 0x003f6f (0x60c bytes) 000459 1787839466.010302 punch from 0x001d6d to 0x0096dc (0x7970 bytes) 000460 1787839466.036650 MIRROR_RESYNC 8 mirrors 000461 1787839468.836586 read 0x00d378 thru 0x016829 (0x94b2 bytes) 000462 1787839468.844646 write 0x034f52 thru 0x036b41 (0x1bf0 bytes) 000463 1787839468.891956 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b42): Mirror 11: 0x9bc5d281 (stale) Mirror 13: 0x9bc5d281 (stale) Mirror 14: 0x9bc5d281 (stale) Mirror 15: 0x9bc5d281 (stale) Mirror 16: 0x9bc5d281 (stale) Mirror 18: 0x9bc5d281 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b42): Mirror 11: 0x9bc5d281 (stale) Mirror 13: 0x9bc5d281 (stale) Mirror 14: 0x9bc5d281 (stale) Mirror 15: 0x9bc5d281 (stale) Mirror 16: 0x9bc5d281 (stale) Mirror 18: 0x9bc5d281 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b42: skipped 000464 1787839469.352512 punch from 0x009c92 to 0x01754b (0xd8ba bytes) 000465 1787839469.379077 punch from 0x0223af to 0x03113a (0xed8c bytes) 000466 1787839469.385794 write 0x01184c thru 0x013437 (0x1bec bytes) 000467 1787839469.421644 mapread 0x0247ee thru 0x02f2f3 (0xab06 bytes) 000468 1787839469.453388 mapwrite 0x02e353 thru 0x03c724 (0xe3d2 bytes) 000469 1787839469.515174 mapread 0x023fcc thru 0x0282d6 (0x430b bytes) 000470 1787839469.516302 read 0x0020c3 thru 0x005858 (0x3796 bytes) 000472 1787839469.526676 punch from 0x013fde to 0x01be9f (0x7ec2 bytes) 000474 1787839469.533015 write 0x028d90 thru 0x02bda4 (0x3015 bytes) 000475 1787839469.541303 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c725): Mirror 13: 0xfe4b691e (stale) Mirror 14: 0xfe4b691e (stale) Mirror 15: 0xfe4b691e (stale) Mirror 16: 0xfe4b691e (stale) Mirror 18: 0xfe4b691e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c725): Mirror 13: 0xfe4b691e (stale) Mirror 14: 0xfe4b691e (stale) Mirror 15: 0xfe4b691e (stale) Mirror 16: 0xfe4b691e (stale) Mirror 18: 0xfe4b691e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c725: skipped 000476 1787839469.969974 punch from 0x039f69 to 0x03b976 (0x1a0e bytes) 000477 1787839469.974626 mapread 0x001404 thru 0x00ff3f (0xeb3c bytes) 000478 1787839470.000418 punch from 0x036886 to 0x03c724 (0x5e9f bytes) 000479 1787839470.006109 mapread 0x01378f thru 0x020b35 (0xd3a7 bytes) 000480 1787839470.022323 read 0x026843 thru 0x030e2f (0xa5ed bytes) 000481 1787839470.030629 mapwrite 0x02e9de thru 0x034c74 (0x6297 bytes) 000482 1787839470.055988 trunc from 0x03c725 to 0x000e3e (0x3b8e6 bytes) 000483 1787839470.079334 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xe3f): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe3f): Mirror 13: 0x9c53436 (stale) Mirror 14: 0x9c53436 (stale) Mirror 15: 0x9c53436 (stale) Mirror 16: 0x9c53436 (stale) Mirror 18: 0x9c53436 (stale) CRC-32 checksum value for chunk [0, 0xe3f): Mirror 1: 0x9c53436 Mirror 19: 0x9c53436 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe3f: skipped 000484 1787839470.617001 read 0x00054a thru 0x000e3e (0x8f5 bytes) 000485 1787839470.623746 mapwrite 0x027241 thru 0x02f23f (0x7fff bytes) 000487 1787839470.778613 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f240): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f240): Mirror 13: 0x1fa5f5ce (stale) Mirror 14: 0x1fa5f5ce (stale) Mirror 15: 0x1fa5f5ce (stale) Mirror 16: 0x1fa5f5ce (stale) Mirror 18: 0x1fa5f5ce (stale) Mirror 19: 0xc7816db6 (stale) CRC-32 checksum value for chunk [0, 0x2f240): Mirror 1: 0x2465288d Mirror 20: 0x2465288d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f240: skipped 000488 1787839471.953001 write 0x000f83 thru 0x002741 (0x17bf bytes) 000489 1787839472.075392 read 0x011750 thru 0x017ff5 (0x68a6 bytes) 000490 1787839472.095571 read 0x02baec thru 0x02f23f (0x3754 bytes) 000491 1787839472.126186 trunc from 0x02f240 to 0x02203f (0xd200 bytes) 000493 1787839472.217374 MIRROR_RESYNC 8 mirrors 000495 1787839474.029326 trunc from 0x020067 to 0x02203f (0x1fd9 bytes) 000496 1787839474.128183 punch from 0x020ea4 to 0x02203f (0x119c bytes) 000497 1787839474.148081 mapwrite 0x01e21d thru 0x026479 (0x825d bytes) 000498 1787839474.260002 punch from 0x003e65 to 0x0073fe (0x359a bytes) 000499 1787839474.275123 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2647a): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2647a): Mirror 13: 0x8ff4f33d (stale) Mirror 14: 0x8ff4f33d (stale) Mirror 15: 0x8ff4f33d (stale) Mirror 16: 0x8ff4f33d (stale) Mirror 18: 0x8ff4f33d (stale) Mirror 19: 0x8ff4f33d (stale) Mirror 20: 0x8ff4f33d (stale) CRC-32 checksum value for chunk [0, 0x2647a): Mirror 1: 0x9fb9b59a Mirror 21: 0x9fb9b59a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2647a: skipped 000500 1787839475.573415 read 0x00aade thru 0x00cec8 (0x23eb bytes) 000501 1787839475.583008 fallocate 0x033f17 thru 0x035561 (0x164b bytes) 000503 1787839475.670354 mapwrite 0x002a15 thru 0x00a3b1 (0x799d bytes) 000505 1787839475.775321 trunc from 0x000071 to 0x00cb9a (0xcb2a bytes) 000506 1787839475.791161 MIRROR_RESYNC 9 mirrors 000507 1787839477.646816 punch from 0x020d4f to 0x026479 (0x572b bytes) 000508 1787839477.704258 punch from 0x00657c to 0x010adb (0xa560 bytes) 000510 1787839477.711679 trunc from 0x00dc22 to 0x013bbb (0x5f9a bytes) 000511 1787839477.718019 mapread 0x000fd3 thru 0x01052a (0xf558 bytes) 000513 1787839477.742020 read 0x005ccd thru 0x00d781 (0x7ab5 bytes) 000514 1787839477.743213 trunc from 0x01d2a8 to 0x026479 (0x91d2 bytes) 000515 1787839477.748621 mapread 0x00222d thru 0x007c1c (0x59f0 bytes) 000516 1787839477.754057 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2647a): Mirror 14: 0x29a5ba2 (stale) Mirror 15: 0x29a5ba2 (stale) Mirror 16: 0x29a5ba2 (stale) Mirror 18: 0x29a5ba2 (stale) Mirror 19: 0x29a5ba2 (stale) Mirror 20: 0x29a5ba2 (stale) Mirror 21: 0x29a5ba2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2647a): Mirror 14: 0x29a5ba2 (stale) Mirror 15: 0x29a5ba2 (stale) Mirror 16: 0x29a5ba2 (stale) Mirror 18: 0x29a5ba2 (stale) Mirror 19: 0x29a5ba2 (stale) Mirror 20: 0x29a5ba2 (stale) Mirror 21: 0x29a5ba2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2647a: skipped 000517 1787839478.246676 punch from 0x020a9e to 0x026479 (0x59dc bytes) 000518 1787839478.252504 write 0x015ccd thru 0x02194d (0xbc81 bytes) 000519 1787839478.268137 punch from 0x00396b to 0x007d9c (0x4432 bytes) 000520 1787839478.273662 trunc from 0x014faf to 0x016448 (0x149a bytes) 000521 1787839478.290843 mapwrite 0x025b7a thru 0x02b0c8 (0x554f bytes) 000522 1787839478.341775 punch from 0x012db3 to 0x013622 (0x870 bytes) 000523 1787839478.347505 trunc from 0x02b0c9 to 0x0039fd (0x276cb bytes) 000524 1787839478.369252 mapread 0x002814 thru 0x0039fd (0x11ea bytes) 000525 1787839478.377461 punch from 0x001b81 to 0x0039fd (0x1e7d bytes) 000526 1787839478.384428 trunc from 0x0006ee to 0x0039fd (0x3310 bytes) 000527 1787839478.390961 trunc from 0x000dc5 to 0x0039fd (0x2c39 bytes) 000528 1787839478.396273 mapwrite 0x00138e thru 0x008eff (0x7b72 bytes) 000530 1787839478.452462 MIRROR_RESYNC 8 mirrors 000531 1787839479.704023 punch from 0x00112d to 0x0049af (0x3883 bytes) 000532 1787839479.754240 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x8f00): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f00): Mirror 14: 0x4b3b18fe (stale) Mirror 15: 0x4b3b18fe (stale) Mirror 16: 0x4b3b18fe (stale) Mirror 18: 0x4b3b18fe (stale) Mirror 19: 0x4b3b18fe (stale) Mirror 20: 0x4b3b18fe (stale) Mirror 21: 0x4b3b18fe (stale) CRC-32 checksum value for chunk [0, 0x8f00): Mirror 1: 0xb89271d1 Mirror 22: 0xb89271d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f00: skipped 000533 1787839481.011657 trunc from 0x001e45 to 0x008eff (0x70bb bytes) 000534 1787839481.122474 trunc from 0x006bf0 to 0x00845f (0x1870 bytes) 000535 1787839481.141492 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8f00): Mirror 15: 0x4b3b18fe (stale) Mirror 16: 0x4b3b18fe (stale) Mirror 18: 0x4b3b18fe (stale) Mirror 19: 0x4b3b18fe (stale) Mirror 20: 0x4b3b18fe (stale) Mirror 21: 0x4b3b18fe (stale) Mirror 22: 0xb89271d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f00): Mirror 15: 0x4b3b18fe (stale) Mirror 16: 0x4b3b18fe (stale) Mirror 18: 0x4b3b18fe (stale) Mirror 19: 0x4b3b18fe (stale) Mirror 20: 0x4b3b18fe (stale) Mirror 21: 0x4b3b18fe (stale) Mirror 22: 0xb89271d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f00: skipped 000536 1787839481.589587 punch from 0x0018a6 to 0x008d74 (0x74cf bytes) 000537 1787839481.594123 fallocate 0x001b15 thru 0x00e947 (0xce33 bytes) 000538 1787839481.604613 write 0x039732 thru 0x039ced (0x5bc bytes) 000539 1787839481.619783 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39cee): Mirror 15: 0x2ff691ba (stale) Mirror 16: 0x2ff691ba (stale) Mirror 18: 0x2ff691ba (stale) Mirror 20: 0x2ff691ba (stale) Mirror 21: 0x2ff691ba (stale) Mirror 22: 0x8758ae86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cee): Mirror 15: 0x2ff691ba (stale) Mirror 16: 0x2ff691ba (stale) Mirror 18: 0x2ff691ba (stale) Mirror 20: 0x2ff691ba (stale) Mirror 21: 0x2ff691ba (stale) Mirror 22: 0x8758ae86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cee: skipped 000540 1787839482.057353 punch from 0x032b9e to 0x039ced (0x7150 bytes) 000541 1787839482.073477 write 0x033c9f thru 0x037a41 (0x3da3 bytes) 000542 1787839482.095534 punch from 0x02ae8e to 0x02aee5 (0x58 bytes) 000543 1787839482.102180 read 0x01e231 thru 0x0241e8 (0x5fb8 bytes) 000545 1787839482.123585 write 0x0134ce thru 0x01e5d9 (0xb10c bytes) 000546 1787839482.131972 trunc from 0x0390c6 to 0x039ced (0xc28 bytes) 000547 1787839482.142828 mapread 0x01a4fc thru 0x01c1c7 (0x1ccc bytes) 000548 1787839482.145196 mapwrite 0x00b167 thru 0x011fb1 (0x6e4b bytes) 000550 1787839482.208855 punch from 0x00a98f to 0x00e8b0 (0x3f22 bytes) 000551 1787839482.222251 write 0x03ef10 thru 0x03ffff (0x10f0 bytes) 000552 1787839482.223650 trunc from 0x040000 to 0x020213 (0x1fdec bytes) 000553 1787839482.277991 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20214): Mirror 15: 0xcf8bef28 (stale) Mirror 16: 0xcf8bef28 (stale) Mirror 20: 0xcf8bef28 (stale) Mirror 21: 0xcf8bef28 (stale) Mirror 22: 0x6d14e68c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20214): Mirror 15: 0xcf8bef28 (stale) Mirror 16: 0xcf8bef28 (stale) Mirror 20: 0xcf8bef28 (stale) Mirror 21: 0xcf8bef28 (stale) Mirror 22: 0x6d14e68c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20214: skipped 000556 1787839482.820676 trunc from 0x020214 to 0x03063c (0x10429 bytes) 000557 1787839482.842740 mapwrite 0x0022f0 thru 0x00af0b (0x8c1c bytes) 000558 1787839482.885784 punch from 0x02dbc0 to 0x03063c (0x2a7d bytes) 000559 1787839482.891377 write 0x00fb9e thru 0x019638 (0x9a9b bytes) 000560 1787839482.897803 write 0x00311a thru 0x007d1b (0x4c02 bytes) 000561 1787839482.899298 fallocate 0x0288c5 thru 0x03035e (0x7a9a bytes) 000562 1787839482.905175 trunc from 0x0068ac to 0x0074cd (0xc22 bytes) 000563 1787839482.922552 read 0x027546 thru 0x02c4a1 (0x4f5c bytes) 000564 1787839482.931153 trunc from 0x03063d to 0x01c1bf (0x1447d bytes) 000565 1787839482.945458 fallocate 0x02a032 thru 0x034635 (0xa604 bytes) 000566 1787839482.950577 trunc from 0x01c1c0 to 0x00f56d (0xcc52 bytes) 000567 1787839482.974266 punch from 0x00f3cd to 0x00f56d (0x1a1 bytes) 000568 1787839482.980017 read 0x0024dd thru 0x00f56d (0xd091 bytes) 000569 1787839482.997972 punch from 0x00218a to 0x00c454 (0xa2cb bytes) 000570 1787839483.003750 punch from 0x002166 to 0x0058b9 (0x3754 bytes) 000572 1787839483.011859 punch from 0x0009f5 to 0x00b5ee (0xabfa bytes) 000574 1787839483.018817 write 0x01b880 thru 0x01b92e (0xaf bytes) 000575 1787839483.028750 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: 0x952ae2ec (stale) Mirror 16: 0x952ae2ec (stale) Mirror 20: 0x952ae2ec (stale) Mirror 22: 0xd06d5e38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: 0x952ae2ec (stale) Mirror 16: 0x952ae2ec (stale) Mirror 20: 0x952ae2ec (stale) Mirror 22: 0xd06d5e38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b92f: skipped 000576 1787839483.482212 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: 0x952ae2ec (stale) Mirror 16: 0x952ae2ec (stale) Mirror 20: 0x952ae2ec (stale) Mirror 22: 0xd06d5e38 (stale) CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 1: 0x7a6bb5d2 Mirror 23: 0x7a6bb5d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b92f: skipped 000577 1787839484.701600 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 15: 0x952ae2ec (stale) Mirror 16: 0x952ae2ec (stale) Mirror 20: 0x952ae2ec (stale) Mirror 22: 0xd06d5e38 (stale) CRC-32 checksum value for chunk [0, 0x1b92f): Mirror 1: 0x7a6bb5d2 Mirror 23: 0x7a6bb5d2 Mirror 24: 0x7a6bb5d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b92f: skipped 000578 1787839485.841363 mapwrite 0x033ee1 thru 0x03b32d (0x744d bytes) 000581 1787839485.957890 trunc from 0x02d992 to 0x02f665 (0x1cd4 bytes) 000582 1787839485.965651 fallocate 0x03a2a1 thru 0x03ffff (0x5d5f bytes) 000583 1787839485.972108 write 0x003bee thru 0x00c10f (0x8522 bytes) 000584 1787839485.985726 write 0x005886 thru 0x013750 (0xdecb bytes) 000585 1787839485.995163 read 0x02a42e thru 0x02bb4e (0x1721 bytes) 000586 1787839486.004254 mapread 0x034b43 thru 0x03b32d (0x67eb bytes) 000587 1787839486.008446 mapread 0x03630b thru 0x039af8 (0x37ee bytes) 000588 1787839486.009832 mapwrite 0x0042c0 thru 0x005ba1 (0x18e2 bytes) 000589 1787839486.031546 read 0x0291a4 thru 0x03414b (0xafa8 bytes) 000590 1787839486.044266 fallocate 0x004fc1 thru 0x014821 (0xf861 bytes) 000591 1787839486.056506 MIRROR_RESYNC 7 mirrors 000592 1787839488.321477 punch from 0x03a52e to 0x03b32d (0xe00 bytes) 000593 1787839488.367218 write 0x00d410 thru 0x0174d8 (0xa0c9 bytes) 000594 1787839488.385326 write 0x020856 thru 0x023414 (0x2bbf bytes) 000595 1787839488.394491 read 0x01641c thru 0x01c413 (0x5ff8 bytes) 000596 1787839488.414488 mapread 0x01ef13 thru 0x020636 (0x1724 bytes) 000597 1787839488.430714 punch from 0x03a7a3 to 0x03ad8f (0x5ed bytes) 000598 1787839488.437600 read 0x00e925 thru 0x018999 (0xa075 bytes) 000599 1787839488.438532 write 0x0292f1 thru 0x0330cb (0x9ddb bytes) 000600 1787839488.457637 MIRROR_RESYNC 7 mirrors 000601 1787839490.671569 fallocate 0x03940e thru 0x03ffff (0x6bf2 bytes) 000603 1787839490.716756 fallocate 0x025818 thru 0x032b87 (0xd370 bytes) 000604 1787839490.722583 trunc from 0x029d0f to 0x039314 (0xf606 bytes) 000605 1787839490.727483 fallocate 0x02c64a thru 0x035143 (0x8afa bytes) 000606 1787839490.737816 MIRROR_RESYNC 7 mirrors 000608 1787839493.149545 trunc from 0x0103a7 to 0x0137b5 (0x340f bytes) 000609 1787839493.205193 mapread 0x01b883 thru 0x020fb1 (0x572f bytes) 000610 1787839493.232513 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xcaaea28b (stale) Mirror 16: 0xcaaea28b (stale) Mirror 20: 0xcaaea28b (stale) Mirror 22: 0xcaaea28b (stale) Mirror 24: 0xcaaea28b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xcaaea28b (stale) Mirror 16: 0xcaaea28b (stale) Mirror 20: 0xcaaea28b (stale) Mirror 22: 0xcaaea28b (stale) Mirror 24: 0xcaaea28b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000611 1787839493.752998 fallocate 0x023989 thru 0x030610 (0xcc88 bytes) 000612 1787839493.767662 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xcaaea28b (stale) Mirror 16: 0xcaaea28b (stale) Mirror 20: 0xcaaea28b (stale) Mirror 22: 0xcaaea28b (stale) Mirror 24: 0xcaaea28b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcaba71df Mirror 25: 0xcaba71df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1787839494.885840 mapread 0x034240 thru 0x03e3e0 (0xa1a1 bytes) 000616 1787839494.909530 trunc from 0x040000 to 0x00206d (0x3df92 bytes) 000617 1787839494.972493 fallocate 0x011d9e thru 0x013b7f (0x1de2 bytes) 000618 1787839494.977737 trunc from 0x00206e to 0x008a76 (0x6a09 bytes) 000619 1787839495.002537 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8a77): Mirror 15: 0x915a4bcb (stale) Mirror 20: 0x915a4bcb (stale) Mirror 22: 0x915a4bcb (stale) Mirror 24: 0x915a4bcb (stale) Mirror 25: 0x915a4bcb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8a77): Mirror 15: 0x915a4bcb (stale) Mirror 20: 0x915a4bcb (stale) Mirror 22: 0x915a4bcb (stale) Mirror 24: 0x915a4bcb (stale) Mirror 25: 0x915a4bcb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8a77: skipped 000620 1787839495.372552 trunc from 0x008a77 to 0x00ff9d (0x7527 bytes) 000621 1787839495.397100 punch from 0x00b8dc to 0x00ff9d (0x46c2 bytes) 000622 1787839495.402657 punch from 0x00b0d0 to 0x00ff9d (0x4ece bytes) 000623 1787839495.412631 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xff9e): Mirror 15: 0x58fa7903 (stale) Mirror 22: 0x58fa7903 (stale) Mirror 24: 0x58fa7903 (stale) Mirror 25: 0x58fa7903 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xff9e): Mirror 15: 0x58fa7903 (stale) Mirror 22: 0x58fa7903 (stale) Mirror 24: 0x58fa7903 (stale) Mirror 25: 0x58fa7903 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xff9e: skipped 000624 1787839495.750451 trunc from 0x00ff9e to 0x00a5a0 (0x59fd bytes) 000625 1787839495.776261 fallocate 0x0382e9 thru 0x03ffff (0x7d17 bytes) 000626 1787839495.781829 trunc from 0x040000 to 0x039d29 (0x62d6 bytes) 000627 1787839495.798982 trunc from 0x016c63 to 0x021bfe (0xaf9c bytes) 000628 1787839495.804252 trunc from 0x02e67c to 0x0337fa (0x517f bytes) 000630 1787839495.810152 trunc from 0x039d2a to 0x01f337 (0x1a9f2 bytes) 000631 1787839495.826465 trunc from 0x01f338 to 0x036c04 (0x178cd bytes) 000632 1787839495.843867 write 0x0389db thru 0x03c156 (0x377c bytes) 000633 1787839495.850790 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: 0x7327959d (stale) Mirror 22: 0x7327959d (stale) Mirror 25: 0xc3fdf698 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: 0x7327959d (stale) Mirror 22: 0x7327959d (stale) Mirror 25: 0xc3fdf698 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c157: skipped 000636 1787839496.270531 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: 0x7327959d (stale) Mirror 22: 0x7327959d (stale) Mirror 25: 0xc3fdf698 (stale) CRC-32 checksum value for chunk [0, 0x3c157): Mirror 1: 0xc5ce2789 Mirror 26: 0xc5ce2789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c157: skipped 000637 1787839497.572349 read 0x003eab thru 0x007b15 (0x3c6b bytes) 000638 1787839497.584369 trunc from 0x028408 to 0x0327c1 (0xa3ba bytes) 000639 1787839497.650459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c157): Mirror 15: 0x7327959d (stale) Mirror 22: 0x7327959d (stale) Mirror 25: 0xc3fdf698 (stale) Mirror 26: 0xc5ce2789 (stale) CRC-32 checksum value for chunk [0, 0x3c157): Mirror 1: 0xc5ce2789 Mirror 27: 0xc5ce2789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c157: skipped 000641 1787839498.599469 mapread 0x027bd7 thru 0x032381 (0xa7ab bytes) 000642 1787839498.641780 trunc from 0x03c157 to 0x0337c4 (0x8992 bytes) 000643 1787839498.694239 mapwrite 0x026a87 thru 0x02a3fc (0x3976 bytes) 000644 1787839498.720612 write 0x015306 thru 0x01c38d (0x7088 bytes) 000645 1787839498.732456 trunc from 0x0337c5 to 0x0174b2 (0x1c312 bytes) 000646 1787839498.756583 trunc from 0x012f53 to 0x0174b2 (0x4560 bytes) 000647 1787839498.767759 punch from 0x00afa1 to 0x0174b2 (0xc512 bytes) 000648 1787839498.773684 trunc from 0x0174b3 to 0x03884f (0x2139d bytes) 000649 1787839498.790237 mapread 0x002ec4 thru 0x012b24 (0xfc61 bytes) 000650 1787839498.811844 write 0x01bcd4 thru 0x020dae (0x50db bytes) 000651 1787839498.824708 punch from 0x011dc6 to 0x01359f (0x17da bytes) 000652 1787839498.834993 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38850): Mirror 15: 0x85ae5725 (stale) Mirror 22: 0x85ae5725 (stale) Mirror 26: 0x7da52b6a (stale) Mirror 27: 0x7da52b6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38850): Mirror 15: 0x85ae5725 (stale) Mirror 22: 0x85ae5725 (stale) Mirror 26: 0x7da52b6a (stale) Mirror 27: 0x7da52b6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38850: skipped 000653 1787839499.285109 mapwrite 0x035309 thru 0x03ffff (0xacf7 bytes) 000654 1787839499.340401 read 0x023309 thru 0x026d8b (0x3a83 bytes) 000655 1787839499.348804 trunc from 0x040000 to 0x032abf (0xd540 bytes) 000656 1787839499.364333 mapwrite 0x00568b thru 0x010387 (0xacfd bytes) 000657 1787839499.401749 trunc from 0x02a162 to 0x032abf (0x895e bytes) 000658 1787839499.407413 read 0x001fae thru 0x00fc95 (0xdce8 bytes) 000659 1787839499.408063 mapwrite 0x008528 thru 0x01070b (0x81e4 bytes) 000660 1787839499.431349 read 0x00807c thru 0x00a664 (0x25e9 bytes) 000661 1787839499.431746 fallocate 0x02a69c thru 0x031567 (0x6ecc bytes) 000662 1787839499.439429 trunc from 0x032ac0 to 0x019c40 (0x18e7f bytes) 000663 1787839499.461519 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19c41): Mirror 15: 0x749dd736 (stale) Mirror 22: 0x749dd736 (stale) Mirror 27: 0x28446030 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c41): Mirror 15: 0x749dd736 (stale) Mirror 22: 0x749dd736 (stale) Mirror 27: 0x28446030 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c41: skipped 000664 1787839499.844355 punch from 0x002480 to 0x00fda3 (0xd924 bytes) 000665 1787839499.854292 mapread 0x0169a5 thru 0x019c40 (0x329c bytes) 000666 1787839499.891455 punch from 0x003735 to 0x0061bb (0x2a87 bytes) 000667 1787839499.896499 trunc from 0x004fd5 to 0x00bdd8 (0x6e04 bytes) 000668 1787839499.904627 trunc from 0x005f7d to 0x00a0d4 (0x4158 bytes) 000669 1787839499.910524 write 0x0036fc thru 0x008ecc (0x57d1 bytes) 000670 1787839499.926598 write 0x03078e thru 0x03a2ae (0x9b21 bytes) 000671 1787839499.928628 trunc from 0x03a2af to 0x010ea3 (0x2940b bytes) 000672 1787839499.946769 trunc from 0x010ea4 to 0x021875 (0x109d2 bytes) 000673 1787839499.973003 trunc from 0x021876 to 0x034aa9 (0x13234 bytes) 000674 1787839499.995009 write 0x01c537 thru 0x01cef9 (0x9c3 bytes) 000675 1787839500.004265 punch from 0x012197 to 0x013afa (0x1964 bytes) 000677 1787839500.010783 write 0x01b71b thru 0x024283 (0x8b69 bytes) 000678 1787839500.035402 read 0x001d4e thru 0x00470b (0x29be bytes) 000679 1787839500.045653 punch from 0x02f4be to 0x034aa9 (0x55ec bytes) 000680 1787839500.052113 trunc from 0x002fd3 to 0x01193a (0xe968 bytes) 000681 1787839500.091605 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34aaa): Mirror 22: 0xfcd044c2 (stale) Mirror 27: 0x8cada888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34aaa): Mirror 22: 0xfcd044c2 (stale) Mirror 27: 0x8cada888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34aaa: skipped 000682 1787839500.433741 trunc from 0x034aaa to 0x001484 (0x33625 bytes) 000683 1787839500.457731 punch from 0x000ef0 to 0x001484 (0x595 bytes) 000684 1787839500.469539 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1485): Mirror 27: 0xadbdaa6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1485): Mirror 27: 0xadbdaa6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1485: skipped 000685 1787839500.776347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1485): Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1485): Mirror 27: 0xadbdaa6 (stale) CRC-32 checksum value for chunk [0, 0x1485): Mirror 1: 0xadbdaa6 Mirror 28: 0xadbdaa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1485: skipped 000686 1787839502.033747 trunc from 0x001485 to 0x033956 (0x324d2 bytes) 000687 1787839502.173430 punch from 0x02e501 to 0x033956 (0x5456 bytes) 000688 1787839502.183735 trunc from 0x0144ff to 0x024359 (0xfe5b bytes) 000689 1787839502.196835 MIRROR_RESYNC 3 mirrors 000690 1787839503.020460 write 0x009f6d thru 0x019381 (0xf415 bytes) 000691 1787839503.090101 trunc from 0x033957 to 0x01c00b (0x1794b bytes) 000692 1787839503.109644 read 0x0195da thru 0x019d52 (0x779 bytes) 000693 1787839503.110457 write 0x02aa94 thru 0x0309e3 (0x5f50 bytes) 000694 1787839503.112419 write 0x01d93c thru 0x020a0c (0x30d1 bytes) 000695 1787839503.132350 mapread 0x01a4e1 thru 0x0299e5 (0xf505 bytes) 000696 1787839503.156731 fallocate 0x035b77 thru 0x03f066 (0x94f0 bytes) 000697 1787839503.161561 mapwrite 0x0153a8 thru 0x01d745 (0x839e bytes) 000698 1787839503.184017 read 0x02bf8f thru 0x0309e3 (0x4a55 bytes) 000699 1787839503.184847 fallocate 0x01d694 thru 0x02baf8 (0xe465 bytes) 000700 1787839503.196627 trunc from 0x0275b2 to 0x0309e3 (0x9432 bytes) 000701 1787839503.202537 trunc from 0x0309e4 to 0x0258f8 (0xb0eb bytes) 000702 1787839503.224842 trunc from 0x0258f9 to 0x00508d (0x2086b bytes) 000703 1787839503.247609 trunc from 0x003501 to 0x00508d (0x1b8d bytes) 000704 1787839503.257428 mapwrite 0x005c8c thru 0x00adb0 (0x5125 bytes) 000705 1787839503.299411 trunc from 0x00adb1 to 0x01d02b (0x1227b bytes) 000706 1787839503.317555 trunc from 0x01d02c to 0x03fe93 (0x22e68 bytes) 000707 1787839503.342404 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fe94): Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe94): Mirror 27: 0x2a15750c (stale) Mirror 28: 0x2a15750c (stale) CRC-32 checksum value for chunk [0, 0x3fe94): Mirror 1: 0x1e30425f Mirror 29: 0x1e30425f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe94: skipped 000708 1787839504.390181 trunc from 0x03fe94 to 0x003a46 (0x3c44d bytes) 000709 1787839504.465405 mapread 0x001ab7 thru 0x003a46 (0x1f90 bytes) 000710 1787839504.484373 trunc from 0x003a47 to 0x03ae80 (0x3743a bytes) 000712 1787839504.531222 fallocate 0x028c0c thru 0x02a584 (0x1979 bytes) 000713 1787839504.546350 trunc from 0x00b76e to 0x019d35 (0xe5c8 bytes) 000714 1787839504.551852 write 0x02da7e thru 0x02e182 (0x705 bytes) 000715 1787839504.566573 trunc from 0x029283 to 0x02ef7c (0x5cfa bytes) 000716 1787839504.582441 write 0x00af58 thru 0x014f4c (0x9ff5 bytes) 000717 1787839504.640759 mapwrite 0x00c2bc thru 0x0185e5 (0xc32a bytes) 000718 1787839504.702084 punch from 0x01d641 to 0x027fce (0xa98e bytes) 000719 1787839504.714331 write 0x02085e thru 0x029529 (0x8ccc bytes) 000720 1787839504.736068 trunc from 0x03ae81 to 0x027c25 (0x1325b bytes) 000721 1787839504.761211 trunc from 0x0033dc to 0x00942e (0x6053 bytes) 000723 1787839504.770232 fallocate 0x031633 thru 0x03e04d (0xca1b bytes) 000724 1787839504.776675 fallocate 0x031248 thru 0x03d0c9 (0xbe82 bytes) 000725 1787839504.792223 mapread 0x001b5e thru 0x0035f8 (0x1a9b bytes) 000726 1787839504.808840 trunc from 0x03d0ca to 0x03974a (0x397f bytes) 000727 1787839504.842074 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3974b): Mirror 27: 0x739de54b (stale) Mirror 29: 0xfa2c1b30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3974b): Mirror 27: 0x739de54b (stale) Mirror 29: 0xfa2c1b30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3974b: skipped 000728 1787839505.353233 MIRROR_RESYNC 3 mirrors 000729 1787839506.631271 trunc from 0x0289c6 to 0x02a1c5 (0x1800 bytes) 000730 1787839506.693176 punch from 0x036131 to 0x03974a (0x361a bytes) 000731 1787839506.705621 trunc from 0x028aa8 to 0x037285 (0xe7de bytes) 000732 1787839506.719769 mapwrite 0x01df18 thru 0x023b35 (0x5c1e bytes) 000733 1787839506.788409 fallocate 0x03cea4 thru 0x03ffff (0x315c bytes) 000734 1787839506.795451 mapread 0x022f27 thru 0x025047 (0x2121 bytes) 000735 1787839506.797447 trunc from 0x0133ec to 0x01f394 (0xbfa9 bytes) 000736 1787839506.803380 mapwrite 0x0107ff thru 0x016684 (0x5e86 bytes) 000737 1787839506.856821 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3974b): Mirror 29: 0x580c7faf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3974b): Mirror 29: 0x580c7faf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3974b: skipped 000740 1787839507.428767 mapread 0x00a045 thru 0x00ce25 (0x2de1 bytes) 000741 1787839507.458265 mapread 0x0162eb thru 0x0221a3 (0xbeb9 bytes) 000742 1787839507.489593 fallocate 0x01b07b thru 0x024494 (0x941a bytes) 000744 1787839507.497213 trunc from 0x03974b to 0x03bcdd (0x2593 bytes) 000745 1787839507.524750 write 0x012126 thru 0x019cae (0x7b89 bytes) 000746 1787839507.528059 trunc from 0x03bcde to 0x018933 (0x233aa bytes) 000747 1787839507.560687 fallocate 0x016b52 thru 0x01b1c6 (0x4675 bytes) 000749 1787839507.567310 fallocate 0x017bef thru 0x01e976 (0x6d88 bytes) 000750 1787839507.575184 trunc from 0x007715 to 0x015d2a (0xe616 bytes) 000751 1787839507.590153 punch from 0x000ff2 to 0x010ccd (0xfcdc bytes) 000752 1787839507.595448 fallocate 0x02ee63 thru 0x03e314 (0xf4b2 bytes) 000753 1787839507.601852 mapwrite 0x02a1eb thru 0x038760 (0xe576 bytes) 000758 1787839507.650295 mapread 0x02939c thru 0x02dc2b (0x4890 bytes) 000759 1787839507.652299 trunc from 0x03e315 to 0x012fad (0x2b367 bytes) 000760 1787839507.669625 mapread 0x00c99f thru 0x0105bf (0x3c21 bytes) 000763 1787839507.686168 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12fae): Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12fae): Mirror 29: 0x41402336 (stale) CRC-32 checksum value for chunk [0, 0x12fae): Mirror 1: 0xd5dc3df6 Mirror 30: 0xd5dc3df6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12fae: skipped 000764 1787839508.721445 mapwrite 0x033e82 thru 0x03b3e9 (0x7568 bytes) 000765 1787839508.925867 mapread 0x02954b thru 0x02f6b7 (0x616d bytes) 000766 1787839508.943457 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b3ea): Mirror 29: 0x4be18441 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3ea): Mirror 29: 0x4be18441 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3ea: skipped 000767 1787839509.215213 write 0x01c105 thru 0x021529 (0x5425 bytes) 000769 1787839509.227717 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b3ea): Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3ea): Mirror 29: 0x4be18441 (stale) CRC-32 checksum value for chunk [0, 0x3b3ea): Mirror 1: 0x9ab9e8c Mirror 30: 0x9ab9e8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3ea: skipped 000771 1787839509.958495 write 0x0146aa thru 0x01b29b (0x6bf2 bytes) 000772 1787839510.024266 read 0x01a4e0 thru 0x0222a6 (0x7dc7 bytes) 000773 1787839510.041600 fallocate 0x016ddd thru 0x026b1b (0xfd3f bytes) 000774 1787839510.048485 fallocate 0x03dd02 thru 0x03ffff (0x22fe bytes) 000775 1787839510.058590 MIRROR_RESYNC 3 mirrors 000776 1787839510.879450 punch from 0x009158 to 0x016041 (0xceea bytes) 000777 1787839510.914128 mapread 0x02fdcc thru 0x031bcf (0x1e04 bytes) 000778 1787839510.935249 trunc from 0x040000 to 0x00a19b (0x35e64 bytes) 000779 1787839510.973235 trunc from 0x00a19c to 0x02e9c4 (0x24829 bytes) 000780 1787839510.991286 trunc from 0x00d2cc to 0x018994 (0xb6c9 bytes) 000781 1787839510.997368 read 0x01b56b thru 0x01fb1c (0x45b2 bytes) 000782 1787839511.005680 mapwrite 0x014157 thru 0x01d474 (0x931e bytes) 000783 1787839511.033006 mapread 0x014e6c thru 0x019cf0 (0x4e85 bytes) 000784 1787839511.033954 mapwrite 0x02bb0c thru 0x038d6d (0xd262 bytes) 000785 1787839511.087093 punch from 0x00c5ea to 0x015432 (0x8e49 bytes) 000786 1787839511.092793 write 0x03e764 thru 0x03ffff (0x189c bytes) 000787 1787839511.095563 trunc from 0x040000 to 0x01a02e (0x25fd1 bytes) 000788 1787839511.112374 read 0x00deac thru 0x0192c8 (0xb41d bytes) 000789 1787839511.130428 trunc from 0x01a02f to 0x03ccad (0x22c7f bytes) 000790 1787839511.146663 write 0x03f475 thru 0x03ffff (0xb8b bytes) 000791 1787839511.149036 trunc from 0x02e27a to 0x03c479 (0xe200 bytes) 000792 1787839511.153953 mapread 0x00bf3d thru 0x013f58 (0x801c bytes) 000793 1787839511.154973 mapread 0x02d6ee thru 0x0341e0 (0x6af3 bytes) 000794 1787839511.165668 mapwrite 0x034c72 thru 0x03b423 (0x67b2 bytes) 000795 1787839511.188242 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0x2892fdc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0x2892fdc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000796 1787839511.467181 mapread 0x0256da thru 0x03464c (0xef73 bytes) 000797 1787839511.496276 trunc from 0x03bea5 to 0x03ffff (0x415b bytes) 000798 1787839511.502237 mapread 0x02b4eb thru 0x032b5c (0x7672 bytes) 000799 1787839511.508369 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0x2892fdc3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe5d353 Mirror 30: 0xcfe5d353 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1787839512.572827 trunc from 0x02081e to 0x027a29 (0x720c bytes) 000802 1787839512.627668 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0x2892fdc3 (stale) Mirror 30: 0xcfe5d353 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe5d353 Mirror 31: 0xcfe5d353 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000803 1787839513.877747 punch from 0x018d1c to 0x01cb16 (0x3dfb bytes) 000804 1787839513.942678 punch from 0x02a1d2 to 0x02f434 (0x5263 bytes) 000807 1787839513.949601 trunc from 0x040000 to 0x015a0d (0x2a5f2 bytes) 000808 1787839513.973731 write 0x03a79c thru 0x03f0e1 (0x4946 bytes) 000809 1787839513.975651 write 0x01deee thru 0x0233d1 (0x54e4 bytes) 000810 1787839513.994327 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 29: 0x78fc7116 (stale) Mirror 30: 0xa258eb29 (stale) Mirror 31: 0xa258eb29 (stale) CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 1: 0x827a2600 Mirror 32: 0x827a2600 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0e2: skipped 000811 1787839515.230860 mapwrite 0x0061d3 thru 0x007221 (0x104f bytes) 000812 1787839515.355622 mapread 0x02cc4d thru 0x02fe65 (0x3219 bytes) 000813 1787839515.368070 read 0x006a39 thru 0x01372d (0xccf5 bytes) 000814 1787839515.378289 read 0x0080aa thru 0x015e75 (0xddcc bytes) 000815 1787839515.389235 write 0x023624 thru 0x025583 (0x1f60 bytes) 000816 1787839515.396481 write 0x0352b2 thru 0x036aa2 (0x17f1 bytes) 000818 1787839515.399673 mapread 0x0223e5 thru 0x0267dd (0x43f9 bytes) 000820 1787839515.404535 read 0x000693 thru 0x00f5ed (0xef5b bytes) 000821 1787839515.406289 trunc from 0x001a4d to 0x00861f (0x6bd3 bytes) 000822 1787839515.415806 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 29: 0x78fc7116 (stale) Mirror 30: 0xa258eb29 (stale) Mirror 31: 0xa258eb29 (stale) Mirror 32: 0x827a2600 (stale) CRC-32 checksum value for chunk [0, 0x3f0e2): Mirror 1: 0x2baddd09 Mirror 33: 0x2baddd09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0e2: skipped 000823 1787839517.059653 read 0x035cc7 thru 0x03aacd (0x4e07 bytes) 000824 1787839517.077029 trunc from 0x03b9c7 to 0x03f0e1 (0x371b bytes) 000825 1787839517.121470 punch from 0x00c1a6 to 0x018131 (0xbf8c bytes) 000826 1787839517.144289 write 0x01f647 thru 0x022df2 (0x37ac bytes) 000828 1787839517.166707 fallocate 0x013517 thru 0x0185ce (0x50b8 bytes) 000829 1787839517.178497 punch from 0x00380f to 0x00e4ef (0xace1 bytes) 000830 1787839517.184314 fallocate 0x010d63 thru 0x01fc75 (0xef13 bytes) 000833 1787839517.189299 mapwrite 0x03ca09 thru 0x03ffff (0x35f7 bytes) 000834 1787839517.255775 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: 0xcfe5d353 (stale) Mirror 31: 0xcfe5d353 (stale) Mirror 32: 0x3e949eee (stale) Mirror 33: 0xcfa85481 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: 0xcfe5d353 (stale) Mirror 31: 0xcfe5d353 (stale) Mirror 32: 0x3e949eee (stale) Mirror 33: 0xcfa85481 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1787839517.662675 mapwrite 0x0189ff thru 0x025382 (0xc984 bytes) 000836 1787839517.697128 mapwrite 0x00e624 thru 0x014f55 (0x6932 bytes) 000837 1787839517.714084 mapread 0x007365 thru 0x00b4d5 (0x4171 bytes) 000838 1787839517.714718 trunc from 0x040000 to 0x0354fa (0xab05 bytes) 000839 1787839517.726371 read 0x004cb2 thru 0x010451 (0xb7a0 bytes) 000840 1787839517.730302 punch from 0x029129 to 0x0354fa (0xc3d2 bytes) 000841 1787839517.733622 mapwrite 0x0290f3 thru 0x0315a0 (0x84ae bytes) 000842 1787839517.763460 fallocate 0x027f0a thru 0x037cf8 (0xfdef bytes) 000843 1787839517.775947 trunc from 0x0354fb to 0x0282e5 (0xd215 bytes) 000845 1787839517.808481 mapwrite 0x03fe95 thru 0x03ffff (0x16b bytes) 000846 1787839517.851929 mapread 0x022b49 thru 0x02ad4c (0x8204 bytes) 000847 1787839517.869533 mapread 0x01a706 thru 0x01d291 (0x2b8c bytes) 000848 1787839517.875340 punch from 0x03e13f to 0x03ffff (0x1ec1 bytes) 000849 1787839517.880696 punch from 0x00d3df to 0x019744 (0xc366 bytes) 000850 1787839517.895431 mapread 0x00c1f8 thru 0x0155cd (0x93d6 bytes) 000851 1787839517.914758 fallocate 0x01e62b thru 0x022457 (0x3e2d bytes) 000852 1787839517.920554 read 0x01cc13 thru 0x01f445 (0x2833 bytes) 000855 1787839517.924588 trunc from 0x018e8c to 0x01ff1d (0x7092 bytes) 000856 1787839517.940639 read 0x03d60f thru 0x03ffff (0x29f1 bytes) 000858 1787839517.960493 punch from 0x0171cc to 0x0258a1 (0xe6d6 bytes) 000859 1787839517.978698 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: 0xcfe5d353 (stale) Mirror 31: 0xcfe5d353 (stale) Mirror 32: 0x3e949eee (stale) Mirror 33: 0xcfa85481 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 34: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000860 1787839519.131494 mapwrite 0x03802c thru 0x03ffff (0x7fd4 bytes) 000862 1787839519.219470 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: 0xcfe5d353 (stale) Mirror 31: 0xcfe5d353 (stale) Mirror 32: 0x3e949eee (stale) Mirror 33: 0xcfa85481 (stale) Mirror 34: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c2afe19 Mirror 35: 0x1c2afe19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000863 1787839520.356531 trunc from 0x040000 to 0x03823b (0x7dc4 bytes) 000864 1787839520.398213 read 0x02ed07 thru 0x032ddf (0x40d9 bytes) 000865 1787839520.406442 punch from 0x001224 to 0x010f40 (0xfd1d bytes) 000866 1787839520.412307 trunc from 0x0324fb to 0x03823b (0x5d41 bytes) 000867 1787839520.416419 fallocate 0x001e02 thru 0x0035b7 (0x17b6 bytes) 000868 1787839520.419543 mapwrite 0x029229 thru 0x03196a (0x8742 bytes) 000869 1787839520.480805 fallocate 0x019213 thru 0x027131 (0xdf1f bytes) 000870 1787839520.490923 trunc from 0x03823c to 0x01c5d4 (0x1bc67 bytes) 000871 1787839520.514366 read 0x019a5a thru 0x01c5d4 (0x2b7b bytes) 000872 1787839520.531466 write 0x03ba42 thru 0x03ffff (0x45be bytes) 000873 1787839520.533714 write 0x0341e0 thru 0x03fb8b (0xb9ac bytes) 000874 1787839520.540253 mapwrite 0x00e9da thru 0x01bc62 (0xd289 bytes) 000875 1787839520.571042 mapread 0x00f7fc thru 0x016e98 (0x769d bytes) 000876 1787839520.572861 fallocate 0x036852 thru 0x037f12 (0x16c1 bytes) 000877 1787839520.578640 trunc from 0x040000 to 0x010de5 (0x2f21a bytes) 000878 1787839520.603333 mapwrite 0x02e7ea thru 0x02e925 (0x13c bytes) 000879 1787839520.645223 mapwrite 0x004b7b thru 0x00e6f8 (0x9b7e bytes) 000880 1787839520.667660 trunc from 0x02e926 to 0x03204a (0x3725 bytes) 000882 1787839520.684158 mapread 0x0078fb thru 0x0088ad (0xfb3 bytes) 000883 1787839520.685145 mapwrite 0x03c40a thru 0x03d4cd (0x10c4 bytes) 000884 1787839520.713222 trunc from 0x007c7f to 0x008925 (0xca7 bytes) 000885 1787839520.717416 mapread 0x0242bf thru 0x02cf79 (0x8cbb bytes) 000886 1787839520.746174 trunc from 0x0178e2 to 0x01809b (0x7ba bytes) 000887 1787839520.749224 fallocate 0x022d3f thru 0x025b84 (0x2e46 bytes) 000888 1787839520.762431 mapread 0x0100f8 thru 0x010e47 (0xd50 bytes) 000889 1787839520.763481 mapread 0x01aa0e thru 0x02215c (0x774f bytes) 000890 1787839520.815107 mapwrite 0x00d65e thru 0x017239 (0x9bdc bytes) 000891 1787839520.884306 read 0x003062 thru 0x00f9b2 (0xc951 bytes) 000892 1787839520.898519 read 0x024402 thru 0x0273c7 (0x2fc6 bytes) 000893 1787839520.899346 fallocate 0x01e699 thru 0x025cec (0x7654 bytes) 000894 1787839520.904000 write 0x02ed3a thru 0x038661 (0x9928 bytes) 000895 1787839520.906744 trunc from 0x00d582 to 0x0153e0 (0x7e5f bytes) 000896 1787839520.912162 punch from 0x01b175 to 0x01b53d (0x3c9 bytes) 000897 1787839520.920225 MIRROR_RESYNC 7 mirrors 000898 1787839523.695250 write 0x024913 thru 0x02d699 (0x8d87 bytes) 000899 1787839523.755418 mapread 0x0338c6 thru 0x03395a (0x95 bytes) 000900 1787839523.762308 punch from 0x03205e to 0x03d4cd (0xb470 bytes) 000901 1787839523.769381 trunc from 0x008f03 to 0x00badc (0x2bda bytes) 000902 1787839523.775154 trunc from 0x03454e to 0x0387dc (0x428f bytes) 000903 1787839523.784396 punch from 0x03b8bb to 0x03d4cd (0x1c13 bytes) 000904 1787839523.798589 fallocate 0x016c24 thru 0x01b803 (0x4be0 bytes) 000905 1787839523.813633 mapread 0x03a386 thru 0x03d4cd (0x3148 bytes) 000906 1787839523.836636 punch from 0x01ba33 to 0x02b93e (0xff0c bytes) 000907 1787839523.880181 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d4ce): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4ce): Mirror 30: 0xae0957e4 (stale) Mirror 31: 0xae0957e4 (stale) Mirror 32: 0xae0957e4 (stale) Mirror 33: 0xae0957e4 (stale) Mirror 34: 0xae0957e4 (stale) Mirror 35: 0xae0957e4 (stale) CRC-32 checksum value for chunk [0, 0x3d4ce): Mirror 1: 0x8d38f36f Mirror 36: 0x8d38f36f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4ce: skipped 000908 1787839526.019439 fallocate 0x00e0d2 thru 0x01b528 (0xd457 bytes) 000909 1787839526.064559 trunc from 0x019fbd to 0x022e5f (0x8ea3 bytes) 000910 1787839526.070246 mapread 0x030aa4 thru 0x034a3f (0x3f9c bytes) 000911 1787839526.100512 MIRROR_RESYNC 8 mirrors 000912 1787839529.145637 read 0x016240 thru 0x01de95 (0x7c56 bytes) 000913 1787839529.154313 write 0x001a5f thru 0x00a081 (0x8623 bytes) 000914 1787839529.196765 trunc from 0x0342e5 to 0x03d4cd (0x91e9 bytes) 000916 1787839529.214832 mapread 0x025cc3 thru 0x033746 (0xda84 bytes) 000917 1787839529.329409 mapread 0x02541c thru 0x030d48 (0xb92d bytes) 000919 1787839529.330980 mapread 0x00929c thru 0x009d1e (0xa83 bytes) 000920 1787839529.337576 write 0x0234b8 thru 0x0292a4 (0x5ded bytes) 000921 1787839529.339109 punch from 0x024e1e to 0x02f0d2 (0xa2b5 bytes) 000922 1787839529.386975 write 0x02a8c5 thru 0x02e417 (0x3b53 bytes) 000923 1787839529.403022 trunc from 0x03d4ce to 0x0384f6 (0x4fd7 bytes) 000924 1787839529.443279 write 0x00c617 thru 0x013ef9 (0x78e3 bytes) 000925 1787839529.464509 trunc from 0x014524 to 0x020caa (0xc787 bytes) 000926 1787839529.483657 read 0x031639 thru 0x037a74 (0x643c bytes) 000928 1787839529.484957 mapwrite 0x00ff06 thru 0x01454c (0x4647 bytes) 000929 1787839529.543335 read 0x030aa8 thru 0x0384f6 (0x7a4f bytes) 000930 1787839529.544175 punch from 0x004d7f to 0x007d25 (0x2fa7 bytes) 000931 1787839529.551210 trunc from 0x0384f7 to 0x038986 (0x490 bytes) 000932 1787839529.567644 trunc from 0x02da97 to 0x032335 (0x489f bytes) 000933 1787839529.574800 mapread 0x01ae42 thru 0x01ee60 (0x401f bytes) 000934 1787839529.576267 read 0x023978 thru 0x02b4c4 (0x7b4d bytes) 000935 1787839529.577425 punch from 0x00fcad to 0x01b4a5 (0xb7f9 bytes) 000936 1787839529.610838 MIRROR_RESYNC 8 mirrors 000937 1787839532.687473 MIRROR_RESYNC 8 mirrors 000939 1787839534.041824 mapread 0x0102b1 thru 0x01d4b8 (0xd208 bytes) 000940 1787839534.095085 punch from 0x0206d9 to 0x02de41 (0xd769 bytes) 000941 1787839534.143471 trunc from 0x01b2ee to 0x027ce9 (0xc9fc bytes) 000942 1787839534.179429 punch from 0x01fb74 to 0x028de2 (0x926f bytes) 000943 1787839534.193138 fallocate 0x02d86c thru 0x035d58 (0x84ed bytes) 000945 1787839534.208440 trunc from 0x038987 to 0x03c664 (0x3cde bytes) 000946 1787839534.288261 read 0x00be85 thru 0x016457 (0xa5d3 bytes) 000947 1787839534.340310 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c665): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c665): Mirror 30: 0x805fe681 (stale) Mirror 31: 0x805fe681 (stale) Mirror 32: 0x805fe681 (stale) Mirror 33: 0x805fe681 (stale) Mirror 34: 0x805fe681 (stale) Mirror 35: 0x805fe681 (stale) Mirror 36: 0x805fe681 (stale) CRC-32 checksum value for chunk [0, 0x3c665): Mirror 1: 0x121f412b Mirror 37: 0x121f412b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c665: skipped 000948 1787839535.697480 punch from 0x02d431 to 0x031a48 (0x4618 bytes) 000949 1787839535.747411 read 0x01a515 thru 0x01a8af (0x39b bytes) 000950 1787839535.754251 read 0x0172ae thru 0x01f15e (0x7eb1 bytes) 000951 1787839535.765209 mapread 0x021629 thru 0x027fcc (0x69a4 bytes) 000952 1787839535.766887 read 0x009bb7 thru 0x00cb1f (0x2f69 bytes) 000954 1787839535.768826 punch from 0x014043 to 0x018ba0 (0x4b5e bytes) 000955 1787839535.773740 mapwrite 0x009223 thru 0x01012d (0x6f0b bytes) 000956 1787839535.796532 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c665): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c665): Mirror 30: 0x805fe681 (stale) Mirror 31: 0x805fe681 (stale) Mirror 32: 0x805fe681 (stale) Mirror 33: 0x805fe681 (stale) Mirror 34: 0x805fe681 (stale) Mirror 35: 0x805fe681 (stale) Mirror 36: 0x805fe681 (stale) Mirror 37: 0x121f412b (stale) CRC-32 checksum value for chunk [0, 0x3c665): Mirror 1: 0x46f752c3 Mirror 38: 0x46f752c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c665: skipped 000957 1787839537.610138 write 0x03672b thru 0x03d5d2 (0x6ea8 bytes) 000958 1787839537.667117 mapread 0x02649f thru 0x02801a (0x1b7c bytes) 000959 1787839537.686196 trunc from 0x008acc to 0x010f19 (0x844e bytes) 000960 1787839537.690236 read 0x024b85 thru 0x02d6b9 (0x8b35 bytes) 000961 1787839537.691481 punch from 0x006452 to 0x015b36 (0xf6e5 bytes) 000962 1787839537.697283 mapread 0x0002f2 thru 0x000bd8 (0x8e7 bytes) 000963 1787839537.713749 MIRROR_RESYNC 10 mirrors 000964 1787839540.136438 read 0x02894f thru 0x02bc58 (0x330a bytes) 000965 1787839540.142498 read 0x02b366 thru 0x030a45 (0x56e0 bytes) 000966 1787839540.153476 punch from 0x02f587 to 0x0337cc (0x4246 bytes) 000967 1787839540.211394 trunc from 0x03d5d3 to 0x001e71 (0x3b761 bytes) 000968 1787839540.252219 write 0x005564 thru 0x00e7a5 (0x9242 bytes) 000969 1787839540.262790 trunc from 0x00433c to 0x00728a (0x2f4f bytes) 000970 1787839540.319284 trunc from 0x00e7a6 to 0x025031 (0x1688c bytes) 000971 1787839540.376059 mapread 0x00e248 thru 0x016bbf (0x8978 bytes) 000972 1787839540.456378 fallocate 0x000010 thru 0x0027b2 (0x27a3 bytes) 000973 1787839540.462119 read 0x003948 thru 0x003e1f (0x4d8 bytes) 000974 1787839540.504846 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25032): Mirror 30: 0x1971e187 (stale) Mirror 31: 0x1971e187 (stale) Mirror 33: 0x1971e187 (stale) Mirror 34: 0x1971e187 (stale) Mirror 35: 0x1971e187 (stale) Mirror 36: 0x1971e187 (stale) Mirror 37: 0x1971e187 (stale) Mirror 38: 0x1971e187 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25032): Mirror 30: 0x1971e187 (stale) Mirror 31: 0x1971e187 (stale) Mirror 33: 0x1971e187 (stale) Mirror 34: 0x1971e187 (stale) Mirror 35: 0x1971e187 (stale) Mirror 36: 0x1971e187 (stale) Mirror 37: 0x1971e187 (stale) Mirror 38: 0x1971e187 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25032: skipped 000975 1787839541.441440 write 0x00c1be thru 0x0197eb (0xd62e bytes) 000976 1787839541.455058 punch from 0x00aec6 to 0x018ad2 (0xdc0d bytes) 000977 1787839541.465785 mapwrite 0x02e532 thru 0x0399d7 (0xb4a6 bytes) 000978 1787839541.542492 fallocate 0x03a3d4 thru 0x03ffff (0x5c2c bytes) 000979 1787839541.559694 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 30: 0x626eddc0 (stale) Mirror 31: 0x626eddc0 (stale) Mirror 33: 0x626eddc0 (stale) Mirror 34: 0x626eddc0 (stale) Mirror 35: 0x626eddc0 (stale) Mirror 36: 0x626eddc0 (stale) Mirror 37: 0x626eddc0 (stale) Mirror 38: 0x626eddc0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9baabcb Mirror 39: 0xd9baabcb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1787839543.322480 mapwrite 0x01a683 thru 0x023f43 (0x98c1 bytes) 000981 1787839543.432062 write 0x039b58 thru 0x03ffff (0x64a8 bytes) 000983 1787839543.449187 punch from 0x02ed96 to 0x02f236 (0x4a1 bytes) 000984 1787839543.456096 mapread 0x005c58 thru 0x010099 (0xa442 bytes) 000985 1787839543.473501 punch from 0x031708 to 0x03ffff (0xe8f8 bytes) 000986 1787839543.494271 mapwrite 0x00b9f0 thru 0x00f73f (0x3d50 bytes) 000987 1787839543.518105 mapwrite 0x0057b5 thru 0x0138da (0xe126 bytes) 000988 1787839543.563135 mapwrite 0x03bde7 thru 0x03ffff (0x4219 bytes) 000989 1787839543.605061 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 31: 0x626eddc0 (stale) Mirror 33: 0x626eddc0 (stale) Mirror 34: 0x626eddc0 (stale) Mirror 35: 0x626eddc0 (stale) Mirror 36: 0x626eddc0 (stale) Mirror 37: 0x626eddc0 (stale) Mirror 38: 0x626eddc0 (stale) Mirror 39: 0xd9baabcb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 31: 0x626eddc0 (stale) Mirror 33: 0x626eddc0 (stale) Mirror 34: 0x626eddc0 (stale) Mirror 35: 0x626eddc0 (stale) Mirror 36: 0x626eddc0 (stale) Mirror 37: 0x626eddc0 (stale) Mirror 38: 0x626eddc0 (stale) Mirror 39: 0xd9baabcb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000990 1787839544.400060 trunc from 0x03ca0b to 0x03ffff (0x35f5 bytes) 000991 1787839544.407322 mapread 0x0068f9 thru 0x011a7b (0xb183 bytes) 000992 1787839544.435025 read 0x0233c8 thru 0x02fb34 (0xc76d bytes) 000993 1787839544.444994 write 0x028a3c thru 0x030f6e (0x8533 bytes) 000994 1787839544.455711 read 0x009c87 thru 0x009fca (0x344 bytes) 000995 1787839544.456780 read 0x0388a4 thru 0x03ffff (0x775c bytes) 000996 1787839544.467363 mapread 0x00085f thru 0x0093ee (0x8b90 bytes) 000998 1787839544.469085 mapread 0x035417 thru 0x03c539 (0x7123 bytes) 000999 1787839544.470481 trunc from 0x040000 to 0x0096ed (0x36912 bytes) 001000 1787839544.488235 fallocate 0x02f427 thru 0x03f1cc (0xfda6 bytes) All operations completed A-OK! PASS 70a (144s) == sanity-flr test 71: check mirror extend parent fid ==== 10:05:51 (1787839551) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1c9:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 10:06:05 (1787839565) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213003 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219193 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 10:06:16 (1787839576) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7670816 .. Extending file size to 1351648 .. Extending file size to 8256096 .. Extending file size to 8208672 .. Extending file size to 8263520 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8288736 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9087744 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9067232 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62656 62657 62660 62661 62664 done 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) 192.168.202.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:07:40 (1787839660) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2406944 .. Extending file size to 8950176 .. Extending file size to 2994048 .. Extending file size to 4794688 .. Extending file size to 5658976 .. Extending file size to 5706272 .. Extending file size to 8621920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9141312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9271648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64441 64443 64444 64445 64446 done 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) 192.168.202.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:09:03 (1787839743) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2316 1268988 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17896 3589124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25788 3581232 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43684 7170356 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2316 1268988 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17896 3589124 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25788 3581232 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43684 7170356 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0838566 s, 25.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.317272 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x1c5475f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:09:26 (1787839766) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875095 s, 24.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 10:09:36 (1787839776) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:09:44 (1787839784) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155634 s, 13.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:09:53 (1787839793) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135985 s, 15.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21e:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:10:03 (1787839803) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117367 s, 17.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x220:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:10:38 (1787839838) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553243 s, 19.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x221:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x221:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x220:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:11:33 (1787839893) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0922579 s, 22.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:12:28 (1787839948) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg229-server: Pool lustre.test_204f created Adding targets to pool oleg229-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg229-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109769 s, 19.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg229-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg229-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg229-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:13:43 (1787840023) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:13:51 (1787840031) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:13:59 (1787840039) oleg229-server: Pool lustre.test_206 created Adding targets to pool oleg229-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg229-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg229-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg229-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg229-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:14:29 (1787840069) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206701 s, 5.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464103 s, 22.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x69907efd Mirror 3: 0x69907efd PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:14:42 (1787840082) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:14:46 (1787840086) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:14:50 (1787840090) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0617934 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x22d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10e0000 Read this (4 bytes): BBBB 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:15:29 (1787840129) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0321857 s, 255 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:15:37 (1787840137) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056288 s, 18.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x120:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 10:15:39.000000000 -0400 Modify: 2026-08-27 10:15:39.000000000 -0400 Change: 2026-08-27 10:15:41.000000000 -0400 Birth: 2026-08-27 10:15:38.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:15:47 (1787840147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474133 s, 22.1 MB/s oleg229-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg229-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg229-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:16:11 (1787840171) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.334188 s, 123 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0237648 s, 172 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 10:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 10:16 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:16:20 (1787840180) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:16:28 (1787840188) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1975 sec =========== 10:16:37 (1787840197) === sanity-flr: start cleanup 10:16:40 (1787840200) === === sanity-flr: finish cleanup 10:16:48 (1787840208) ===