-----============= acceptance-small: sanity-flr ============----- Wed May 13 02:07:10 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:07:28 (1778652448) === oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8907c6169000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8907c6169000.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:08:14 (1778652494) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 02:08:17 (1778652497) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:08:34 (1778652514) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:08:37 (1778652517) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:08:40 (1778652520) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:09:06 (1778652546) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:09:10 (1778652550) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 02:09:13 (1778652553) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8907c6169000.stats=clear osc.lustre-OST0001-osc-ffff8907c6169000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:09:20 (1778652560) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:09:31 (1778652571) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:09:40 (1778652580) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:09:43 (1778652583) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:09:50 (1778652590) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:09:59 (1778652599) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:10:04 (1778652604) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:10:10 (1778652610) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:10:16 (1778652616) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:10:25 (1778652625) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:10:30 (1778652630) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.519553 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783683 s, 13.4 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:10:37 (1778652637) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.285208 s, 40.4 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:10:44 (1778652644) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:10:52 (1778652652) Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (51s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:11:43 (1778652703) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.164784 s, 25.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (26s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:12:09 (1778652729) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (48s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:12:57 (1778652777) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82715 s, 28.7 MB/s -rw-r--r-- 1 root root 52428800 May 13 02:13 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.595s user 0m0.000s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.581s user 0m0.001s sys 0m0.197s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.773s user 0m0.002s sys 0m0.204s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8907c6169000.ost_server_uuid in FULL state after 0 sec PASS 33b (54s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:13:51 (1778652831) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:13:53 (1778652833) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:13:55 (1778652835) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 02:13:57 (1778652837) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 02:14:02 (1778652842) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.357721 s, 35.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:14:09 (1778652849) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270848 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166642 s, 62.9 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:14:13 (1778652853) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:14:18 (1778652858) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109272 s, 937 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000720538 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201552 s, 508 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111086 s, 922 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 02:14:24 (1778652864) create test files with size 15737069 .. 102c621551d9c5ffaed46a3f4c2d0eec - 5974f95bbf3d67b6f9c511eadaaaaeec - bc91cd2310aa82cdfe63b8b1b6d5a474 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 02:14:35 (1778652875) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:14:42 (1778652882) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:14:46 (1778652886) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505214 s, 20.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191724 s, 54.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 02:14:52 (1778652892) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.049148 s, 42.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.22105 s, 19.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118085 s, 17.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 02:15:01 (1778652901) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 02:15:02 (1778652902) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:15:05 (1778652905) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303015 s, 34.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 02:15:09 (1778652909) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.159446 s, 19.7 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:15:14 (1778652914) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:15:20 (1778652920) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.193262 s, 54.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:15:28 (1778652928) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8907c5cf6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.178236 s, 58.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8907c7cad000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:15:37 (1778652937) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (21s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:15:58 (1778652958) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (28s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:16:26 (1778652986) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:16:31 (1778652991) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.262703 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.206958 s, 15.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc26b6a6a (nosync,stale) Mirror 2: 0xc26b6a6a (nosync,stale) ** verify mirror 2 doesn't change original checksum: a3210aca81318b3e576b600afb7c10eb - mirror 1 checksum: 07f06b6586ec2190dfccc19da659d772 - mirror 2 checksum: a3210aca81318b3e576b600afb7c10eb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 13 02:16:34 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 13 02:16:34 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a3210aca81318b3e576b600afb7c10eb - mirror 1 checksum: 07f06b6586ec2190dfccc19da659d772 - mirror 2 checksum: 07f06b6586ec2190dfccc19da659d772 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 13 02:16:34 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 13 02:16:34 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:16:40 (1778653000) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372168 s, 28.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:16:46 (1778653006) Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 02:16:57 (1778653017) shut down facet: mds1 facet_host: oleg138-server facet_failover_host: oleg138-server Failover mds1 to oleg138-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 02:17:19 (1778653039) targets are mounted 02:17:19 (1778653039) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903052 s, 11.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:17:43 (1778653063) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0839248 s, 244 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0444228 s, 461 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (18s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:18:00 (1778653080) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.106253 s, 193 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162022 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (16s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:18:16 (1778653096) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0138811 s, 1.2 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:18:29 (1778653109) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0627345 s, 326 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:18:39 (1778653119) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00415383 s, 986 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0468202 s, 87.5 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:18:48 (1778653128) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:19:10 (1778653150) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:19:39 (1778653179) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:20:04 (1778653204) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:20:30 (1778653230) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 02:20:38 (1778653238) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57166 mirror split pid 57168 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57166 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57168 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:21:47 (1778653307) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 3309 fallocating to largest ever: 0x154b 000001 1778653309.760374 fallocate 0x0008ae thru 0x00154a (0xc9d bytes) truncating to largest ever: 0x186c4 000002 1778653309.779012 trunc from 00000000 to 0x0186c3 (0x186c4 bytes) 000003 1778653309.811604 write 0x013e12 thru 0x01fc20 (0xbe0f bytes) 000006 1778653309.826210 read 0x012c0b thru 0x01e07f (0xb475 bytes) truncating to largest ever: 0x39703 000007 1778653309.852139 trunc from 0x01fc21 to 0x039702 (0x19ae2 bytes) 000009 1778653309.901426 mapread 0x016334 thru 0x01bbb3 (0x5880 bytes) 000010 1778653309.915128 write 0x03c735 thru 0x03ceaa (0x776 bytes) 000012 1778653309.916855 mapread 0x02bd14 thru 0x03a0f4 (0xe3e1 bytes) 000013 1778653310.001955 trunc from 0x01b249 to 0x0223bf (0x7177 bytes) 000014 1778653310.040333 punch from 0x019b78 to 0x01c436 (0x28bf bytes) 000015 1778653310.064734 MIRROR_EXTEND to 2 mirrors 000016 1778653311.069807 mapread 0x0318e0 thru 0x035f9e (0x46bf bytes) 000017 1778653311.152625 punch from 0x003b60 to 0x012226 (0xe6c7 bytes) 000018 1778653311.202722 trunc from 0x030022 to 0x039dfe (0x9ddd bytes) 000019 1778653311.219015 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ceab): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ceab): Mirror 2: 0x99ab0731 (stale) CRC-32 checksum value for chunk [0, 0x3ceab): Mirror 1: 0x99ab0731 Mirror 3: 0x99ab0731 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ceab: skipped 000020 1778653312.365740 punch from 0x00610a to 0x0076a3 (0x159a bytes) 000021 1778653312.421423 trunc from 0x03ceab to 0x02f547 (0xd963 bytes) 000022 1778653312.475077 mapread 0x014c2b thru 0x015b5a (0xf30 bytes) 000023 1778653312.491911 trunc from 0x02f548 to 0x02cd3a (0x280d bytes) 000024 1778653312.518898 trunc from 0x004bca to 0x008570 (0x39a7 bytes) 000025 1778653312.533053 write 0x01e3ad thru 0x029c07 (0xb85b bytes) 000026 1778653312.549881 read 0x02a052 thru 0x02cd3a (0x2ce9 bytes) 000027 1778653312.556645 trunc from 0x02cd3b to 0x034b95 (0x7e5b bytes) 000028 1778653312.577461 trunc from 0x0305df to 0x034b95 (0x45b7 bytes) 000029 1778653312.583569 punch from 0x014592 to 0x020887 (0xc2f6 bytes) 000030 1778653312.603637 trunc from 0x034b96 to 0x03112e (0x3a67 bytes) fallocating to largest ever: 0x40000 000031 1778653312.620926 fallocate 0x03de34 thru 0x03ffff (0x21cc bytes) 000032 1778653312.629352 punch from 0x01864f to 0x018951 (0x303 bytes) 000033 1778653312.636379 trunc from 0x01b1b6 to 0x023a85 (0x88d0 bytes) 000034 1778653312.642637 trunc from 0x040000 to 0x006483 (0x39b7c bytes) 000035 1778653312.662323 mapread 0x001a2f thru 0x001a6e (0x40 bytes) 000036 1778653312.671816 mapwrite 0x03f023 thru 0x03ffff (0xfdd bytes) 000038 1778653312.723942 fallocate 0x0256ef thru 0x027cae (0x25c0 bytes) 000039 1778653312.733684 read 0x03b918 thru 0x03ffff (0x46e8 bytes) 000040 1778653312.743529 mapwrite 0x037a36 thru 0x03ffff (0x85ca bytes) 000041 1778653312.812448 punch from 0x02bd75 to 0x034363 (0x85ef bytes) 000042 1778653312.830539 trunc from 0x023cc4 to 0x030cc0 (0xcffd bytes) 000043 1778653312.845733 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57ceebef (stale) Mirror 3: 0x57ceebef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb4f41c48 Mirror 4: 0xb4f41c48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000045 1778653314.421300 trunc from 0x040000 to 0x00f255 (0x30daa bytes) 000046 1778653314.484955 read 0x00abf7 thru 0x00c427 (0x1831 bytes) 000047 1778653314.491693 fallocate 0x01d815 thru 0x01e96e (0x115a bytes) 000048 1778653314.497627 read 0x008681 thru 0x00f255 (0x6bd5 bytes) 000049 1778653314.506941 punch from 0x00b9c5 to 0x00f255 (0x3891 bytes) 000050 1778653314.511972 fallocate 0x030f63 thru 0x03e6c2 (0xd760 bytes) 000051 1778653314.517062 trunc from 0x00f256 to 0x01b5c7 (0xc372 bytes) 000052 1778653314.539495 punch from 0x0136fe to 0x01494e (0x1251 bytes) 000053 1778653314.544911 trunc from 0x01b5c8 to 0x00d659 (0xdf6e bytes) 000054 1778653314.559393 write 0x0367c5 thru 0x03ffff (0x983b bytes) 000055 1778653314.560949 read 0x03072e thru 0x030a79 (0x34c bytes) 000056 1778653314.573577 fallocate 0x0177e8 thru 0x01d114 (0x592d bytes) 000057 1778653314.579205 trunc from 0x040000 to 0x003be9 (0x3c416 bytes) 000058 1778653314.600513 trunc from 0x0017b5 to 0x003035 (0x1881 bytes) 000059 1778653314.605971 fallocate 0x02a66d thru 0x02c671 (0x2005 bytes) 000061 1778653314.611762 punch from 0x02866e to 0x029a68 (0x13fb bytes) 000062 1778653314.616530 mapread 0x0227b9 thru 0x023046 (0x88e bytes) 000063 1778653314.626507 fallocate 0x00265b thru 0x008ab8 (0x645e bytes) 000064 1778653314.631947 read 0x00e845 thru 0x00ff04 (0x16c0 bytes) 000065 1778653314.648901 fallocate 0x027132 thru 0x03030c (0x91db bytes) 000066 1778653314.654351 write 0x023959 thru 0x02ffd7 (0xc67f bytes) 000067 1778653314.656524 read 0x0018b9 thru 0x00eee0 (0xd628 bytes) 000068 1778653314.657728 read 0x01a08f thru 0x026555 (0xc4c7 bytes) 000069 1778653314.658488 punch from 0x02db36 to 0x02fc30 (0x20fb bytes) 000070 1778653314.672793 trunc from 0x02ffd8 to 0x02c8f4 (0x36e3 bytes) 000072 1778653314.690865 trunc from 0x02c8f5 to 0x0108e4 (0x1c010 bytes) 000073 1778653314.711364 mapwrite 0x02d371 thru 0x033d4a (0x69da bytes) 000074 1778653314.767933 write 0x0302de thru 0x035ab7 (0x57da bytes) 000075 1778653314.770052 mapread 0x023cd8 thru 0x029f33 (0x625c bytes) 000076 1778653314.779864 MIRROR_RESYNC 4 mirrors 000077 1778653316.015691 mapread 0x01c4f2 thru 0x0286b8 (0xc1c7 bytes) 000078 1778653316.073077 read 0x000812 thru 0x006a94 (0x6283 bytes) 000079 1778653316.099882 trunc from 0x0090b1 to 0x012cb0 (0x9c00 bytes) 000080 1778653316.148413 mapread 0x021748 thru 0x02ef91 (0xd84a bytes) 000081 1778653316.168785 punch from 0x0181db to 0x0235ce (0xb3f4 bytes) 000082 1778653316.172765 punch from 0x0086f8 to 0x0117a1 (0x90aa bytes) 000083 1778653316.176590 write 0x017af5 thru 0x01bd27 (0x4233 bytes) 000084 1778653316.186057 trunc from 0x0178ce to 0x021c1e (0xa351 bytes) 000085 1778653316.200335 read 0x0062aa thru 0x008df1 (0x2b48 bytes) 000086 1778653316.227688 write 0x02ad35 thru 0x0357dd (0xaaa9 bytes) 000087 1778653316.230866 trunc from 0x035ab8 to 0x004146 (0x31971 bytes) 000088 1778653316.258296 fallocate 0x00acd7 thru 0x013eec (0x9216 bytes) 000089 1778653316.274761 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13eed): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13eed): Mirror 2: 0xebb831e3 (stale) Mirror 3: 0xebb831e3 (stale) Mirror 4: 0xebb831e3 (stale) CRC-32 checksum value for chunk [0, 0x13eed): Mirror 1: 0xebb831e3 Mirror 5: 0xebb831e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13eed: skipped 000090 1778653317.039818 mapread 0x0109d2 thru 0x013eec (0x351b bytes) 000091 1778653317.076337 trunc from 0x0068d0 to 0x013eec (0xd61d bytes) 000092 1778653317.168446 read 0x01375d thru 0x013eec (0x790 bytes) 000093 1778653317.176659 mapwrite 0x006ce9 thru 0x00a4ec (0x3804 bytes) 000094 1778653317.221884 trunc from 0x013eed to 0x03d6dd (0x297f1 bytes) 000095 1778653317.302868 punch from 0x036a25 to 0x0399f1 (0x2fcd bytes) 000096 1778653317.309774 trunc from 0x0375e4 to 0x03d6dd (0x60fa bytes) 000097 1778653317.323657 mapwrite 0x038f02 thru 0x03d5c1 (0x46c0 bytes) 000098 1778653317.361399 write 0x023382 thru 0x02e8cb (0xb54a bytes) 000100 1778653317.392369 punch from 0x02f867 to 0x034af3 (0x528d bytes) 000101 1778653317.402484 write 0x0078bb thru 0x01520e (0xd954 bytes) 000102 1778653317.424695 read 0x002aec thru 0x00e309 (0xb81e bytes) 000103 1778653317.425556 mapread 0x01fcd3 thru 0x02303b (0x3369 bytes) 000105 1778653317.434234 read 0x01b1e3 thru 0x027567 (0xc385 bytes) 000107 1778653317.458864 trunc from 0x02a9f8 to 0x03135d (0x6966 bytes) 000108 1778653317.511726 punch from 0x03689d to 0x03951b (0x2c7f bytes) 000109 1778653317.524270 write 0x02ca83 thru 0x033d65 (0x72e3 bytes) 000111 1778653317.555429 punch from 0x00e080 to 0x0178bc (0x983d bytes) 000112 1778653317.582303 read 0x00ccf0 thru 0x0181f6 (0xb507 bytes) 000113 1778653317.605139 punch from 0x030546 to 0x03a870 (0xa32b bytes) 000114 1778653317.644211 trunc from 0x03d6de to 0x02fe64 (0xd879 bytes) 000115 1778653317.701548 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fe65): Mirror 3: 0x96545000 (stale) Mirror 4: 0x96545000 (stale) Mirror 5: 0x5cc114ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fe65): Mirror 3: 0x96545000 (stale) Mirror 4: 0x96545000 (stale) Mirror 5: 0x5cc114ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fe65: skipped 000116 1778653318.316413 punch from 0x02da88 to 0x02fe64 (0x23dd bytes) 000117 1778653318.327556 mapwrite 0x02720b thru 0x028aaf (0x18a5 bytes) 000118 1778653318.354747 mapread 0x00c685 thru 0x00e372 (0x1cee bytes) 000119 1778653318.374012 trunc from 0x02fe65 to 0x00f582 (0x208e2 bytes) 000120 1778653318.445612 mapwrite 0x01ffca thru 0x0217e0 (0x1817 bytes) 000122 1778653318.535633 punch from 0x00821e to 0x0100f4 (0x7ed7 bytes) 000123 1778653318.548356 fallocate 0x00377b thru 0x00ab17 (0x739d bytes) 000124 1778653318.557398 mapread 0x00a39a thru 0x01a12e (0xfd95 bytes) 000125 1778653318.580808 mapwrite 0x003495 thru 0x00a570 (0x70dc bytes) 000126 1778653318.607691 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x217e1): Mirror 3: 0x606be7a7 (stale) Mirror 4: 0x606be7a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x217e1): Mirror 3: 0x606be7a7 (stale) Mirror 4: 0x606be7a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x217e1: skipped 000127 1778653319.320980 punch from 0x00e787 to 0x01aec3 (0xc73d bytes) 000128 1778653319.338974 MIRROR_RESYNC 3 mirrors 000129 1778653320.490616 MIRROR_EXTEND to 4 mirrors 000131 1778653321.642800 fallocate 0x0198c3 thru 0x01de46 (0x4584 bytes) 000132 1778653321.696326 trunc from 0x011349 to 0x011ece (0xb86 bytes) 000133 1778653321.701994 punch from 0x00a87f to 0x010b45 (0x62c7 bytes) 000134 1778653321.708232 punch from 0x00bf80 to 0x00f189 (0x320a bytes) 000135 1778653321.711968 mapwrite 0x00a1d0 thru 0x015e5c (0xbc8d bytes) 000136 1778653321.763260 fallocate 0x005872 thru 0x00bf06 (0x6695 bytes) 000137 1778653321.772388 mapread 0x005291 thru 0x008b88 (0x38f8 bytes) 000138 1778653321.774693 fallocate 0x03c4c6 thru 0x03ffff (0x3b3a bytes) 000139 1778653321.784328 punch from 0x0205e8 to 0x020dcd (0x7e6 bytes) 000140 1778653321.790672 trunc from 0x040000 to 0x0391a7 (0x6e58 bytes) 000141 1778653321.823902 punch from 0x0041bc to 0x00a715 (0x655a bytes) 000142 1778653321.831448 write 0x03008b thru 0x03bebb (0xbe31 bytes) 000143 1778653321.859493 MIRROR_RESYNC 4 mirrors 000144 1778653323.765695 mapread 0x01c711 thru 0x01e9a9 (0x2299 bytes) 000145 1778653323.803248 trunc from 0x035dfc to 0x03bebb (0x60c0 bytes) 000146 1778653323.878234 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0xbe1bf317 (stale) Mirror 5: 0xbe1bf317 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0xbe1bf317 (stale) Mirror 5: 0xbe1bf317 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bebc: skipped 000147 1778653324.838763 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0xbe1bf317 (stale) Mirror 5: 0xbe1bf317 (stale) CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 1: 0xd3463836 Mirror 6: 0xd3463836 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bebc: skipped 000148 1778653326.782706 mapwrite 0x033a13 thru 0x037b61 (0x414f bytes) 000149 1778653326.933284 mapwrite 0x00bae4 thru 0x0134b2 (0x79cf bytes) 000150 1778653326.987663 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0xbe1bf317 (stale) Mirror 5: 0xbe1bf317 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0xbe1bf317 (stale) Mirror 5: 0xbe1bf317 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bebc: skipped 000152 1778653327.493895 fallocate 0x0212ad thru 0x0251e3 (0x3f37 bytes) 000154 1778653327.500999 write 0x013dd5 thru 0x023517 (0xf743 bytes) 000155 1778653327.518477 write 0x0062b0 thru 0x00ce12 (0x6b63 bytes) 000156 1778653327.532720 trunc from 0x0058ae to 0x005af7 (0x24a bytes) 000158 1778653327.540513 trunc from 0x0382bb to 0x03bebb (0x3c01 bytes) 000159 1778653327.570438 MIRROR_RESYNC 3 mirrors 000160 1778653328.934601 MIRROR_RESYNC 3 mirrors 000161 1778653329.553545 mapwrite 0x02843f thru 0x02f6f9 (0x72bb bytes) 000162 1778653329.669344 write 0x025f85 thru 0x034622 (0xe69e bytes) 000163 1778653329.684742 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 4: 0x93599b64 (stale) Mirror 5: 0x93599b64 (stale) CRC-32 checksum value for chunk [0, 0x3bebc): Mirror 1: 0x926e33d1 Mirror 6: 0x926e33d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bebc: skipped 000164 1778653330.872916 punch from 0x01adeb to 0x01b695 (0x8ab bytes) 000165 1778653330.938106 read 0x0396fc thru 0x03bebb (0x27c0 bytes) 000166 1778653330.945987 fallocate 0x000104 thru 0x003438 (0x3335 bytes) 000167 1778653330.957235 mapread 0x02bb14 thru 0x03b63a (0xfb27 bytes) 000168 1778653331.020298 trunc from 0x033b67 to 0x03bebb (0x8355 bytes) 000170 1778653331.031999 trunc from 0x0102e0 to 0x01d69e (0xd3bf bytes) 000171 1778653331.045455 fallocate 0x03825a thru 0x03b534 (0x32db bytes) 000172 1778653331.053783 write 0x006f42 thru 0x00ba95 (0x4b54 bytes) 000173 1778653331.071288 trunc from 0x024922 to 0x02bd58 (0x7437 bytes) 000174 1778653331.083446 MIRROR_RESYNC 4 mirrors 000175 1778653332.727039 MIRROR_EXTEND to 5 mirrors 000176 1778653333.835458 trunc from 0x03bebc to 0x01f6af (0x1c80c bytes) 000177 1778653333.939047 trunc from 0x016a3c to 0x01f6af (0x8c74 bytes) 000178 1778653333.945803 trunc from 0x004c13 to 0x0067b8 (0x1ba6 bytes) 000179 1778653333.954619 write 0x0134ae thru 0x016c25 (0x3778 bytes) 000180 1778653333.967316 read 0x004708 thru 0x008771 (0x406a bytes) 000181 1778653333.977711 mapwrite 0x006a1a thru 0x00d06e (0x6655 bytes) 000183 1778653334.051821 write 0x010129 thru 0x01b3c6 (0xb29e bytes) 000184 1778653334.062733 trunc from 0x01ab1f to 0x01ae8c (0x36e bytes) 000186 1778653334.093888 write 0x0228b7 thru 0x026b32 (0x427c bytes) 000187 1778653334.095099 trunc from 0x025d08 to 0x026b32 (0xe2b bytes) 000189 1778653334.118003 punch from 0x0227c1 to 0x026b32 (0x4372 bytes) 000191 1778653334.125982 read 0x010a9e thru 0x01e8d0 (0xde33 bytes) 000192 1778653334.148416 fallocate 0x02bee4 thru 0x03ac01 (0xed1e bytes) 000193 1778653334.155570 trunc from 0x029a04 to 0x0341a3 (0xa7a0 bytes) 000194 1778653334.163016 mapread 0x00e3d1 thru 0x01decb (0xfafb bytes) 000195 1778653334.165278 trunc from 0x0126cb to 0x020b34 (0xe46a bytes) 000196 1778653334.171387 fallocate 0x028dc7 thru 0x02dc18 (0x4e52 bytes) 000197 1778653334.176309 mapread 0x002564 thru 0x010821 (0xe2be bytes) 000198 1778653334.179615 mapwrite 0x021d86 thru 0x025fb9 (0x4234 bytes) 000199 1778653334.202572 trunc from 0x03ac02 to 0x001274 (0x3998d bytes) 000200 1778653334.227311 punch from 0x000633 to 0x001274 (0xc42 bytes) 000202 1778653334.232843 trunc from 0x001275 to 0x025507 (0x24293 bytes) 000203 1778653334.253894 mapread 0x01053b thru 0x01998c (0x9452 bytes) 000204 1778653334.319265 read 0x009846 thru 0x012308 (0x8ac3 bytes) 000205 1778653334.333102 punch from 0x01b5a3 to 0x02290c (0x736a bytes) 000206 1778653334.341382 mapwrite 0x01c45a thru 0x02517e (0x8d25 bytes) 000207 1778653334.431486 mapwrite 0x03fea1 thru 0x03fff8 (0x158 bytes) 000209 1778653334.537583 read 0x03219f thru 0x0382b3 (0x6115 bytes) 000210 1778653334.538350 punch from 0x00af20 to 0x010b8b (0x5c6c bytes) 000211 1778653334.549656 trunc from 0x03fff9 to 0x0011dc (0x3ee1c bytes) 000212 1778653334.627295 trunc from 0x0011dd to 0x015017 (0x13e3b bytes) 000213 1778653334.677210 mapread 0x00c32f thru 0x015017 (0x8ce9 bytes) 000214 1778653334.696591 punch from 0x011547 to 0x015017 (0x3ad1 bytes) 000215 1778653334.708498 write 0x01ad20 thru 0x026c6b (0xbf4c bytes) 000216 1778653334.711248 mapread 0x009ba7 thru 0x01611e (0xc578 bytes) 000217 1778653334.741626 read 0x024c82 thru 0x026c6b (0x1fea bytes) 000218 1778653334.742130 trunc from 0x026c6c to 0x02c8b4 (0x5c49 bytes) 000219 1778653334.758967 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c8b5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c8b5): Mirror 4: 0x63e00d55 (stale) Mirror 5: 0x63e00d55 (stale) Mirror 6: 0x63e00d55 (stale) Mirror 7: 0x63e00d55 (stale) CRC-32 checksum value for chunk [0, 0x2c8b5): Mirror 1: 0x101b6c19 Mirror 8: 0x101b6c19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c8b5: skipped 000220 1778653335.884478 trunc from 0x022004 to 0x027a89 (0x5a86 bytes) 000221 1778653335.968300 mapwrite 0x01d96a thru 0x02a0a9 (0xc740 bytes) 000222 1778653336.070864 trunc from 0x02c8b5 to 0x039aae (0xd1fa bytes) 000223 1778653336.107502 fallocate 0x005d6d thru 0x00fb30 (0x9dc4 bytes) 000226 1778653336.127065 write 0x02c9c9 thru 0x031ad8 (0x5110 bytes) 000227 1778653336.138533 fallocate 0x030c17 thru 0x03d3e5 (0xc7cf bytes) 000228 1778653336.148896 punch from 0x03cc2b to 0x03d3e5 (0x7bb bytes) 000229 1778653336.156736 read 0x0153bf thru 0x016e53 (0x1a95 bytes) 000230 1778653336.164497 fallocate 0x006482 thru 0x00ba3a (0x55b9 bytes) 000231 1778653336.173918 mapwrite 0x03ed0d thru 0x03ffff (0x12f3 bytes) 000232 1778653336.300927 mapread 0x018e11 thru 0x022d44 (0x9f34 bytes) 000233 1778653336.307936 trunc from 0x040000 to 0x01be7c (0x24183 bytes) 000234 1778653336.392386 punch from 0x009c38 to 0x00a08d (0x456 bytes) 000235 1778653336.417880 read 0x01853b thru 0x01be7c (0x3942 bytes) 000236 1778653336.418577 read 0x011988 thru 0x01622d (0x48a6 bytes) 000237 1778653336.476537 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1be7d): Mirror 4: 0x4668cd0b (stale) Mirror 5: 0x4668cd0b (stale) Mirror 6: 0x4668cd0b (stale) Mirror 8: 0x765f8a23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1be7d): Mirror 4: 0x4668cd0b (stale) Mirror 5: 0x4668cd0b (stale) Mirror 6: 0x4668cd0b (stale) Mirror 8: 0x765f8a23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1be7d: skipped 000238 1778653337.190048 punch from 0x002b2a to 0x00e7dc (0xbcb3 bytes) 000239 1778653337.201821 fallocate 0x020f80 thru 0x030bdd (0xfc5e bytes) 000240 1778653337.215459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30bde): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30bde): Mirror 4: 0x82a5ea75 (stale) Mirror 5: 0x82a5ea75 (stale) Mirror 6: 0x82a5ea75 (stale) Mirror 8: 0x93ac9b31 (stale) CRC-32 checksum value for chunk [0, 0x30bde): Mirror 1: 0xb3276da3 Mirror 9: 0xb3276da3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30bde: skipped 000241 1778653338.798668 read 0x02d409 thru 0x02f0a5 (0x1c9d bytes) 000242 1778653338.824102 mapread 0x011376 thru 0x020fce (0xfc59 bytes) 000243 1778653338.948457 fallocate 0x02ae72 thru 0x03687d (0xba0c bytes) 000244 1778653339.045732 mapread 0x00b173 thru 0x012c0d (0x7a9b bytes) 000245 1778653339.066312 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3687e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3687e): Mirror 4: 0xdc179ad3 (stale) Mirror 5: 0xdc179ad3 (stale) Mirror 6: 0xdc179ad3 (stale) Mirror 8: 0x57261741 (stale) Mirror 9: 0xbff50629 (stale) CRC-32 checksum value for chunk [0, 0x3687e): Mirror 1: 0xbff50629 Mirror 10: 0xbff50629 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3687e: skipped 000246 1778653340.213108 punch from 0x020dc2 to 0x02c5a3 (0xb7e2 bytes) 000247 1778653340.293819 trunc from 0x03687e to 0x01360c (0x23271 bytes) 000248 1778653340.340122 mapread 0x007d15 thru 0x00e80c (0x6af8 bytes) 000249 1778653340.364454 mapwrite 0x00d3a1 thru 0x0134b2 (0x6112 bytes) 000250 1778653340.394632 read 0x00c539 thru 0x01139f (0x4e67 bytes) 000251 1778653340.395319 punch from 0x01149c to 0x0135f8 (0x215d bytes) 000252 1778653340.403429 mapread 0x00448f thru 0x005dca (0x193c bytes) 000253 1778653340.404884 fallocate 0x00616a thru 0x00f825 (0x96bc bytes) 000254 1778653340.411785 read 0x009de5 thru 0x00eb39 (0x4d55 bytes) 000255 1778653340.413106 fallocate 0x01b223 thru 0x026be6 (0xb9c4 bytes) 000256 1778653340.420566 punch from 0x021885 to 0x0254a2 (0x3c1e bytes) 000257 1778653340.429933 punch from 0x01d378 to 0x01d889 (0x512 bytes) 000258 1778653340.436349 punch from 0x00aadd to 0x0192bb (0xe7df bytes) 000260 1778653340.444483 read 0x017c4b thru 0x01d1de (0x5594 bytes) 000261 1778653340.465252 fallocate 0x034a83 thru 0x0365dc (0x1b5a bytes) 000262 1778653340.471636 write 0x02215b thru 0x030e74 (0xed1a bytes) 000263 1778653340.480336 punch from 0x01c31d to 0x02131e (0x5002 bytes) 000264 1778653340.488067 write 0x00b596 thru 0x01400d (0x8a78 bytes) 000265 1778653340.489958 mapwrite 0x002afd thru 0x00f0ba (0xc5be bytes) 000266 1778653340.519743 trunc from 0x0365dd to 0x01e744 (0x17e98 bytes) 000267 1778653340.539926 mapwrite 0x00168a thru 0x004f4d (0x38c4 bytes) 000268 1778653340.571511 mapread 0x019404 thru 0x01e744 (0x5341 bytes) 000269 1778653340.607330 punch from 0x007eb5 to 0x00a8ad (0x29f9 bytes) 000270 1778653340.622291 fallocate 0x025541 thru 0x02698c (0x144c bytes) 000271 1778653340.634194 mapwrite 0x036fc1 thru 0x03c464 (0x54a4 bytes) 000273 1778653340.719054 write 0x004147 thru 0x010f6b (0xce25 bytes) 000274 1778653340.722268 mapwrite 0x03431c thru 0x03ffff (0xbce4 bytes) 000275 1778653340.819891 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x46b77a3c (stale) Mirror 5: 0x46b77a3c (stale) Mirror 6: 0x46b77a3c (stale) Mirror 8: 0x1212b349 (stale) Mirror 9: 0x351bca62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x46b77a3c (stale) Mirror 5: 0x46b77a3c (stale) Mirror 6: 0x46b77a3c (stale) Mirror 8: 0x1212b349 (stale) Mirror 9: 0x351bca62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000276 1778653341.583789 mapwrite 0x006cf5 thru 0x012907 (0xbc13 bytes) 000277 1778653341.660503 read 0x008299 thru 0x0122ba (0xa022 bytes) 000278 1778653341.661243 punch from 0x014d78 to 0x01891b (0x3ba4 bytes) 000280 1778653341.672398 MIRROR_RESYNC 6 mirrors 000281 1778653343.649793 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 000282 1778653344.251067 fallocate 0x000a48 thru 0x00dd89 (0xd342 bytes) 000283 1778653344.334407 trunc from 0x00c06c to 0x019061 (0xcff6 bytes) 000284 1778653344.365007 fallocate 0x03cba5 thru 0x03ffff (0x345b bytes) 000285 1778653344.378997 fallocate 0x01dc75 thru 0x01eadb (0xe67 bytes) 000286 1778653344.384978 punch from 0x017d6c to 0x021a15 (0x9caa bytes) 000287 1778653344.393900 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4dcf7c3d (stale) Mirror 6: 0x4dcf7c3d (stale) Mirror 8: 0x4dcf7c3d (stale) Mirror 9: 0x4dcf7c3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91789efc Mirror 10: 0x91789efc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000288 1778653345.347781 trunc from 0x040000 to 0x00d278 (0x32d87 bytes) 000289 1778653345.442143 punch from 0x002303 to 0x003906 (0x1604 bytes) 000290 1778653345.446347 write 0x03e665 thru 0x03ffff (0x199b bytes) 000291 1778653345.464238 trunc from 0x040000 to 0x004039 (0x3bfc6 bytes) 000292 1778653345.517422 mapwrite 0x036d9e thru 0x03f7f6 (0x8a59 bytes) 000293 1778653345.640183 write 0x02ea63 thru 0x03b965 (0xcf03 bytes) 000294 1778653345.679456 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f7f7): Mirror 6: 0xe20780a5 (stale) Mirror 8: 0xe20780a5 (stale) Mirror 9: 0xe20780a5 (stale) Mirror 10: 0x4b5f0e5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7f7): Mirror 6: 0xe20780a5 (stale) Mirror 8: 0xe20780a5 (stale) Mirror 9: 0xe20780a5 (stale) Mirror 10: 0x4b5f0e5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7f7: skipped 000295 1778653346.173047 trunc from 0x026865 to 0x02ca5f (0x61fb bytes) 000296 1778653346.182460 mapwrite 0x00ec9e thru 0x0126f9 (0x3a5c bytes) 000297 1778653346.227276 read 0x03c5b9 thru 0x03f7f6 (0x323e bytes) 000298 1778653346.233627 fallocate 0x0204ac thru 0x0236d0 (0x3225 bytes) 000299 1778653346.238462 punch from 0x03ac1a to 0x03f124 (0x450b bytes) 000300 1778653346.250481 mapread 0x02a8a0 thru 0x039e70 (0xf5d1 bytes) 000301 1778653346.276845 mapread 0x021265 thru 0x02ec10 (0xd9ac bytes) 000302 1778653346.292699 fallocate 0x03f48a thru 0x03ffff (0xb76 bytes) 000303 1778653346.297799 mapread 0x02194a thru 0x02dfb3 (0xc66a bytes) 000304 1778653346.298696 punch from 0x02fe47 to 0x033d1f (0x3ed9 bytes) 000305 1778653346.310079 trunc from 0x031115 to 0x03be15 (0xad01 bytes) 000306 1778653346.315670 trunc from 0x03f7f7 to 0x0187d6 (0x27020 bytes) 000307 1778653346.395029 punch from 0x01718a to 0x0187d6 (0x164d bytes) 000308 1778653346.401642 mapwrite 0x007446 thru 0x00cdec (0x59a7 bytes) 000309 1778653346.440191 punch from 0x00fe3c to 0x0187d6 (0x899b bytes) 000310 1778653346.444859 trunc from 0x0187d7 to 0x030f34 (0x1875e bytes) 000311 1778653346.459826 mapread 0x011536 thru 0x0159df (0x44aa bytes) 000312 1778653346.473948 punch from 0x005609 to 0x00e03e (0x8a36 bytes) 000314 1778653346.480133 read 0x0268b5 thru 0x02c4a3 (0x5bef bytes) 000315 1778653346.498960 trunc from 0x030f35 to 0x02641c (0xab18 bytes) 000316 1778653346.519613 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2641d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2641d): Mirror 6: 0x9ecf438a (stale) Mirror 8: 0x9ecf438a (stale) Mirror 9: 0x9ecf438a (stale) Mirror 10: 0xa34e8d37 (stale) CRC-32 checksum value for chunk [0, 0x2641d): Mirror 1: 0x7f77f4a5 Mirror 11: 0x7f77f4a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2641d: skipped 000317 1778653347.201320 trunc from 0x022883 to 0x02641c (0x3b9a bytes) 000318 1778653347.246869 write 0x02f0f7 thru 0x035f56 (0x6e60 bytes) 000319 1778653347.255617 write 0x015f37 thru 0x01b93c (0x5a06 bytes) 000320 1778653347.270245 MIRROR_RESYNC 6 mirrors 000322 1778653349.381980 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 000323 1778653349.895507 write 0x002bc6 thru 0x00c4a8 (0x98e3 bytes) 000324 1778653350.024276 MIRROR_RESYNC 5 mirrors 000325 1778653352.295475 write 0x038e4d thru 0x03c863 (0x3a17 bytes) 000326 1778653352.329235 read 0x00ec07 thru 0x014fa3 (0x639d bytes) 000327 1778653352.345635 write 0x027d3d thru 0x033b58 (0xbe1c bytes) 000328 1778653352.355850 mapread 0x01f2e8 thru 0x0247db (0x54f4 bytes) 000329 1778653352.379659 MIRROR_RESYNC 5 mirrors 000330 1778653354.627779 mapread 0x006213 thru 0x012b32 (0xc920 bytes) 000331 1778653354.658235 fallocate 0x0026a3 thru 0x00902b (0x6989 bytes) 000332 1778653354.696794 read 0x02c7f1 thru 0x033990 (0x71a0 bytes) 000333 1778653354.711489 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c864): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c864): Mirror 6: 0xb1889ccc (stale) Mirror 8: 0xb1889ccc (stale) Mirror 9: 0xb1889ccc (stale) Mirror 10: 0xb1889ccc (stale) CRC-32 checksum value for chunk [0, 0x3c864): Mirror 1: 0xb1889ccc Mirror 11: 0xb1889ccc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c864: skipped 000334 1778653356.275979 trunc from 0x03c864 to 0x03b52d (0x1336 bytes) 000335 1778653356.381939 mapwrite 0x00e6ce thru 0x011cf6 (0x3629 bytes) 000336 1778653356.455780 trunc from 0x03b52e to 0x02bdc6 (0xf767 bytes) 000337 1778653356.494286 write 0x003124 thru 0x00c1a5 (0x9082 bytes) 000338 1778653356.505748 trunc from 0x02bdc7 to 0x011a9e (0x1a328 bytes) 000339 1778653356.530288 punch from 0x00fbc1 to 0x011a9e (0x1ede bytes) 000340 1778653356.547194 read 0x00cc58 thru 0x011a9e (0x4e47 bytes) 000341 1778653356.564210 fallocate 0x009cf0 thru 0x016ef4 (0xd205 bytes) 000342 1778653356.586389 write 0x03f840 thru 0x03ffff (0x7c0 bytes) 000343 1778653356.589583 mapread 0x00fb8c thru 0x0128e2 (0x2d57 bytes) 000345 1778653356.608326 trunc from 0x02821d to 0x029ba3 (0x1987 bytes) 000346 1778653356.625715 mapread 0x0238de thru 0x029f8c (0x66af bytes) 000347 1778653356.645116 trunc from 0x040000 to 0x00f7e9 (0x30816 bytes) 000349 1778653356.685724 mapread 0x0076ad thru 0x00eff5 (0x7949 bytes) 000350 1778653356.693997 trunc from 0x00f7ea to 0x006818 (0x8fd1 bytes) 000352 1778653356.758286 trunc from 0x006819 to 0x01316e (0xc956 bytes) 000353 1778653356.794793 mapread 0x0077e6 thru 0x00c76c (0x4f87 bytes) 000354 1778653356.820847 fallocate 0x01b64e thru 0x02ac13 (0xf5c6 bytes) 000355 1778653356.830901 write 0x029a0a thru 0x02ce4f (0x3446 bytes) 000356 1778653356.871533 mapwrite 0x0138f4 thru 0x0162b6 (0x29c3 bytes) 000357 1778653356.938365 MIRROR_RESYNC 6 mirrors 000359 1778653358.866542 mapread 0x0097a2 thru 0x018854 (0xf0b3 bytes) 000360 1778653358.901079 read 0x01bf81 thru 0x029586 (0xd606 bytes) 000362 1778653358.920868 trunc from 0x02082b to 0x020e1c (0x5f2 bytes) 000363 1778653359.019376 read 0x019c28 thru 0x023090 (0x9469 bytes) 000364 1778653359.108882 mapwrite 0x019c2c thru 0x01d69b (0x3a70 bytes) 000365 1778653359.168655 MIRROR_RESYNC 6 mirrors 000367 1778653360.873660 mapwrite 0x01d660 thru 0x020fe7 (0x3988 bytes) 000368 1778653360.974436 punch from 0x001a3f to 0x00fc55 (0xe217 bytes) 000369 1778653360.980525 write 0x007864 thru 0x012ee7 (0xb684 bytes) 000371 1778653361.002516 trunc from 0x019a9b to 0x0282fb (0xe861 bytes) 000372 1778653361.008507 trunc from 0x02ce50 to 0x00ce90 (0x1ffbf bytes) 000373 1778653361.027084 trunc from 0x00ce91 to 0x02d37d (0x204ed bytes) 000374 1778653361.049042 MIRROR_RESYNC 6 mirrors 000375 1778653362.282879 trunc from 0x024c38 to 0x02d37d (0x8746 bytes) 000376 1778653362.317549 mapread 0x028f13 thru 0x02d37d (0x446b bytes) 000377 1778653362.340561 punch from 0x01f7b1 to 0x029f4c (0xa79c bytes) 000378 1778653362.345817 mapwrite 0x00b273 thru 0x00cd3f (0x1acd bytes) 000379 1778653362.365997 punch from 0x009975 to 0x00cf32 (0x35be bytes) 000380 1778653362.372946 mapwrite 0x02d52b thru 0x03810b (0xabe1 bytes) 000381 1778653362.433888 punch from 0x010926 to 0x0169cf (0x60aa bytes) 000382 1778653362.441904 fallocate 0x038c95 thru 0x03ffff (0x736b bytes) 000383 1778653362.450827 write 0x017b62 thru 0x024ae7 (0xcf86 bytes) 000384 1778653362.472123 punch from 0x010b40 to 0x01c3fb (0xb8bc bytes) 000385 1778653362.487449 fallocate 0x03550a thru 0x03e22f (0x8d26 bytes) 000386 1778653362.493120 mapwrite 0x03ba83 thru 0x03c56a (0xae8 bytes) 000387 1778653362.530248 mapread 0x027f4f thru 0x031a02 (0x9ab4 bytes) 000388 1778653362.532015 read 0x020bdf thru 0x02dfce (0xd3f0 bytes) 000389 1778653362.538064 mapread 0x00410d thru 0x00e5d9 (0xa4cd bytes) 000390 1778653362.550138 mapwrite 0x02b8cc thru 0x03b89f (0xffd4 bytes) 000391 1778653362.580923 trunc from 0x000855 to 0x00a1d8 (0x9984 bytes) 000392 1778653362.591210 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c56b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c56b): Mirror 6: 0xc4629b17 (stale) Mirror 8: 0xc4629b17 (stale) Mirror 9: 0xc4629b17 (stale) Mirror 10: 0xc4629b17 (stale) Mirror 11: 0xc4629b17 (stale) CRC-32 checksum value for chunk [0, 0x3c56b): Mirror 1: 0x60f062d5 Mirror 12: 0x60f062d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c56b: skipped 000394 1778653363.381588 fallocate 0x032af4 thru 0x03e54a (0xba57 bytes) 000395 1778653363.466456 mapwrite 0x005436 thru 0x0115f9 (0xc1c4 bytes) 000396 1778653363.571827 mapread 0x0312b2 thru 0x0365f0 (0x533f bytes) 000397 1778653363.595025 trunc from 0x03c56b to 0x00e81c (0x2dd4e bytes) 000398 1778653363.713852 trunc from 0x00e81d to 0x022a6f (0x14253 bytes) 000399 1778653363.745836 trunc from 0x005986 to 0x00e239 (0x88b4 bytes) 000400 1778653363.764072 mapwrite 0x021a28 thru 0x02ac3b (0x9214 bytes) 000401 1778653363.891421 mapread 0x023447 thru 0x02ac3b (0x77f5 bytes) 000402 1778653363.892619 fallocate 0x038d7a thru 0x03efe5 (0x626c bytes) 000403 1778653363.900821 punch from 0x007b21 to 0x00c492 (0x4972 bytes) 000404 1778653363.922877 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ac3c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ac3c): Mirror 6: 0x89297566 (stale) Mirror 8: 0x89297566 (stale) Mirror 9: 0x89297566 (stale) Mirror 10: 0x89297566 (stale) Mirror 11: 0x89297566 (stale) Mirror 12: 0xa049346b (stale) CRC-32 checksum value for chunk [0, 0x2ac3c): Mirror 1: 0xb23ea619 Mirror 13: 0xb23ea619 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ac3c: skipped 000405 1778653365.579530 read 0x0201cb thru 0x024f43 (0x4d79 bytes) 000407 1778653365.614519 MIRROR_RESYNC 8 mirrors 000408 1778653367.742917 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 000409 1778653368.144757 mapread 0x027827 thru 0x02ac3b (0x3415 bytes) 000410 1778653368.167769 fallocate 0x010bbb thru 0x01477f (0x3bc5 bytes) 000411 1778653368.222968 trunc from 0x004f73 to 0x007a48 (0x2ad6 bytes) 000412 1778653368.231632 write 0x01ba55 thru 0x01e0f6 (0x26a2 bytes) 000413 1778653368.254817 mapread 0x01fcc1 thru 0x02aa4d (0xad8d bytes) 000414 1778653368.272637 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ac3c): Mirror 6: 0xb23ea619 (stale) Mirror 9: 0xb23ea619 (stale) Mirror 10: 0xb23ea619 (stale) Mirror 12: 0xb23ea619 (stale) Mirror 13: 0xb23ea619 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ac3c): Mirror 6: 0xb23ea619 (stale) Mirror 9: 0xb23ea619 (stale) Mirror 10: 0xb23ea619 (stale) Mirror 12: 0xb23ea619 (stale) Mirror 13: 0xb23ea619 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ac3c: skipped 000417 1778653368.852005 mapread 0x01cb8e thru 0x0256ed (0x8b60 bytes) 000418 1778653368.891046 read 0x013781 thru 0x01fdcd (0xc64d bytes) 000419 1778653368.905034 trunc from 0x02ac3c to 0x027147 (0x3af4 bytes) 000420 1778653368.929934 read 0x021e9e thru 0x027147 (0x52aa bytes) 000421 1778653368.979569 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27148): Mirror 6: 0x1353ecd9 (stale) Mirror 10: 0x1353ecd9 (stale) Mirror 12: 0x1353ecd9 (stale) Mirror 13: 0x1353ecd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27148): Mirror 6: 0x1353ecd9 (stale) Mirror 10: 0x1353ecd9 (stale) Mirror 12: 0x1353ecd9 (stale) Mirror 13: 0x1353ecd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27148: skipped 000422 1778653369.574871 fallocate 0x031b9c thru 0x03afdc (0x9441 bytes) 000424 1778653369.586106 mapwrite 0x03d141 thru 0x03fd81 (0x2c41 bytes) 000425 1778653369.682773 write 0x0373c1 thru 0x03ffff (0x8c3f bytes) 000426 1778653369.696835 trunc from 0x00f9b1 to 0x011154 (0x17a4 bytes) 000427 1778653369.704718 write 0x01f9f2 thru 0x0293bc (0x99cb bytes) 000428 1778653369.724513 mapwrite 0x03ea79 thru 0x03ffff (0x1587 bytes) 000429 1778653369.752873 mapread 0x001462 thru 0x00b2ca (0x9e69 bytes) 000430 1778653369.770060 read 0x0105f7 thru 0x01559a (0x4fa4 bytes) 000431 1778653369.805381 write 0x00a3c2 thru 0x019483 (0xf0c2 bytes) 000432 1778653369.807443 mapread 0x00dba9 thru 0x01cff7 (0xf44f bytes) 000433 1778653369.821814 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x26066706 (stale) Mirror 10: 0x26066706 (stale) Mirror 12: 0x26066706 (stale) Mirror 13: 0x26066706 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfe960e6 Mirror 14: 0xdfe960e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1778653370.754942 read 0x0344d4 thru 0x0359e8 (0x1515 bytes) 000435 1778653370.774667 read 0x016a17 thru 0x01bb43 (0x512d bytes) 000436 1778653370.795818 punch from 0x021335 to 0x02d787 (0xc453 bytes) 000437 1778653370.841446 mapread 0x02d263 thru 0x039064 (0xbe02 bytes) 000438 1778653370.984889 read 0x03bcba thru 0x03f548 (0x388f bytes) 000439 1778653371.003407 trunc from 0x01248d to 0x013dc9 (0x193d bytes) 000440 1778653371.025877 write 0x02e35f thru 0x03bdb6 (0xda58 bytes) 000441 1778653371.046798 mapwrite 0x035d2b thru 0x036618 (0x8ee bytes) 000442 1778653371.103447 trunc from 0x01c236 to 0x01fb9b (0x3966 bytes) 000444 1778653371.135317 trunc from 0x040000 to 0x02b19d (0x14e62 bytes) 000445 1778653371.208441 mapwrite 0x02376c thru 0x023cee (0x583 bytes) 000446 1778653371.230078 write 0x02cfe2 thru 0x03b726 (0xe745 bytes) 000447 1778653371.232249 write 0x009c2b thru 0x00c4cc (0x28a2 bytes) 000448 1778653371.233773 trunc from 0x03b727 to 0x03956d (0x21b9 bytes) 000449 1778653371.258220 trunc from 0x01416d to 0x01c4b8 (0x834c bytes) 000451 1778653371.266006 fallocate 0x025d38 thru 0x02e2ab (0x8574 bytes) 000452 1778653371.270055 mapwrite 0x0128ab thru 0x021456 (0xebac bytes) 000454 1778653371.499560 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3956e): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3956e): Mirror 6: 0x1aff315 (stale) Mirror 10: 0x1aff315 (stale) Mirror 12: 0x1aff315 (stale) Mirror 13: 0x1aff315 (stale) Mirror 14: 0x53b160f1 (stale) CRC-32 checksum value for chunk [0, 0x3956e): Mirror 1: 0x555568a1 Mirror 15: 0x555568a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3956e: skipped 000455 1778653372.660714 punch from 0x01e23c to 0x02a92e (0xc6f3 bytes) 000456 1778653372.741744 punch from 0x038380 to 0x03956d (0x11ee bytes) 000457 1778653372.755419 write 0x010336 thru 0x0129ff (0x26ca bytes) 000458 1778653372.801895 fallocate 0x00e3cb thru 0x010463 (0x2099 bytes) 000459 1778653372.817977 trunc from 0x01e967 to 0x029501 (0xab9b bytes) 000461 1778653372.856929 write 0x002925 thru 0x00b69e (0x8d7a bytes) 000462 1778653372.886708 write 0x004882 thru 0x00afc2 (0x6741 bytes) 000463 1778653372.888995 mapread 0x01ff89 thru 0x02fe8a (0xff02 bytes) 000464 1778653372.951101 MIRROR_RESYNC 7 mirrors 000466 1778653375.069384 read 0x011a11 thru 0x0129ed (0xfdd bytes) 000467 1778653375.075001 trunc from 0x03956e to 0x02b7ed (0xdd80 bytes) 000468 1778653375.134858 write 0x038948 thru 0x03ffff (0x76b8 bytes) 000469 1778653375.136818 write 0x034f86 thru 0x03ffff (0xb07a bytes) 000471 1778653375.143336 trunc from 0x018442 to 0x01c094 (0x3c53 bytes) 000472 1778653375.149745 read 0x0292d7 thru 0x02ac31 (0x195b bytes) 000473 1778653375.195372 punch from 0x03b6cb to 0x03ffff (0x4935 bytes) 000474 1778653375.251756 trunc from 0x040000 to 0x039800 (0x67ff bytes) 000475 1778653375.277345 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39801): Mirror 10: 0x3bf51466 (stale) Mirror 12: 0x3bf51466 (stale) Mirror 13: 0x3bf51466 (stale) Mirror 14: 0x3bf51466 (stale) Mirror 15: 0x3bf51466 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39801): Mirror 10: 0x3bf51466 (stale) Mirror 12: 0x3bf51466 (stale) Mirror 13: 0x3bf51466 (stale) Mirror 14: 0x3bf51466 (stale) Mirror 15: 0x3bf51466 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39801: skipped 000476 1778653376.068645 write 0x01adbe thru 0x0205c5 (0x5808 bytes) 000477 1778653376.086313 mapwrite 0x010b55 thru 0x016b11 (0x5fbd bytes) 000478 1778653376.136218 trunc from 0x02a584 to 0x0349a1 (0xa41e bytes) 000479 1778653376.141751 mapread 0x01e905 thru 0x020611 (0x1d0d bytes) 000480 1778653376.146700 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39801): Mirror 10: 0x3bf51466 (stale) Mirror 12: 0x3bf51466 (stale) Mirror 13: 0x3bf51466 (stale) Mirror 15: 0x3bf51466 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39801): Mirror 10: 0x3bf51466 (stale) Mirror 12: 0x3bf51466 (stale) Mirror 13: 0x3bf51466 (stale) Mirror 15: 0x3bf51466 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39801: skipped 000481 1778653376.976437 write 0x03bf1a thru 0x03ffff (0x40e6 bytes) 000482 1778653377.007745 MIRROR_RESYNC 5 mirrors 000483 1778653380.125126 fallocate 0x0044db thru 0x00b59b (0x70c1 bytes) 000484 1778653380.205379 trunc from 0x040000 to 0x00debf (0x32140 bytes) 000485 1778653380.263713 trunc from 0x00b7ed to 0x00d878 (0x208c bytes) 000486 1778653380.271767 fallocate 0x01d8ed thru 0x028867 (0xaf7b bytes) 000487 1778653380.290665 trunc from 0x028868 to 0x00ad96 (0x1dad1 bytes) 000488 1778653380.317436 fallocate 0x00dec6 thru 0x014665 (0x67a0 bytes) 000489 1778653380.332705 trunc from 0x00ad97 to 0x00c3e0 (0x164a bytes) 000490 1778653380.401756 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc3e1): Mirror 10: 0xb58a1baa (stale) Mirror 12: 0xb58a1baa (stale) Mirror 15: 0xb58a1baa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3e1): Mirror 10: 0xb58a1baa (stale) Mirror 12: 0xb58a1baa (stale) Mirror 15: 0xb58a1baa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3e1: skipped 000491 1778653381.078282 mapread 0x009adf thru 0x00c3e0 (0x2902 bytes) 000492 1778653381.120781 fallocate 0x03b65e thru 0x03ffff (0x49a2 bytes) 000493 1778653381.134849 trunc from 0x040000 to 0x025b33 (0x1a4cc bytes) 000494 1778653381.194197 read 0x01a223 thru 0x025b33 (0xb911 bytes) 000495 1778653381.209281 trunc from 0x01e6eb to 0x025b33 (0x7449 bytes) 000496 1778653381.236779 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25b34): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b34): Mirror 10: 0x177a97b8 (stale) Mirror 12: 0x177a97b8 (stale) Mirror 15: 0x177a97b8 (stale) CRC-32 checksum value for chunk [0, 0x25b34): Mirror 1: 0x1f40c450 Mirror 16: 0x1f40c450 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b34: skipped 000497 1778653382.628401 read 0x007d17 thru 0x00b96d (0x3c57 bytes) 000498 1778653382.641514 punch from 0x01f9ba to 0x025b33 (0x617a bytes) 000499 1778653382.690032 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25b34): Mirror 10: 0x177a97b8 (stale) Mirror 15: 0x177a97b8 (stale) Mirror 16: 0x1f40c450 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b34): Mirror 10: 0x177a97b8 (stale) Mirror 15: 0x177a97b8 (stale) Mirror 16: 0x1f40c450 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b34: skipped 000500 1778653383.075630 trunc from 0x025b34 to 0x005ecb (0x1fc68 bytes) 000501 1778653383.183303 punch from 0x0033f7 to 0x004f80 (0x1b8a bytes) 000502 1778653383.216200 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ecc): Mirror 15: 0x7f977671 (stale) Mirror 16: 0x7f977671 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ecc): Mirror 15: 0x7f977671 (stale) Mirror 16: 0x7f977671 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ecc: skipped 000503 1778653383.582204 mapwrite 0x037ead thru 0x03ffff (0x8153 bytes) 000505 1778653383.700809 write 0x0102ea thru 0x0175d9 (0x72f0 bytes) 000506 1778653383.723931 mapread 0x020cec thru 0x02d054 (0xc369 bytes) 000507 1778653383.746977 mapread 0x01536d thru 0x024fa9 (0xfc3d bytes) 000508 1778653383.749658 trunc from 0x016d41 to 0x020e7e (0xa13e bytes) 000509 1778653383.763801 mapwrite 0x00bf6c thru 0x018ddd (0xce72 bytes) 000511 1778653383.800587 write 0x03a822 thru 0x03ffff (0x57de bytes) 000512 1778653383.801782 trunc from 0x036771 to 0x03fde4 (0x9674 bytes) 000513 1778653383.820389 mapread 0x00f1fc thru 0x017a10 (0x8815 bytes) 000514 1778653383.823069 write 0x0398b6 thru 0x03ffff (0x674a bytes) 000515 1778653383.836248 trunc from 0x040000 to 0x03dfdb (0x2024 bytes) 000517 1778653383.848815 trunc from 0x03dfdc to 0x038935 (0x56a6 bytes) 000518 1778653383.859346 trunc from 0x03743d to 0x038935 (0x14f9 bytes) 000520 1778653383.863226 mapread 0x0271c4 thru 0x030a93 (0x98d0 bytes) 000521 1778653383.864108 write 0x01fd78 thru 0x021070 (0x12f9 bytes) 000522 1778653383.867854 trunc from 0x036cd8 to 0x038935 (0x1c5e bytes) 000523 1778653383.871641 mapread 0x02ed68 thru 0x038935 (0x9bce bytes) 000525 1778653383.879929 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38936): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38936): Mirror 15: 0x19aa09fb (stale) Mirror 16: 0x11edcd29 (stale) CRC-32 checksum value for chunk [0, 0x38936): Mirror 1: 0xa23167de Mirror 17: 0xa23167de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38936: skipped 000526 1778653385.874226 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38936): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38936): Mirror 15: 0x19aa09fb (stale) Mirror 16: 0x11edcd29 (stale) CRC-32 checksum value for chunk [0, 0x38936): Mirror 1: 0xa23167de Mirror 17: 0xa23167de Mirror 18: 0xa23167de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38936: skipped 000527 1778653387.717001 fallocate 0x03ff09 thru 0x03ffff (0xf7 bytes) 000528 1778653387.874288 trunc from 0x01663f to 0x01b28a (0x4c4c bytes) 000529 1778653387.889130 read 0x010f6c thru 0x013cc3 (0x2d58 bytes) 000530 1778653387.908800 punch from 0x033def to 0x03fb9a (0xbdac bytes) 000531 1778653387.929243 mapread 0x0265d1 thru 0x029c0e (0x363e bytes) 000532 1778653387.982295 mapwrite 0x00fc26 thru 0x017ec6 (0x82a1 bytes) 000533 1778653388.152685 write 0x02c0cf thru 0x031eaa (0x5ddc bytes) 000534 1778653388.187566 mapwrite 0x0061c1 thru 0x0105e9 (0xa429 bytes) 000535 1778653388.290094 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x7a6d0c19 (stale) Mirror 17: 0x853821e9 (stale) Mirror 18: 0x853821e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x7a6d0c19 (stale) Mirror 17: 0x853821e9 (stale) Mirror 18: 0x853821e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1778653389.296385 fallocate 0x0104fe thru 0x01da7b (0xd57e bytes) 000537 1778653389.316620 fallocate 0x025107 thru 0x032988 (0xd882 bytes) 000538 1778653389.347634 trunc from 0x040000 to 0x028d08 (0x172f7 bytes) 000539 1778653389.429639 mapwrite 0x01dba3 thru 0x02122d (0x368b bytes) 000540 1778653389.621804 trunc from 0x028d09 to 0x021d15 (0x6ff3 bytes) 000542 1778653389.699119 read 0x01cfbf thru 0x0204e5 (0x3527 bytes) 000543 1778653389.709593 fallocate 0x03353e thru 0x03e2a5 (0xad68 bytes) 000544 1778653389.723457 mapread 0x0153c3 thru 0x0194cf (0x410d bytes) 000545 1778653389.749205 write 0x0084a8 thru 0x010cd5 (0x882e bytes) 000546 1778653389.774993 punch from 0x0159b8 to 0x024aae (0xf0f7 bytes) 000547 1778653389.797962 mapread 0x021c51 thru 0x030b3b (0xeeeb bytes) 000548 1778653389.842682 mapwrite 0x037184 thru 0x03ffff (0x8e7c bytes) 000549 1778653389.978073 mapwrite 0x028091 thru 0x036e82 (0xedf2 bytes) 000550 1778653390.031436 mapread 0x022ab3 thru 0x02684e (0x3d9c bytes) 000551 1778653390.032590 fallocate 0x00d0c2 thru 0x019a09 (0xc948 bytes) 000552 1778653390.066619 write 0x01db62 thru 0x01f2be (0x175d bytes) 000553 1778653390.069503 fallocate 0x0191f8 thru 0x0261ea (0xcff3 bytes) 000554 1778653390.093599 punch from 0x02740f to 0x034cb3 (0xd8a5 bytes) 000555 1778653390.133078 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x853821e9 (stale) Mirror 18: 0x853821e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x853821e9 (stale) Mirror 18: 0x853821e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000556 1778653390.950112 trunc from 0x0366eb to 0x03ffff (0x9915 bytes) 000557 1778653390.957082 read 0x008a7f thru 0x016fda (0xe55c bytes) 000558 1778653390.978051 write 0x029985 thru 0x030af5 (0x7171 bytes) 000559 1778653391.026014 mapread 0x0052d3 thru 0x0092b5 (0x3fe3 bytes) 000560 1778653391.044359 write 0x03c636 thru 0x03ffff (0x39ca bytes) 000561 1778653391.074347 mapread 0x03daa2 thru 0x03fec6 (0x2425 bytes) 000562 1778653391.076988 trunc from 0x040000 to 0x03ebf5 (0x140a bytes) 000563 1778653391.107890 trunc from 0x03ebf6 to 0x01bd7e (0x22e77 bytes) 000564 1778653391.146581 read 0x00cdd0 thru 0x00e3f1 (0x1622 bytes) 000565 1778653391.148605 trunc from 0x0071ad to 0x00b477 (0x42cb bytes) 000566 1778653391.189534 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bd7f): Mirror 18: 0xfebecb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd7f): Mirror 18: 0xfebecb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd7f: skipped 000567 1778653391.860506 read 0x018666 thru 0x019f0b (0x18a6 bytes) 000568 1778653391.873559 fallocate 0x01df10 thru 0x02829c (0xa38d bytes) 000569 1778653391.890620 trunc from 0x02829d to 0x0140f9 (0x141a3 bytes) 000570 1778653391.941337 write 0x0182d5 thru 0x018ffe (0xd2a bytes) 000571 1778653391.943099 punch from 0x00455e to 0x00fe74 (0xb917 bytes) 000572 1778653391.969311 trunc from 0x018fff to 0x02a3e1 (0x113e3 bytes) 000573 1778653392.012545 mapwrite 0x035ba3 thru 0x036154 (0x5b2 bytes) 000574 1778653392.142103 fallocate 0x011d3a thru 0x01ce83 (0xb14a bytes) 000576 1778653392.161535 trunc from 0x036155 to 0x03d9f7 (0x78a3 bytes) 000577 1778653392.189226 write 0x029cb5 thru 0x035603 (0xb94f bytes) 000578 1778653392.222928 trunc from 0x00abdf to 0x01226e (0x7690 bytes) 000579 1778653392.238641 mapread 0x01ea18 thru 0x020f0f (0x24f8 bytes) 000581 1778653392.266907 mapread 0x03d5be thru 0x03d9f7 (0x43a bytes) 000582 1778653392.277297 read 0x032030 thru 0x039eec (0x7ebd bytes) 000583 1778653392.278612 read 0x000e35 thru 0x00e9d7 (0xdba3 bytes) 000584 1778653392.289611 read 0x034b25 thru 0x03d9f7 (0x8ed3 bytes) 000585 1778653392.290349 mapread 0x027f52 thru 0x02ff62 (0x8011 bytes) 000586 1778653392.296599 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 18: 0x445913f4 (stale) CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 1: 0x805f6440 Mirror 19: 0x805f6440 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9f8: skipped 000587 1778653394.180519 punch from 0x003852 to 0x00877d (0x4f2c bytes) 000588 1778653394.253218 fallocate 0x02e080 thru 0x02e659 (0x5da bytes) 000589 1778653394.274522 write 0x008804 thru 0x00f262 (0x6a5f bytes) 000590 1778653394.295398 mapread 0x035126 thru 0x03d9f7 (0x88d2 bytes) 000591 1778653394.312492 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 18: 0x445913f4 (stale) Mirror 19: 0x805f6440 (stale) CRC-32 checksum value for chunk [0, 0x3d9f8): Mirror 1: 0xf80479d4 Mirror 20: 0xf80479d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9f8: skipped 000592 1778653396.266943 write 0x031023 thru 0x038c1c (0x7bfa bytes) 000593 1778653396.330571 trunc from 0x03a904 to 0x03d099 (0x2796 bytes) 000594 1778653396.335828 trunc from 0x03d9f8 to 0x0014bf (0x3c538 bytes) 000595 1778653396.357642 mapwrite 0x032575 thru 0x03c283 (0x9d0f bytes) 000596 1778653396.417505 trunc from 0x03c284 to 0x01b490 (0x20df3 bytes) 000598 1778653396.438265 fallocate 0x014cfa thru 0x01f302 (0xa609 bytes) 000599 1778653396.444478 punch from 0x01182a to 0x019270 (0x7a47 bytes) 000600 1778653396.450383 trunc from 0x01f303 to 0x01fd05 (0xa03 bytes) 000601 1778653396.473300 punch from 0x013278 to 0x01fd05 (0xca8e bytes) 000602 1778653396.482205 mapwrite 0x00d7eb thru 0x00f8c8 (0x20de bytes) 000603 1778653396.505363 trunc from 0x01fd06 to 0x00c8bc (0x13449 bytes) 000604 1778653396.519990 punch from 0x001a75 to 0x00c8bc (0xae48 bytes) 000605 1778653396.526290 write 0x005e9a thru 0x0116d1 (0xb838 bytes) 000606 1778653396.536929 write 0x01b90c thru 0x024c9f (0x9394 bytes) 000607 1778653396.539115 punch from 0x001096 to 0x003a2e (0x2999 bytes) 000608 1778653396.544879 read 0x01c47c thru 0x01fd21 (0x38a6 bytes) 000609 1778653396.545766 mapwrite 0x021382 thru 0x02e9de (0xd65d bytes) 000610 1778653396.604333 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e9df): Mirror 19: 0x9bb778c2 (stale) Mirror 20: 0xbd80af25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9df): Mirror 19: 0x9bb778c2 (stale) Mirror 20: 0xbd80af25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9df: skipped 000611 1778653397.042714 MIRROR_RESYNC 3 mirrors 000612 1778653398.939398 punch from 0x010b64 to 0x01b4fd (0xa99a bytes) 000613 1778653399.073474 mapwrite 0x0093a7 thru 0x012822 (0x947c bytes) 000614 1778653399.127902 fallocate 0x00f384 thru 0x00f518 (0x195 bytes) 000615 1778653399.134866 read 0x029c04 thru 0x02c6b3 (0x2ab0 bytes) 000616 1778653399.149341 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e9df): Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9df): Mirror 19: 0x9ac63773 (stale) Mirror 20: 0x9ac63773 (stale) CRC-32 checksum value for chunk [0, 0x2e9df): Mirror 1: 0x7f16f5d9 Mirror 21: 0x7f16f5d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9df: skipped 000617 1778653400.482660 read 0x0064bc thru 0x00cf26 (0x6a6b bytes) 000618 1778653400.534026 read 0x01aa3a thru 0x01dd5b (0x3322 bytes) 000619 1778653400.621403 MIRROR_RESYNC 4 mirrors 000620 1778653402.381473 mapread 0x0299f1 thru 0x02e9de (0x4fee bytes) 000621 1778653402.430808 write 0x03cbc3 thru 0x03d9ea (0xe28 bytes) 000622 1778653402.512240 mapread 0x03863f thru 0x03d9ea (0x53ac bytes) 000623 1778653402.528890 trunc from 0x03d9eb to 0x03d539 (0x4b1 bytes) 000624 1778653402.582920 read 0x0240e7 thru 0x027654 (0x356e bytes) 000625 1778653402.597795 fallocate 0x0096d9 thru 0x010d4b (0x7673 bytes) 000627 1778653402.606870 mapread 0x03c83d thru 0x03d539 (0xcfd bytes) 000628 1778653402.608899 punch from 0x01c40b to 0x02292f (0x6525 bytes) 000629 1778653402.615771 trunc from 0x03d53a to 0x00ec86 (0x2e8b3 bytes) 000630 1778653402.633824 trunc from 0x00ec87 to 0x031c00 (0x22f7a bytes) 000631 1778653402.656971 read 0x02a3b8 thru 0x02e251 (0x3e9a bytes) 000633 1778653402.685257 write 0x020650 thru 0x02b9ca (0xb37b bytes) 000634 1778653402.692044 fallocate 0x012c28 thru 0x01b14b (0x8524 bytes) 000635 1778653402.722678 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31c01): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c01): Mirror 19: 0x3018b0ec (stale) Mirror 20: 0x3018b0ec (stale) Mirror 21: 0x3018b0ec (stale) CRC-32 checksum value for chunk [0, 0x31c01): Mirror 1: 0x8697e367 Mirror 22: 0x8697e367 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c01: skipped 000636 1778653404.820998 read 0x00e57a thru 0x016711 (0x8198 bytes) 000637 1778653404.866681 write 0x01b98a thru 0x02513a (0x97b1 bytes) 000638 1778653404.955052 mapwrite 0x02b545 thru 0x033912 (0x83ce bytes) 000639 1778653405.051111 mapread 0x032be7 thru 0x033912 (0xd2c bytes) 000640 1778653405.052597 read 0x001cbd thru 0x0064e3 (0x4827 bytes) 000641 1778653405.093274 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33913): Mirror 19: 0x12677be7 (stale) Mirror 20: 0x12677be7 (stale) Mirror 22: 0x6f51c9ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33913): Mirror 19: 0x12677be7 (stale) Mirror 20: 0x12677be7 (stale) Mirror 22: 0x6f51c9ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33913: skipped 000642 1778653405.791800 trunc from 0x026984 to 0x02a7bf (0x3e3c bytes) 000643 1778653405.804335 write 0x004ece thru 0x0116f9 (0xc82c bytes) 000644 1778653405.846539 read 0x022ed7 thru 0x026092 (0x31bc bytes) 000646 1778653405.872937 read 0x025496 thru 0x02dfc6 (0x8b31 bytes) 000647 1778653405.884917 read 0x02efc2 thru 0x032061 (0x30a0 bytes) 000648 1778653405.886786 write 0x014fa0 thru 0x0156e8 (0x749 bytes) 000649 1778653405.887894 trunc from 0x02ea53 to 0x03095b (0x1f09 bytes) 000650 1778653405.900773 punch from 0x013579 to 0x0147a8 (0x1230 bytes) 000651 1778653405.937465 trunc from 0x01081b to 0x01fa79 (0xf25f bytes) 000652 1778653405.953503 mapread 0x007bcb thru 0x017237 (0xf66d bytes) 000653 1778653405.975231 mapread 0x01ca19 thru 0x01e2b8 (0x18a0 bytes) 000654 1778653405.976300 punch from 0x015af9 to 0x018589 (0x2a91 bytes) 000655 1778653405.986354 mapread 0x031406 thru 0x033912 (0x250d bytes) 000656 1778653405.987900 mapread 0x019c01 thru 0x01d3fc (0x37fc bytes) 000657 1778653405.989626 trunc from 0x01ac93 to 0x02a2ef (0xf65d bytes) 000659 1778653405.998221 trunc from 0x033913 to 0x02bfea (0x7928 bytes) 000660 1778653406.013124 trunc from 0x009e6c to 0x016dea (0xcf7f bytes) 000661 1778653406.021052 punch from 0x023d00 to 0x027fe6 (0x42e7 bytes) 000662 1778653406.029098 mapwrite 0x01d2e4 thru 0x028eed (0xbc0a bytes) 000663 1778653406.150388 fallocate 0x036381 thru 0x03c9b2 (0x6632 bytes) 000664 1778653406.160645 read 0x00cee7 thru 0x01871c (0xb836 bytes) 000665 1778653406.200329 punch from 0x0046fa to 0x012c6c (0xe573 bytes) 000666 1778653406.251368 write 0x02298a thru 0x02de6a (0xb4e1 bytes) 000667 1778653406.271803 trunc from 0x03970a to 0x03c9b2 (0x32a9 bytes) 000670 1778653406.298239 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9b3): Mirror 20: 0x858927bd (stale) Mirror 22: 0x35f83fad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9b3): Mirror 20: 0x858927bd (stale) Mirror 22: 0x35f83fad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9b3: skipped 000671 1778653407.096281 fallocate 0x01f9d2 thru 0x02c6ba (0xcce9 bytes) 000672 1778653407.106139 mapwrite 0x01f0fb thru 0x01feed (0xdf3 bytes) 000673 1778653407.177708 mapread 0x018bc0 thru 0x01ef3e (0x637f bytes) 000674 1778653407.233359 mapread 0x013191 thru 0x021e3b (0xecab bytes) 000675 1778653407.240876 write 0x020280 thru 0x02479b (0x451c bytes) 000676 1778653407.285124 read 0x0114a2 thru 0x01738a (0x5ee9 bytes) 000678 1778653407.317285 write 0x034be1 thru 0x03ffff (0xb41f bytes) 000679 1778653407.359538 fallocate 0x0389f2 thru 0x03ffff (0x760e bytes) 000680 1778653407.400138 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x5dacc69c (stale) Mirror 22: 0x21212048 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe9e68283 Mirror 23: 0xe9e68283 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1778653409.288607 fallocate 0x025b86 thru 0x035b6e (0xffe9 bytes) 000682 1778653409.402303 fallocate 0x01828e thru 0x01a236 (0x1fa9 bytes) 000683 1778653409.416933 read 0x01dca4 thru 0x02cd6b (0xf0c8 bytes) 000684 1778653409.464933 trunc from 0x040000 to 0x028eed (0x17112 bytes) 000685 1778653409.542762 trunc from 0x028eee to 0x01f046 (0x9ea7 bytes) 000686 1778653409.571819 fallocate 0x00f397 thru 0x01a3df (0xb049 bytes) 000687 1778653409.580283 trunc from 0x01f047 to 0x024cdf (0x5c99 bytes) 000688 1778653409.606808 trunc from 0x024ce0 to 0x00580b (0x1f4d4 bytes) 000689 1778653409.630812 trunc from 0x005515 to 0x00580b (0x2f7 bytes) 000690 1778653409.638932 mapwrite 0x011ced thru 0x0219f0 (0xfd04 bytes) 000691 1778653409.857423 punch from 0x018f51 to 0x0219f0 (0x8aa0 bytes) 000692 1778653409.870235 trunc from 0x012c62 to 0x01ca79 (0x9e18 bytes) 000693 1778653409.878827 trunc from 0x0219f1 to 0x017e8d (0x9b63 bytes) 000694 1778653409.919812 read 0x00b8eb thru 0x00c64e (0xd64 bytes) 000695 1778653409.947084 read 0x016853 thru 0x017e8d (0x163b bytes) 000696 1778653409.947865 read 0x0146c4 thru 0x017e8d (0x37ca bytes) 000697 1778653409.948539 trunc from 0x00d882 to 0x00d894 (0x13 bytes) 000698 1778653409.961219 mapread 0x008c01 thru 0x0175d5 (0xe9d5 bytes) 000699 1778653409.986294 punch from 0x0035b6 to 0x010b15 (0xd560 bytes) 000700 1778653410.014180 mapwrite 0x02bb34 thru 0x033cc4 (0x8191 bytes) 000701 1778653410.114893 read 0x0244be thru 0x025181 (0xcc4 bytes) 000702 1778653410.117829 read 0x026398 thru 0x02fbd3 (0x983c bytes) 000703 1778653410.119697 mapwrite 0x0322e1 thru 0x035c88 (0x39a8 bytes) 000704 1778653410.166248 trunc from 0x001591 to 0x001fbc (0xa2c bytes) 000705 1778653410.173468 punch from 0x03185d to 0x035c88 (0x442c bytes) 000706 1778653410.180069 write 0x024e8b thru 0x029661 (0x47d7 bytes) 000707 1778653410.181766 read 0x0129a4 thru 0x020db9 (0xe416 bytes) 000708 1778653410.183990 mapwrite 0x039982 thru 0x03ffff (0x667e bytes) 000709 1778653410.241787 mapwrite 0x037af3 thru 0x03ba8f (0x3f9d bytes) 000710 1778653410.258734 trunc from 0x006494 to 0x008df9 (0x2966 bytes) 000711 1778653410.263963 punch from 0x03013a to 0x031401 (0x12c8 bytes) 000712 1778653410.270259 mapwrite 0x016785 thru 0x020252 (0x9ace bytes) 000713 1778653410.299366 mapwrite 0x01efe1 thru 0x020dd4 (0x1df4 bytes) 000714 1778653410.314080 write 0x039acd thru 0x03ffff (0x6533 bytes) 000715 1778653410.316235 mapread 0x03fdb0 thru 0x03ffff (0x250 bytes) 000716 1778653410.318106 punch from 0x000848 to 0x00c9f5 (0xc1ae bytes) 000717 1778653410.324705 trunc from 0x003ad2 to 0x008b46 (0x5075 bytes) 000718 1778653410.330029 fallocate 0x01970c thru 0x022d96 (0x968b bytes) 000719 1778653410.340514 write 0x02dae5 thru 0x02f336 (0x1852 bytes) 000721 1778653410.346301 mapread 0x019f94 thru 0x02008a (0x60f7 bytes) 000722 1778653410.352816 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x21212048 (stale) Mirror 23: 0xe9e68283 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x21212048 (stale) Mirror 23: 0xe9e68283 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000723 1778653410.861583 mapwrite 0x03947a thru 0x03ffff (0x6b86 bytes) 000724 1778653410.937914 fallocate 0x018927 thru 0x01fe47 (0x7521 bytes) 000725 1778653410.944813 write 0x036626 thru 0x03ffff (0x99da bytes) 000726 1778653410.959075 trunc from 0x040000 to 0x03b8db (0x4724 bytes) 000728 1778653410.999300 mapread 0x007d5e thru 0x012739 (0xa9dc bytes) 000729 1778653411.026000 read 0x0089d9 thru 0x011363 (0x898b bytes) 000730 1778653411.027075 write 0x025c91 thru 0x029ecb (0x423b bytes) 000731 1778653411.042919 fallocate 0x016e52 thru 0x01bf80 (0x512f bytes) 000732 1778653411.051982 write 0x00c41a thru 0x016575 (0xa15c bytes) 000733 1778653411.067631 MIRROR_RESYNC 3 mirrors 000734 1778653413.023294 write 0x03632b thru 0x03ffff (0x9cd5 bytes) 000736 1778653413.077130 trunc from 0x007645 to 0x01717c (0xfb38 bytes) 000737 1778653413.085851 mapread 0x012678 thru 0x0211f1 (0xeb7a bytes) 000740 1778653413.132992 mapread 0x000170 thru 0x00a722 (0xa5b3 bytes) 000741 1778653413.161255 write 0x03a549 thru 0x03ffff (0x5ab7 bytes) 000743 1778653413.172293 fallocate 0x030c8c thru 0x036f6a (0x62df bytes) 000744 1778653413.185544 mapread 0x02b37b thru 0x03a5a6 (0xf22c bytes) 000745 1778653413.215092 trunc from 0x000895 to 0x007f13 (0x767f bytes) 000746 1778653413.223092 read 0x0306ad thru 0x0331dd (0x2b31 bytes) 000747 1778653413.227098 punch from 0x03e991 to 0x03ffff (0x166f bytes) 000748 1778653413.270060 fallocate 0x005853 thru 0x0115af (0xbd5d bytes) 000749 1778653413.282027 write 0x01737e thru 0x02422d (0xceb0 bytes) 000750 1778653413.289260 fallocate 0x029500 thru 0x035e58 (0xc959 bytes) 000752 1778653413.310017 punch from 0x004c23 to 0x0090ee (0x44cc bytes) 000753 1778653413.332068 mapread 0x00c36d thru 0x0173b9 (0xb04d bytes) 000754 1778653413.350002 write 0x01f41f thru 0x026bb1 (0x7793 bytes) 000755 1778653413.364105 trunc from 0x040000 to 0x0391de (0x6e21 bytes) 000756 1778653413.454992 trunc from 0x0391df to 0x032e6d (0x6371 bytes) 000757 1778653413.499318 fallocate 0x036d18 thru 0x03fd3d (0x9026 bytes) 000758 1778653413.517748 mapread 0x00a466 thru 0x01a41c (0xffb7 bytes) 000759 1778653413.527062 mapread 0x019a9c thru 0x01a3a4 (0x909 bytes) 000760 1778653413.529111 punch from 0x037664 to 0x03fd3d (0x86da bytes) 000761 1778653413.545476 mapread 0x03d103 thru 0x03d58b (0x489 bytes) 000762 1778653413.569288 trunc from 0x02bbe9 to 0x036596 (0xa9ae bytes) 000763 1778653413.589342 mapread 0x006175 thru 0x01445e (0xe2ea bytes) 000764 1778653413.629024 punch from 0x011abb to 0x01c536 (0xaa7c bytes) 000765 1778653413.664940 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fd3e): Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd3e): Mirror 22: 0x8651a0bb (stale) Mirror 23: 0x8651a0bb (stale) CRC-32 checksum value for chunk [0, 0x3fd3e): Mirror 1: 0x1fa461f4 Mirror 24: 0x1fa461f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd3e: skipped 000766 1778653415.001867 write 0x0009b4 thru 0x00b55b (0xaba8 bytes) 000767 1778653415.059917 punch from 0x0180c2 to 0x0219a7 (0x98e6 bytes) 000768 1778653415.066733 trunc from 0x03fd3e to 0x00e8e4 (0x31459 bytes) 000769 1778653415.109473 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe8e5): Mirror 22: 0x6439589d (stale) Mirror 24: 0xf06b22e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe8e5): Mirror 22: 0x6439589d (stale) Mirror 24: 0xf06b22e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe8e5: skipped 000770 1778653415.587131 fallocate 0x01dd25 thru 0x01de86 (0x162 bytes) 000771 1778653415.597881 read 0x000f18 thru 0x00976e (0x8857 bytes) 000772 1778653415.625587 fallocate 0x028956 thru 0x02a26b (0x1916 bytes) 000773 1778653415.638757 punch from 0x0023e5 to 0x008564 (0x6180 bytes) 000774 1778653415.652955 MIRROR_RESYNC 3 mirrors 000775 1778653416.820091 read 0x015dc6 thru 0x016589 (0x7c4 bytes) 000776 1778653416.828072 read 0x000920 thru 0x00096f (0x50 bytes) 000779 1778653416.844123 mapwrite 0x01fd1f thru 0x021f0b (0x21ed bytes) 000780 1778653416.939056 read 0x0111bf thru 0x01f8b0 (0xe6f2 bytes) 000781 1778653416.955323 trunc from 0x008396 to 0x00e687 (0x62f2 bytes) 000782 1778653416.974883 MIRROR_RESYNC 3 mirrors 000783 1778653418.311856 fallocate 0x019933 thru 0x01dd15 (0x43e3 bytes) 000785 1778653418.413105 read 0x00aceb thru 0x014e47 (0xa15d bytes) 000786 1778653418.437925 mapread 0x028a61 thru 0x02a26b (0x180b bytes) 000789 1778653418.483626 mapread 0x0111d1 thru 0x01254e (0x137e bytes) 000791 1778653418.490599 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a26c): Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a26c): Mirror 22: 0xe9f349b5 (stale) Mirror 24: 0xe9f349b5 (stale) CRC-32 checksum value for chunk [0, 0x2a26c): Mirror 1: 0xe9f349b5 Mirror 25: 0xe9f349b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a26c: skipped 000792 1778653419.644045 trunc from 0x02a26c to 0x038f39 (0xecce bytes) 000793 1778653419.720463 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f3a): Mirror 22: 0x399874c6 (stale) Mirror 25: 0x399874c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f3a): Mirror 22: 0x399874c6 (stale) Mirror 25: 0x399874c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f3a: skipped 000794 1778653420.222093 trunc from 0x038f3a to 0x006e71 (0x320c8 bytes) 000796 1778653420.263025 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6e72): Mirror 22: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6e72): Mirror 22: 0x2b6731d5 (stale) Mirror 25: 0x2b6731d5 (stale) CRC-32 checksum value for chunk [0, 0x6e72): Mirror 1: 0x2b6731d5 Mirror 26: 0x2b6731d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6e72: skipped 000798 1778653421.765854 write 0x039dd8 thru 0x03ffff (0x6228 bytes) 000799 1778653421.826405 trunc from 0x040000 to 0x01acea (0x25315 bytes) 000800 1778653421.879451 punch from 0x018454 to 0x01acea (0x2897 bytes) 000801 1778653421.905050 punch from 0x017361 to 0x01acea (0x398a bytes) 000802 1778653421.912535 trunc from 0x001587 to 0x00979e (0x8218 bytes) 000803 1778653421.926788 fallocate 0x015187 thru 0x022a94 (0xd90e bytes) 000804 1778653421.944802 write 0x002dfc thru 0x00c3f3 (0x95f8 bytes) 000805 1778653421.983827 fallocate 0x022f22 thru 0x02a7e8 (0x78c7 bytes) 000806 1778653421.993076 mapread 0x0157ba thru 0x022a94 (0xd2db bytes) 000807 1778653422.031837 trunc from 0x022a95 to 0x010917 (0x1217d bytes) 000808 1778653422.056002 read 0x008e5b thru 0x00efaf (0x6155 bytes) 000809 1778653422.062468 fallocate 0x020d2f thru 0x0258b3 (0x4b85 bytes) 000810 1778653422.069096 punch from 0x00cb78 to 0x00f17c (0x2605 bytes) 000812 1778653422.096633 mapwrite 0x034652 thru 0x03ffff (0xb9ae bytes) 000813 1778653422.210004 fallocate 0x01f3db thru 0x027d56 (0x897c bytes) 000814 1778653422.220378 mapwrite 0x008eb7 thru 0x00b614 (0x275e bytes) 000815 1778653422.252813 mapread 0x00b4f5 thru 0x019d1b (0xe827 bytes) 000816 1778653422.292770 mapread 0x01b3e1 thru 0x0259a5 (0xa5c5 bytes) 000818 1778653422.320444 MIRROR_RESYNC 4 mirrors 000819 1778653423.569540 read 0x03f7c1 thru 0x03ffff (0x83f bytes) 000820 1778653423.574903 mapread 0x032978 thru 0x03b5e8 (0x8c71 bytes) 000821 1778653423.624594 punch from 0x000ae9 to 0x000b48 (0x60 bytes) 000822 1778653423.695520 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x2739b036 (stale) Mirror 25: 0x2739b036 (stale) Mirror 26: 0x2739b036 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc69d2b5 Mirror 27: 0xdc69d2b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000823 1778653425.126826 read 0x008f2f thru 0x00a0a8 (0x117a bytes) 000824 1778653425.182023 mapread 0x00b4a6 thru 0x00ee52 (0x39ad bytes) 000825 1778653425.269335 trunc from 0x040000 to 0x0207c8 (0x1f837 bytes) 000826 1778653425.358936 mapwrite 0x029aba thru 0x039363 (0xf8aa bytes) 000827 1778653425.505977 mapread 0x01ce96 thru 0x02c6d8 (0xf843 bytes) 000828 1778653425.536026 mapwrite 0x02a79a thru 0x03283f (0x80a6 bytes) 000829 1778653425.620934 punch from 0x00c07a to 0x00f054 (0x2fdb bytes) 000830 1778653425.640213 read 0x0204e9 thru 0x02868f (0x81a7 bytes) 000831 1778653425.641248 fallocate 0x037a31 thru 0x03ffff (0x85cf bytes) 000832 1778653425.672305 MIRROR_RESYNC 5 mirrors 000834 1778653427.860880 fallocate 0x011b6b thru 0x01c5e2 (0xaa78 bytes) 000835 1778653427.917262 mapwrite 0x03b1ec thru 0x03ffff (0x4e14 bytes) 000836 1778653428.011254 mapread 0x034b34 thru 0x035860 (0xd2d bytes) 000837 1778653428.028465 fallocate 0x00ee8b thru 0x017dfe (0x8f74 bytes) 000838 1778653428.035895 read 0x019be3 thru 0x01e8e6 (0x4d04 bytes) 000839 1778653428.046464 trunc from 0x040000 to 0x020084 (0x1ff7b bytes) 000840 1778653428.085028 punch from 0x016652 to 0x01e5d2 (0x7f81 bytes) 000841 1778653428.098869 punch from 0x003db0 to 0x012aed (0xed3e bytes) 000842 1778653428.108775 fallocate 0x022440 thru 0x026eaf (0x4a70 bytes) 000843 1778653428.114904 trunc from 0x002b1c to 0x00c0fe (0x95e3 bytes) 000844 1778653428.131853 MIRROR_RESYNC 5 mirrors 000845 1778653429.647256 mapwrite 0x007a21 thru 0x011195 (0x9775 bytes) 000846 1778653429.832898 fallocate 0x03bf14 thru 0x03ffff (0x40ec bytes) 000847 1778653429.854299 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26eb0): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26eb0): Mirror 22: 0xea88bde9 (stale) Mirror 25: 0xea88bde9 (stale) Mirror 26: 0xea88bde9 (stale) Mirror 27: 0xea88bde9 (stale) CRC-32 checksum value for chunk [0, 0x26eb0): Mirror 1: 0x3abb1489 Mirror 28: 0x3abb1489 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26eb0: skipped 000848 1778653431.436767 mapread 0x009565 thru 0x013f82 (0xaa1e bytes) 000849 1778653431.483739 punch from 0x018a8a to 0x01f6de (0x6c55 bytes) 000850 1778653431.540651 mapread 0x00a8fa thru 0x00c2a6 (0x19ad bytes) 000851 1778653431.566808 fallocate 0x01f7c1 thru 0x02dd7c (0xe5bc bytes) 000852 1778653431.581870 punch from 0x019d7f to 0x01e29e (0x4520 bytes) 000853 1778653431.598696 write 0x02945c thru 0x02b019 (0x1bbe bytes) 000854 1778653431.611613 punch from 0x01e655 to 0x028f0e (0xa8ba bytes) 000855 1778653431.627294 write 0x01402a thru 0x0232ed (0xf2c4 bytes) 000856 1778653431.669668 trunc from 0x00800f to 0x0142d3 (0xc2c5 bytes) 000857 1778653431.740566 mapwrite 0x03bcfa thru 0x03ffff (0x4306 bytes) 000858 1778653431.835916 trunc from 0x040000 to 0x009101 (0x36efe bytes) 000859 1778653431.882780 punch from 0x000ed2 to 0x009101 (0x8230 bytes) 000860 1778653431.900419 trunc from 0x004c58 to 0x009101 (0x44aa bytes) 000861 1778653431.911650 mapwrite 0x02f85c thru 0x03df25 (0xe6ca bytes) 000862 1778653431.983003 mapread 0x01c4ef thru 0x0288e1 (0xc3f3 bytes) 000863 1778653432.007333 mapread 0x01af15 thru 0x01b33c (0x428 bytes) 000864 1778653432.008597 punch from 0x001023 to 0x008186 (0x7164 bytes) 000865 1778653432.022400 mapread 0x03a6f4 thru 0x03df25 (0x3832 bytes) 000866 1778653432.024590 read 0x00a7cd thru 0x019a77 (0xf2ab bytes) 000867 1778653432.045730 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3df26): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df26): Mirror 22: 0xf68741f4 (stale) Mirror 25: 0xf68741f4 (stale) Mirror 26: 0xf68741f4 (stale) Mirror 27: 0xf68741f4 (stale) Mirror 28: 0x4905eb44 (stale) CRC-32 checksum value for chunk [0, 0x3df26): Mirror 1: 0x4d62965 Mirror 29: 0x4d62965 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df26: skipped 000869 1778653433.183115 fallocate 0x01baaf thru 0x0260db (0xa62d bytes) 000870 1778653433.233000 trunc from 0x03df26 to 0x037eb3 (0x6072 bytes) 000871 1778653433.266754 punch from 0x009954 to 0x015ce8 (0xc395 bytes) 000872 1778653433.274269 punch from 0x02d776 to 0x03521f (0x7aaa bytes) 000873 1778653433.282544 write 0x004b34 thru 0x009450 (0x491d bytes) 000874 1778653433.295345 read 0x008292 thru 0x00c370 (0x40df bytes) 000875 1778653433.302677 mapwrite 0x00e54e thru 0x018fb7 (0xaa6a bytes) 000876 1778653433.349686 mapread 0x019e88 thru 0x0259e0 (0xbb59 bytes) 000877 1778653433.363387 mapwrite 0x0130bb thru 0x014261 (0x11a7 bytes) 000878 1778653433.377715 read 0x015713 thru 0x024621 (0xef0f bytes) 000879 1778653433.383863 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37eb4): Mirror 22: 0xb733bb7d (stale) Mirror 25: 0xb733bb7d (stale) Mirror 26: 0xb733bb7d (stale) Mirror 27: 0xb733bb7d (stale) Mirror 28: 0x997ef90f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eb4): Mirror 22: 0xb733bb7d (stale) Mirror 25: 0xb733bb7d (stale) Mirror 26: 0xb733bb7d (stale) Mirror 27: 0xb733bb7d (stale) Mirror 28: 0x997ef90f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eb4: skipped 000880 1778653433.811929 write 0x034c33 thru 0x03ffff (0xb3cd bytes) 000882 1778653433.824800 fallocate 0x0253c5 thru 0x02704b (0x1c87 bytes) 000883 1778653433.830274 fallocate 0x027392 thru 0x035b3d (0xe7ac bytes) 000885 1778653433.836734 write 0x031488 thru 0x03fcf3 (0xe86c bytes) 000886 1778653433.842133 mapwrite 0x020e90 thru 0x02f6b2 (0xe823 bytes) 000887 1778653433.882345 fallocate 0x0241c9 thru 0x032c9c (0xead4 bytes) 000888 1778653433.887504 mapwrite 0x013b5a thru 0x01505c (0x1503 bytes) 000889 1778653433.916345 mapwrite 0x025916 thru 0x02f365 (0x9a50 bytes) 000890 1778653433.940761 mapread 0x012de1 thru 0x01b25d (0x847d bytes) 000891 1778653433.943237 trunc from 0x01d74b to 0x029998 (0xc24e bytes) 000892 1778653433.948895 trunc from 0x040000 to 0x00f146 (0x30eb9 bytes) 000893 1778653433.966305 write 0x0187f9 thru 0x0213d2 (0x8bda bytes) 000894 1778653433.969399 read 0x012574 thru 0x0167ff (0x428c bytes) 000895 1778653433.976874 mapread 0x002e07 thru 0x006cb3 (0x3ead bytes) 000896 1778653433.990689 trunc from 0x0213d3 to 0x03c6d5 (0x1b303 bytes) 000897 1778653434.010323 trunc from 0x026050 to 0x030eda (0xae8b bytes) 000899 1778653434.017110 trunc from 0x02deea to 0x035c90 (0x7da7 bytes) 000901 1778653434.022889 read 0x02fcff thru 0x034cb4 (0x4fb6 bytes) 000903 1778653434.039717 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 22: 0x2ce91958 (stale) Mirror 25: 0x2ce91958 (stale) Mirror 26: 0x2ce91958 (stale) Mirror 27: 0x2ce91958 (stale) Mirror 28: 0x7b43d8b6 (stale) CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 1: 0x3a59452b Mirror 29: 0x3a59452b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6d6: skipped 000904 1778653435.183803 trunc from 0x008cf0 to 0x010c16 (0x7f27 bytes) 000905 1778653435.232838 write 0x037bd3 thru 0x03ffff (0x842d bytes) 000906 1778653435.246129 punch from 0x015596 to 0x01930b (0x3d76 bytes) 000907 1778653435.252323 fallocate 0x0134cf thru 0x0166dc (0x320e bytes) 000908 1778653435.261674 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x7275b9e8 (stale) Mirror 25: 0x7275b9e8 (stale) Mirror 26: 0x7275b9e8 (stale) Mirror 27: 0x7275b9e8 (stale) Mirror 28: 0xbb123653 (stale) Mirror 29: 0x3269b019 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9016d93f Mirror 30: 0x9016d93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1778653436.456506 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x7275b9e8 (stale) Mirror 26: 0x7275b9e8 (stale) Mirror 27: 0x7275b9e8 (stale) Mirror 28: 0xbb123653 (stale) Mirror 29: 0x3269b019 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x7275b9e8 (stale) Mirror 26: 0x7275b9e8 (stale) Mirror 27: 0x7275b9e8 (stale) Mirror 28: 0xbb123653 (stale) Mirror 29: 0x3269b019 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9016d93f Mirror 30: 0x9016d93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1778653437.286133 write 0x027c31 thru 0x02b705 (0x3ad5 bytes) 000911 1778653437.332853 trunc from 0x040000 to 0x03d08d (0x2f72 bytes) 000912 1778653437.348491 punch from 0x0325a9 to 0x033f60 (0x19b8 bytes) 000914 1778653437.352248 write 0x036f7e thru 0x03ffff (0x9082 bytes) 000915 1778653437.362660 punch from 0x03fb2f to 0x03ffff (0x4d1 bytes) 000916 1778653437.381811 write 0x02fb8d thru 0x03cc79 (0xd0ed bytes) 000917 1778653437.383890 mapread 0x01e2ab thru 0x022173 (0x3ec9 bytes) 000918 1778653437.396095 read 0x009a55 thru 0x00eacf (0x507b bytes) 000919 1778653437.409237 MIRROR_RESYNC 7 mirrors 000920 1778653439.627934 read 0x015c36 thru 0x018b27 (0x2ef2 bytes) 000922 1778653439.644989 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 000923 1778653439.977273 trunc from 0x027ee4 to 0x031e09 (0x9f26 bytes) 000925 1778653440.025627 write 0x018206 thru 0x026b58 (0xe953 bytes) 000926 1778653440.052312 trunc from 0x029080 to 0x038558 (0xf4d9 bytes) 000927 1778653440.061719 read 0x03b545 thru 0x03fb99 (0x4655 bytes) 000931 1778653440.073597 trunc from 0x040000 to 0x013b52 (0x2c4ad bytes) 000932 1778653440.103586 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13b53): Mirror 26: 0x8bc5dfc1 (stale) Mirror 28: 0x8bc5dfc1 (stale) Mirror 29: 0x8bc5dfc1 (stale) Mirror 30: 0x8bc5dfc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b53): Mirror 26: 0x8bc5dfc1 (stale) Mirror 28: 0x8bc5dfc1 (stale) Mirror 29: 0x8bc5dfc1 (stale) Mirror 30: 0x8bc5dfc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b53: skipped 000933 1778653440.441281 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13b53): Mirror 26: 0x8bc5dfc1 (stale) Mirror 28: 0x8bc5dfc1 (stale) Mirror 29: 0x8bc5dfc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b53): Mirror 26: 0x8bc5dfc1 (stale) Mirror 28: 0x8bc5dfc1 (stale) Mirror 29: 0x8bc5dfc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b53: skipped 000934 1778653440.738211 fallocate 0x001d27 thru 0x002554 (0x82e bytes) 000935 1778653440.742866 punch from 0x0112d0 to 0x013b52 (0x2883 bytes) 000938 1778653440.748170 mapread 0x00936e thru 0x00f24c (0x5edf bytes) 000939 1778653440.764207 read 0x001f44 thru 0x009910 (0x79cd bytes) 000940 1778653440.772565 trunc from 0x013b53 to 0x00297b (0x111d7 bytes) 000941 1778653440.805753 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x297c): Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297c): Mirror 26: 0x98bd157c (stale) Mirror 28: 0x98bd157c (stale) Mirror 29: 0x98bd157c (stale) CRC-32 checksum value for chunk [0, 0x297c): Mirror 1: 0x98bd157c Mirror 30: 0x98bd157c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297c: skipped 000942 1778653441.669862 mapwrite 0x039f5a thru 0x03ffff (0x60a6 bytes) 000943 1778653441.795966 mapread 0x02e25c thru 0x03b9a0 (0xd745 bytes) 000944 1778653441.813646 read 0x028979 thru 0x02f733 (0x6dbb bytes) 000945 1778653441.815636 write 0x00cb8b thru 0x00e61a (0x1a90 bytes) 000946 1778653441.828567 read 0x011dc4 thru 0x021b5f (0xfd9c bytes) 000947 1778653441.842502 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0xad24081f (stale) Mirror 29: 0xad24081f (stale) Mirror 30: 0xdf54cab2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0xad24081f (stale) Mirror 29: 0xad24081f (stale) Mirror 30: 0xdf54cab2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000948 1778653442.224344 trunc from 0x040000 to 0x023549 (0x1cab6 bytes) 000949 1778653442.241145 read 0x015467 thru 0x016e8a (0x1a24 bytes) 000950 1778653442.245546 read 0x001223 thru 0x0072df (0x60bd bytes) 000951 1778653442.255493 mapwrite 0x00d1e3 thru 0x015f63 (0x8d81 bytes) 000952 1778653442.278078 trunc from 0x02354a to 0x00d099 (0x164b0 bytes) 000953 1778653442.292032 write 0x026dc5 thru 0x02deb2 (0x70ee bytes) 000954 1778653442.297551 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2deb3): Mirror 29: 0x3edcdde4 (stale) Mirror 30: 0x98eaf80d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2deb3): Mirror 29: 0x3edcdde4 (stale) Mirror 30: 0x98eaf80d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2deb3: skipped 000955 1778653442.586622 punch from 0x012e00 to 0x014e06 (0x2007 bytes) 000956 1778653442.591201 read 0x0075b2 thru 0x008308 (0xd57 bytes) 000957 1778653442.596286 punch from 0x0098af to 0x015eb0 (0xc602 bytes) 000958 1778653442.601359 mapread 0x0220ea thru 0x02215a (0x71 bytes) 000959 1778653442.608044 trunc from 0x0274e4 to 0x02deb2 (0x69cf bytes) 000960 1778653442.614860 trunc from 0x01ef84 to 0x01f31c (0x399 bytes) 000961 1778653442.620056 mapwrite 0x0093e3 thru 0x018d2c (0xf94a bytes) 000962 1778653442.664969 mapwrite 0x0266c3 thru 0x030256 (0x9b94 bytes) 000963 1778653442.705639 fallocate 0x01a4ff thru 0x01bd4a (0x184c bytes) 000964 1778653442.709954 fallocate 0x02f067 thru 0x036f5a (0x7ef4 bytes) 000965 1778653442.717487 MIRROR_RESYNC 3 mirrors 000966 1778653443.687571 mapwrite 0x00a0ba thru 0x019b98 (0xfadf bytes) 000967 1778653443.794992 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30257): Mirror 30: 0xfe5a49b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30257): Mirror 30: 0xfe5a49b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30257: skipped 000968 1778653444.014800 punch from 0x02ed04 to 0x02fc95 (0xf92 bytes) 000970 1778653444.022273 write 0x022df2 thru 0x030a07 (0xdc16 bytes) 000971 1778653444.034438 trunc from 0x0188e2 to 0x023384 (0xaaa3 bytes) 000972 1778653444.057492 trunc from 0x01d65d to 0x02a255 (0xcbf9 bytes) 000973 1778653444.067635 MIRROR_RESYNC 2 mirrors 000975 1778653444.760676 write 0x0191cd thru 0x01bb91 (0x29c5 bytes) 000976 1778653444.814695 punch from 0x0145cd to 0x01cd9b (0x87cf bytes) 000977 1778653444.825497 punch from 0x0193f6 to 0x01addc (0x19e7 bytes) 000978 1778653444.835730 MIRROR_RESYNC 2 mirrors 000979 1778653445.457401 trunc from 0x00eabb to 0x010804 (0x1d4a bytes) 000980 1778653445.499955 mapwrite 0x03035e thru 0x03b8c6 (0xb569 bytes) 000981 1778653445.579641 read 0x026cd0 thru 0x032d62 (0xc093 bytes) 000982 1778653445.589706 fallocate 0x02881e thru 0x02dcbc (0x549f bytes) 000983 1778653445.594714 trunc from 0x00fb6a to 0x018ddf (0x9276 bytes) 000984 1778653445.600463 trunc from 0x03b8c7 to 0x02b899 (0x1002d bytes) 000985 1778653445.625777 fallocate 0x012de4 thru 0x020952 (0xdb6f bytes) 000986 1778653445.631009 fallocate 0x03195d thru 0x037d0c (0x63b0 bytes) 000987 1778653445.641937 MIRROR_RESYNC 2 mirrors 000988 1778653446.532032 MIRROR_RESYNC 2 mirrors 000989 1778653446.884550 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 000990 1778653447.079495 trunc from 0x02fded to 0x037d0c (0x7f20 bytes) 000993 1778653447.115076 read 0x00b789 thru 0x015002 (0x987a bytes) 000994 1778653447.133199 MIRROR_EXTEND to 2 mirrors 000995 1778653447.973064 punch from 0x02cfb6 to 0x037d0c (0xad57 bytes) 000996 1778653448.001515 mapwrite 0x0110d5 thru 0x020259 (0xf185 bytes) 000997 1778653448.068301 fallocate 0x00e584 thru 0x012df5 (0x4872 bytes) 000998 1778653448.073019 trunc from 0x037d0d to 0x0380be (0x3b2 bytes) 000999 1778653448.127910 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15c:0x0]: Invalid argument (22) 001000 1778653448.279401 trunc from 0x00c730 to 0x01785b (0xb12c bytes) All operations completed A-OK! PASS 70a (146s) == sanity-flr test 71: check mirror extend parent fid ==== 02:24:13 (1778653453) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1a9:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 02:24:23 (1778653463) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201245 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099878 s, 10.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:24:31 (1778653471) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7361664 .. Extending file size to 6253920 .. done Extending file size to 8876960 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7746976 .. Extending file size to 8878656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8933280 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60943 60944 60946 60947 60949 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:25:52 (1778653552) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1682464 .. Extending file size to 1142016 .. done Extending file size to 4205056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4272992 .. Extending file size to 4442272 .. Extending file size to 7048544 .. Extending file size to 8831968 .. Extending file size to 9123392 .. Extending file size to 7961568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8394016 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8517536 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8743104 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62543 62545 62546 62548 62549 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:27:14 (1778653634) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17784 3589236 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25740 3581280 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43524 7170516 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17784 3589236 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25740 3581280 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43524 7170516 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148416 s, 14.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236371 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9da57be4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:27:35 (1778653655) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0911306 s, 23.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:27:42 (1778653662) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:27:49 (1778653669) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0821732 s, 25.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:27:57 (1778653677) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114019 s, 18.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:28:05 (1778653685) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101228 s, 20.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:28:36 (1778653716) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645263 s, 16.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:29:21 (1778653761) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111632 s, 18.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:30:12 (1778653812) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg138-server: Pool lustre.test_204f created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141575 s, 14.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg138-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:31:22 (1778653882) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:31:29 (1778653889) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:31:38 (1778653898) oleg138-server: Pool lustre.test_206 created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg138-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:32:06 (1778653926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100542 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040696 s, 25.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x59c58bd8 Mirror 3: 0x59c58bd8 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:32:16 (1778653936) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:32:20 (1778653940) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:32:23 (1778653943) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0363262 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1660000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x10e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x10e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x10e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb30000 Read this (4 bytes): BBBB 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:32:56 (1778653976) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00956981 s, 856 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:33:01 (1778653981) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414596 s, 25.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611260 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-13 02:33:02.000000000 -0400 Modify: 2026-05-13 02:33:02.000000000 -0400 Change: 2026-05-13 02:33:03.000000000 -0400 Birth: 2026-05-13 02:33:03.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:33:07 (1778653987) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238129 s, 44.0 MB/s oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg138-server: O/280000401/d27/13b: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:33:26 (1778654006) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.217212 s, 189 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0151655 s, 270 kB/s size after second write -rw-r--r-- 1 root root 4096 May 13 02:33 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x81:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 13 02:33 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:33:32 (1778654012) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:33:38 (1778654018) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1594 sec =========== 02:33:46 (1778654026) === sanity-flr: start cleanup 02:33:47 (1778654027) === === sanity-flr: finish cleanup 02:33:54 (1778654034) ===