-----============= acceptance-small: sanity-flr ============----- Wed Apr 22 10:54:33 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg129-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:54:40 (1776869680) === oleg129-client.virtnet: executing check_config_client /mnt/lustre oleg129-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg129-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ada83ffd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ada83ffd000.idle_timeout=debug disable quota as required oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:54:53 (1776869693) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8181] == sanity-flr test 0a: lfs mirror create with -N option == 10:54:54 (1776869694) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:54:59 (1776869699) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:55:01 (1776869701) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:55:02 (1776869702) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:55:10 (1776869710) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:55:12 (1776869712) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 10:55:13 (1776869713) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ada83ffd000.stats=clear osc.lustre-OST0001-osc-ffff9ada83ffd000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:55:16 (1776869716) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:55:20 (1776869720) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:55:23 (1776869723) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:55:24 (1776869724) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:55:27 (1776869727) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 10:55:30 (1776869730) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:55:32 (1776869732) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:55:34 (1776869734) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:55:37 (1776869737) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:55:40 (1776869740) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:55:42 (1776869742) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.362044 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319145 s, 32.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:55:45 (1776869745) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.0776221 s, 94.6 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:55:48 (1776869748) Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:56:14 (1776869774) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0483227 s, 43.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:56:27 (1776869787) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:56:52 (1776869812) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.900437 s, 58.2 MB/s -rw-r--r-- 1 root root 52428800 Apr 22 10:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.283s user 0m0.001s sys 0m0.140s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.312s user 0m0.000s sys 0m0.151s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.329s user 0m0.001s sys 0m0.149s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ada83ffd000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:57:19 (1776869839) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:57:20 (1776869840) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:57:22 (1776869842) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:57:23 (1776869843) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 10:57:25 (1776869845) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.152849 s, 82.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:57:30 (1776869850) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140536 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137327 s, 76.4 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:57:32 (1776869852) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:57:36 (1776869856) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000638773 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000684927 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000643542 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000708875 s, 1.4 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 10:57:39 (1776869859) create test files with size 15755006 .. 34e8730f4ab06c31462af698742fea95 - 502924c22273f8a77a1349151e2ad81d - 60c2c259b215e983661485cf392e672f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 10:57:46 (1776869866) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:57:49 (1776869869) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:57:52 (1776869872) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242453 s, 43.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x33:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110001 s, 95.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x34:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 10:57:55 (1776869875) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0320302 s, 65.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0992808 s, 42.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423573 s, 49.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 10:57:59 (1776869879) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:58:00 (1776869880) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:58:01 (1776869881) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138819 s, 75.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 10:58:04 (1776869884) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0975527 s, 32.2 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:58:08 (1776869888) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:58:12 (1776869892) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10978 s, 95.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:58:28 (1776869908) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9ada867cf000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.143848 s, 72.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9adab0490000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:58:33 (1776869913) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:58:46 (1776869926) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:59:00 (1776869940) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:59:01 (1776869941) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0906793 s, 34.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0595221 s, 52.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x66e2a3de (stale) Mirror 2: 0x66e2a3de (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f5013179e77b139b76f260217323cba0 - mirror 1 checksum: 82fa76d0b7eac19c1f11775d64a9afb4 - mirror 2 checksum: f5013179e77b139b76f260217323cba0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 22 10:59:01 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 22 10:59:01 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f5013179e77b139b76f260217323cba0 - mirror 1 checksum: 82fa76d0b7eac19c1f11775d64a9afb4 - mirror 2 checksum: 82fa76d0b7eac19c1f11775d64a9afb4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 22 10:59:01 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 22 10:59:01 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:59:04 (1776869944) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111782 s, 93.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:59:06 (1776869946) Failing mds1 on oleg129-server Stopping /mnt/lustre-mds1 (opts:) on oleg129-server 10:59:14 (1776869954) shut down facet: mds1 facet_host: oleg129-server facet_failover_host: oleg129-server Failover mds1 to oleg129-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 10:59:28 (1776869968) targets are mounted 10:59:28 (1776869968) facet_failover done oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150068 s, 69.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:59:37 (1776869977) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139545 s, 1.5 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00653853 s, 3.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:59:40 (1776869980) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0131863 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00328821 s, 6.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:59:43 (1776869983) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00294648 s, 5.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:59:45 (1776869985) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0150526 s, 1.4 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:59:48 (1776869988) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00280188 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00762657 s, 537 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:59:50 (1776869990) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:59:57 (1776869997) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:00:17 (1776870017) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:00:35 (1776870035) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:00:53 (1776870053) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 11:00:55 (1776870055) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 56336 mirror split pid 56337 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 56336 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 56337 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 11:01:57 (1776870117) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 118 000001 1776870118.438744 mapwrite 0x03132d thru 0x03ab3c (0x9810 bytes) fallocating to largest ever: 0x2ee5e 000002 1776870118.458858 fallocate 0x028ad7 thru 0x02ee5d (0x6387 bytes) 000003 1776870118.461506 mapread 0x00e133 thru 0x00fda5 (0x1c73 bytes) 000006 1776870118.466712 mapread 0x005107 thru 0x01452f (0xf429 bytes) 000007 1776870118.470697 trunc from 0x01b4ea to 0x0220c2 (0x6bd9 bytes) 000008 1776870118.472553 write 0x03b23d thru 0x03ffff (0x4dc3 bytes) 000009 1776870118.473574 punch from 0x03deb6 to 0x03ffff (0x214a bytes) 000010 1776870118.477809 punch from 0x027f6f to 0x037361 (0xf3f3 bytes) fallocating to largest ever: 0x34749 000012 1776870118.480194 fallocate 0x026c3c thru 0x034748 (0xdb0d bytes) 000013 1776870118.483004 mapwrite 0x020563 thru 0x02deb3 (0xd951 bytes) 000014 1776870118.502046 write 0x01eb15 thru 0x02cf92 (0xe47e bytes) 000015 1776870118.503463 write 0x01c83c thru 0x02c556 (0xfd1b bytes) 000018 1776870118.506405 trunc from 0x040000 to 0x01760a (0x289f5 bytes) 000019 1776870118.514182 MIRROR_EXTEND to 2 mirrors 000020 1776870118.682595 mapwrite 0x01ea26 thru 0x02c28d (0xd868 bytes) fallocating to largest ever: 0x3d02f 000021 1776870118.721207 fallocate 0x037dea thru 0x03d02e (0x5245 bytes) 000022 1776870118.723396 mapread 0x01c901 thru 0x023fdd (0x76dd bytes) 000023 1776870118.724302 write 0x0251aa thru 0x034482 (0xf2d9 bytes) 000024 1776870118.725545 trunc from 0x0087f6 to 0x0180b3 (0xf8be bytes) 000025 1776870118.729374 MIRROR_RESYNC 2 mirrors 000026 1776870118.888974 mapread 0x007f8a thru 0x00fda6 (0x7e1d bytes) 000027 1776870118.894443 mapwrite 0x025781 thru 0x03034e (0xabce bytes) 000028 1776870118.935029 fallocate 0x021bee thru 0x0222fa (0x70d bytes) 000029 1776870118.937266 mapwrite 0x01d518 thru 0x01f330 (0x1e19 bytes) 000030 1776870118.944907 write 0x003343 thru 0x005757 (0x2415 bytes) 000031 1776870118.948326 mapread 0x0215d7 thru 0x021d8b (0x7b5 bytes) 000032 1776870118.949079 trunc from 0x034483 to 0x00e9d2 (0x25ab0 bytes) 000033 1776870118.963029 read 0x00de6f thru 0x00e9d2 (0xb64 bytes) 000034 1776870118.963531 trunc from 0x00e9d3 to 0x03360c (0x24c3a bytes) 000035 1776870118.970555 punch from 0x01f4ef to 0x0284bf (0x8fd1 bytes) 000036 1776870118.972697 mapread 0x004870 thru 0x00f6fd (0xae8e bytes) 000037 1776870118.976609 trunc from 0x03360d to 0x01be5e (0x177ae bytes) 000039 1776870118.982141 fallocate 0x012ed2 thru 0x01c04f (0x917e bytes) 000040 1776870118.985719 MIRROR_RESYNC 2 mirrors 000041 1776870119.151088 mapwrite 0x00f7b6 thru 0x01f02f (0xf87a bytes) 000043 1776870119.177221 mapread 0x0009a9 thru 0x00a6bc (0x9d14 bytes) 000044 1776870119.180966 punch from 0x0003b6 to 0x0091ff (0x8e4a bytes) 000045 1776870119.183147 write 0x02e6f0 thru 0x035645 (0x6f56 bytes) 000046 1776870119.184442 mapwrite 0x006ee2 thru 0x007004 (0x123 bytes) 000047 1776870119.195886 read 0x02bb71 thru 0x035645 (0x9ad5 bytes) 000048 1776870119.198191 write 0x00993a thru 0x00bc71 (0x2338 bytes) 000049 1776870119.199142 trunc from 0x03373c to 0x035645 (0x1f0a bytes) 000050 1776870119.200829 trunc from 0x021cd9 to 0x02a3fb (0x8723 bytes) 000052 1776870119.203005 mapwrite 0x00d063 thru 0x010ee7 (0x3e85 bytes) 000053 1776870119.211311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000054 1776870119.256483 trunc from 0x035646 to 0x0090ed (0x2c558 bytes) 000055 1776870119.265324 trunc from 0x0090ee to 0x0251e7 (0x1c0fa bytes) 000056 1776870119.271109 trunc from 0x0251e8 to 0x0300ad (0xaec6 bytes) 000057 1776870119.279857 write 0x033d01 thru 0x03e3d9 (0xa6d9 bytes) 000058 1776870119.281182 fallocate 0x00500a thru 0x00bd9b (0x6d92 bytes) 000059 1776870119.284464 trunc from 0x03e3da to 0x0273ab (0x1702e bytes) 000060 1776870119.290147 trunc from 0x0273ac to 0x02cb4f (0x57a4 bytes) 000061 1776870119.294745 fallocate 0x00d0fd thru 0x012b34 (0x5a38 bytes) fallocating to largest ever: 0x40000 000062 1776870119.297005 fallocate 0x034273 thru 0x03ffff (0xbd8d bytes) 000064 1776870119.299295 mapwrite 0x033b54 thru 0x035798 (0x1c45 bytes) 000066 1776870119.317089 mapread 0x02713c thru 0x034818 (0xd6dd bytes) 000067 1776870119.321271 mapwrite 0x00b060 thru 0x00d9c4 (0x2965 bytes) 000069 1776870119.329601 mapread 0x032211 thru 0x035798 (0x3588 bytes) 000070 1776870119.330242 write 0x026ce4 thru 0x02fb6a (0x8e87 bytes) 000072 1776870119.331633 read 0x00a734 thru 0x00f0e0 (0x49ad bytes) 000073 1776870119.332059 trunc from 0x01ef68 to 0x02e4f1 (0xf58a bytes) 000074 1776870119.336608 punch from 0x032384 to 0x033d3d (0x19ba bytes) 000075 1776870119.338241 read 0x0263fa thru 0x035798 (0xf39f bytes) 000076 1776870119.341297 mapread 0x027db2 thru 0x030104 (0x8353 bytes) 000077 1776870119.342155 trunc from 0x0188df to 0x01e0c4 (0x57e6 bytes) 000078 1776870119.344165 punch from 0x015367 to 0x023b75 (0xe80f bytes) 000079 1776870119.346177 trunc from 0x035799 to 0x029e4f (0xb949 bytes) 000080 1776870119.351185 mapread 0x01bae7 thru 0x025692 (0x9bac bytes) 000082 1776870119.365665 fallocate 0x038aa0 thru 0x03ffff (0x7560 bytes) 000084 1776870119.367680 mapwrite 0x031273 thru 0x0381bc (0x6f4a bytes) 000085 1776870119.383936 trunc from 0x040000 to 0x01e737 (0x218c8 bytes) 000086 1776870119.390366 fallocate 0x02d31a thru 0x036c96 (0x997d bytes) 000087 1776870119.394659 fallocate 0x03363a thru 0x03b312 (0x7cd9 bytes) 000088 1776870119.396878 trunc from 0x03b313 to 0x0255c3 (0x15d4f bytes) 000089 1776870119.402216 fallocate 0x022bfa thru 0x02f47c (0xc883 bytes) 000090 1776870119.404798 trunc from 0x02f47d to 0x032963 (0x34e7 bytes) 000091 1776870119.409213 mapread 0x023479 thru 0x024a4e (0x15d6 bytes) 000094 1776870119.414339 trunc from 0x010aa4 to 0x0206f1 (0xfc4e bytes) 000096 1776870119.417019 write 0x02ccd5 thru 0x034858 (0x7b84 bytes) 000098 1776870119.419589 read 0x02a721 thru 0x034858 (0xa138 bytes) 000099 1776870119.423072 fallocate 0x0299e8 thru 0x037248 (0xd861 bytes) 000100 1776870119.425210 trunc from 0x00f8c3 to 0x01201d (0x275b bytes) 000101 1776870119.427326 mapread 0x02b08e thru 0x035b9d (0xab10 bytes) 000103 1776870119.430714 mapwrite 0x036ecd thru 0x037639 (0x76d bytes) 000104 1776870119.439965 write 0x025b42 thru 0x032443 (0xc902 bytes) 000105 1776870119.442467 MIRROR_EXTEND to 2 mirrors 000107 1776870119.631134 read 0x036fa0 thru 0x037639 (0x69a bytes) 000108 1776870119.633018 trunc from 0x03763a to 0x034633 (0x3006 bytes) 000109 1776870119.654593 trunc from 0x034634 to 0x037583 (0x2f50 bytes) 000110 1776870119.660658 mapwrite 0x028089 thru 0x034d52 (0xccca bytes) 000111 1776870119.675124 read 0x004d01 thru 0x00b4a1 (0x67a1 bytes) 000112 1776870119.678877 read 0x02b2f4 thru 0x035e15 (0xab22 bytes) 000113 1776870119.679414 trunc from 0x0327b6 to 0x037583 (0x4dce bytes) 000114 1776870119.683162 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37584): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37584): Mirror 2: 0x8e629d28 (stale) CRC-32 checksum value for chunk [0, 0x37584): Mirror 1: 0xa84eb35d Mirror 3: 0xa84eb35d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37584: skipped 000115 1776870119.942446 read 0x00c86b thru 0x01805e (0xb7f4 bytes) 000116 1776870119.946882 write 0x001319 thru 0x00f157 (0xde3f bytes) 000117 1776870119.963332 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37584): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37584): Mirror 2: 0x8e629d28 (stale) Mirror 3: 0xa84eb35d (stale) CRC-32 checksum value for chunk [0, 0x37584): Mirror 1: 0xa0354dc Mirror 4: 0xa0354dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37584: skipped 000119 1776870120.235359 trunc from 0x037584 to 0x0049e8 (0x32b9b bytes) 000120 1776870120.251995 trunc from 0x0049e9 to 0x001582 (0x3466 bytes) 000121 1776870120.257778 punch from 0x000a1f to 0x001582 (0xb64 bytes) 000122 1776870120.261044 punch from 0x000add to 0x001582 (0xaa6 bytes) 000123 1776870120.262930 punch from 0x000544 to 0x001582 (0x103f bytes) 000124 1776870120.264344 mapwrite 0x002577 thru 0x005e81 (0x390b bytes) 000125 1776870120.277555 mapread 0x00300f thru 0x005e81 (0x2e73 bytes) 000127 1776870120.278869 punch from 0x0031bc to 0x005e81 (0x2cc6 bytes) 000128 1776870120.281070 trunc from 0x005e82 to 0x013670 (0xd7ef bytes) 000130 1776870120.286490 mapread 0x0117c4 thru 0x013670 (0x1ead bytes) 000131 1776870120.289508 mapread 0x00e2cf thru 0x013670 (0x53a2 bytes) 000132 1776870120.291621 MIRROR_RESYNC 4 mirrors 000133 1776870120.501297 read 0x00df63 thru 0x013670 (0x570e bytes) 000134 1776870120.503666 write 0x028448 thru 0x02e436 (0x5fef bytes) 000135 1776870120.516541 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e437): Mirror 3: 0x5a4142e (stale) Mirror 4: 0x5a4142e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e437): Mirror 3: 0x5a4142e (stale) Mirror 4: 0x5a4142e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e437: skipped 000136 1776870120.611207 mapwrite 0x019bca thru 0x01c733 (0x2b6a bytes) 000137 1776870120.625194 MIRROR_RESYNC 3 mirrors 000138 1776870120.860592 punch from 0x01fb6d to 0x02764f (0x7ae3 bytes) 000139 1776870120.870615 trunc from 0x0207c1 to 0x02d5cc (0xce0c bytes) 000140 1776870120.872594 write 0x0288cd thru 0x02d206 (0x493a bytes) 000141 1776870120.875715 punch from 0x01502a to 0x02400f (0xefe6 bytes) 000142 1776870120.877340 mapread 0x008728 thru 0x00e9a6 (0x627f bytes) 000143 1776870120.880700 trunc from 0x00c758 to 0x01354f (0x6df8 bytes) 000144 1776870120.882629 trunc from 0x01a740 to 0x0283e2 (0xdca3 bytes) 000145 1776870120.886530 mapwrite 0x032282 thru 0x03b4cf (0x924e bytes) 000146 1776870120.909790 mapread 0x024d32 thru 0x025cdf (0xfae bytes) 000147 1776870120.913221 mapwrite 0x034bd4 thru 0x03f61f (0xaa4c bytes) 000149 1776870120.926752 punch from 0x006efc to 0x016040 (0xf145 bytes) 000150 1776870120.928408 fallocate 0x00e11e thru 0x011991 (0x3874 bytes) 000151 1776870120.931126 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f620): Mirror 4: 0x49d87fc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f620): Mirror 4: 0x49d87fc4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f620: skipped 000154 1776870121.008356 trunc from 0x010851 to 0x013a69 (0x3219 bytes) 000155 1776870121.009965 read 0x030af0 thru 0x037693 (0x6ba4 bytes) 000156 1776870121.013154 trunc from 0x03f620 to 0x029219 (0x16406 bytes) 000157 1776870121.019466 mapwrite 0x0013a4 thru 0x0107b3 (0xf410 bytes) 000158 1776870121.032257 punch from 0x00c18d to 0x01b1e3 (0xf057 bytes) 000159 1776870121.033779 write 0x03845d thru 0x03d7cb (0x536f bytes) 000160 1776870121.034557 mapread 0x0251d2 thru 0x02bdf0 (0x6c1f bytes) 000161 1776870121.037499 write 0x000e21 thru 0x00e01f (0xd1ff bytes) 000162 1776870121.040000 mapwrite 0x03b1a2 thru 0x03f555 (0x43b4 bytes) 000163 1776870121.051054 trunc from 0x0126d6 to 0x01a113 (0x7a3e bytes) 000164 1776870121.052351 punch from 0x031db0 to 0x03c084 (0xa2d5 bytes) 000165 1776870121.053755 punch from 0x00eec8 to 0x015e3f (0x6f78 bytes) 000166 1776870121.056345 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f556): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f556): Mirror 4: 0x7a91a67a (stale) CRC-32 checksum value for chunk [0, 0x3f556): Mirror 1: 0xf49903ef Mirror 5: 0xf49903ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f556: skipped 000167 1776870121.278848 write 0x007554 thru 0x00a9f0 (0x349d bytes) 000168 1776870121.290014 write 0x031548 thru 0x039fbc (0x8a75 bytes) 000169 1776870121.292891 trunc from 0x03dd15 to 0x03f555 (0x1841 bytes) 000170 1776870121.294534 read 0x01ec77 thru 0x023471 (0x47fb bytes) 000171 1776870121.296930 fallocate 0x038e40 thru 0x03ffff (0x71c0 bytes) 000173 1776870121.299488 mapread 0x029541 thru 0x0358c8 (0xc388 bytes) 000174 1776870121.302584 trunc from 0x040000 to 0x028e62 (0x1719d bytes) 000175 1776870121.315626 mapread 0x01cafd thru 0x027a2e (0xaf32 bytes) 000176 1776870121.319582 write 0x029de6 thru 0x02fd2e (0x5f49 bytes) 000177 1776870121.320461 trunc from 0x02fd2f to 0x039e16 (0xa0e8 bytes) 000178 1776870121.325520 punch from 0x02bb67 to 0x02e0ba (0x2554 bytes) 000179 1776870121.330290 read 0x0129b8 thru 0x01736e (0x49b7 bytes) 000180 1776870121.333635 read 0x02e3ed thru 0x03639d (0x7fb1 bytes) 000181 1776870121.335497 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e17): Mirror 5: 0x92db743c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e17): Mirror 5: 0x92db743c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e17: skipped 000182 1776870121.420944 read 0x01825a thru 0x01fe96 (0x7c3d bytes) 000183 1776870121.424379 trunc from 0x039e17 to 0x02f446 (0xa9d0 bytes) 000184 1776870121.430521 read 0x029b8f thru 0x02f446 (0x58b8 bytes) 000185 1776870121.434694 mapwrite 0x038d8f thru 0x03ffff (0x7271 bytes) 000186 1776870121.451830 mapread 0x003ff5 thru 0x00e10b (0xa117 bytes) 000187 1776870121.453027 read 0x01b7a1 thru 0x01bbb6 (0x416 bytes) 000188 1776870121.453624 punch from 0x0149de to 0x01e00a (0x962d bytes) 000190 1776870121.456382 trunc from 0x010abf to 0x018255 (0x7797 bytes) 000191 1776870121.458671 trunc from 0x040000 to 0x02f7bb (0x10844 bytes) 000192 1776870121.463491 mapwrite 0x00cb28 thru 0x00e267 (0x1740 bytes) 000193 1776870121.467926 mapread 0x0079ab thru 0x011294 (0x98ea bytes) 000195 1776870121.471894 mapread 0x01f360 thru 0x0246e4 (0x5385 bytes) 000196 1776870121.473613 trunc from 0x01669c to 0x025fe8 (0xf94d bytes) 000197 1776870121.475949 write 0x03603d thru 0x03ffff (0x9fc3 bytes) 000198 1776870121.476756 mapread 0x00319c thru 0x009b46 (0x69ab bytes) 000199 1776870121.478652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000201 1776870121.518928 trunc from 0x02a554 to 0x0383f2 (0xde9f bytes) 000202 1776870121.520789 read 0x0347ad thru 0x036b96 (0x23ea bytes) 000203 1776870121.523043 mapread 0x01a6aa thru 0x01dfc5 (0x391c bytes) 000205 1776870121.529728 punch from 0x0087ed to 0x00d37b (0x4b8f bytes) 000207 1776870121.532703 punch from 0x000880 to 0x00669e (0x5e1f bytes) 000208 1776870121.534625 fallocate 0x00e7e1 thru 0x012f32 (0x4752 bytes) 000209 1776870121.536546 read 0x00d892 thru 0x0163ee (0x8b5d bytes) 000210 1776870121.540412 write 0x006de1 thru 0x0157e3 (0xea03 bytes) 000211 1776870121.543131 punch from 0x01bdc2 to 0x026f0a (0xb149 bytes) 000215 1776870121.547956 fallocate 0x02bb0c thru 0x02dbe5 (0x20da bytes) 000216 1776870121.550175 punch from 0x02dee8 to 0x03ae72 (0xcf8b bytes) 000219 1776870121.552755 punch from 0x02af84 to 0x02b50b (0x588 bytes) 000220 1776870121.555605 punch from 0x01d884 to 0x027a69 (0xa1e6 bytes) 000221 1776870121.559424 MIRROR_EXTEND to 2 mirrors 000222 1776870121.771655 fallocate 0x016823 thru 0x024960 (0xe13e bytes) 000223 1776870121.785142 read 0x01dd40 thru 0x01f521 (0x17e2 bytes) 000224 1776870121.787489 trunc from 0x040000 to 0x01a00b (0x25ff4 bytes) 000225 1776870121.802227 mapread 0x008918 thru 0x014c61 (0xc34a bytes) 000226 1776870121.820042 fallocate 0x00c705 thru 0x019172 (0xca6e bytes) 000228 1776870121.822921 mapwrite 0x0174b3 thru 0x026153 (0xeca1 bytes) 000229 1776870121.842227 mapwrite 0x03d486 thru 0x03ffff (0x2b7a bytes) 000230 1776870121.854943 mapwrite 0x0290ac thru 0x03836d (0xf2c2 bytes) 000231 1776870121.872471 MIRROR_RESYNC 2 mirrors 000232 1776870122.089056 punch from 0x037d7b to 0x03ffff (0x8285 bytes) 000235 1776870122.099389 write 0x00ef5a thru 0x019b30 (0xabd7 bytes) 000236 1776870122.103031 trunc from 0x02c660 to 0x02e849 (0x21ea bytes) 000237 1776870122.104730 mapwrite 0x011020 thru 0x014703 (0x36e4 bytes) 000239 1776870122.111491 mapwrite 0x0147b7 thru 0x0181c9 (0x3a13 bytes) 000240 1776870122.117900 write 0x0116ae thru 0x01de1c (0xc76f bytes) 000241 1776870122.122363 punch from 0x02cb81 to 0x03ba46 (0xeec6 bytes) 000242 1776870122.127831 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000243 1776870122.176968 mapread 0x01e932 thru 0x022f9e (0x466d bytes) 000244 1776870122.186676 fallocate 0x021f73 thru 0x0311b1 (0xf23f bytes) 000245 1776870122.189094 read 0x0016fe thru 0x00e585 (0xce88 bytes) 000246 1776870122.192205 write 0x02e3e1 thru 0x034c8d (0x68ad bytes) 000247 1776870122.198064 fallocate 0x02bb99 thru 0x02c124 (0x58c bytes) 000248 1776870122.200982 read 0x0074c5 thru 0x0170aa (0xfbe6 bytes) 000249 1776870122.204076 trunc from 0x025175 to 0x029d9c (0x4c28 bytes) 000250 1776870122.206136 trunc from 0x029361 to 0x032484 (0x9124 bytes) 000251 1776870122.211202 mapwrite 0x00324f thru 0x01081f (0xd5d1 bytes) 000252 1776870122.221651 punch from 0x0142ca to 0x01f7d8 (0xb50f bytes) 000253 1776870122.224281 write 0x037889 thru 0x03d117 (0x588f bytes) 000254 1776870122.227060 MIRROR_EXTEND to 2 mirrors 000255 1776870122.460293 mapwrite 0x005a45 thru 0x013f92 (0xe54e bytes) 000256 1776870122.503014 fallocate 0x00aba5 thru 0x01586e (0xacca bytes) 000257 1776870122.506771 write 0x03a278 thru 0x03ffff (0x5d88 bytes) 000258 1776870122.512695 write 0x01d402 thru 0x024168 (0x6d67 bytes) 000259 1776870122.516886 read 0x00e71a thru 0x0125b9 (0x3ea0 bytes) 000260 1776870122.517796 trunc from 0x040000 to 0x01040d (0x2fbf2 bytes) 000261 1776870122.530992 punch from 0x0018c6 to 0x00b5ef (0x9d2a bytes) 000262 1776870122.533457 mapwrite 0x004592 thru 0x004f92 (0xa01 bytes) 000263 1776870122.550230 trunc from 0x01040e to 0x00fea4 (0x569 bytes) 000264 1776870122.558701 trunc from 0x00bc8b to 0x00fea4 (0x421a bytes) 000266 1776870122.561989 fallocate 0x037222 thru 0x0375c5 (0x3a4 bytes) 000267 1776870122.564368 write 0x0183d5 thru 0x01ee97 (0x6ac3 bytes) 000268 1776870122.567945 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ee98): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee98): Mirror 2: 0x9d303bfd (stale) CRC-32 checksum value for chunk [0, 0x1ee98): Mirror 1: 0x51f4b21e Mirror 3: 0x51f4b21e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee98: skipped 000269 1776870122.800851 trunc from 0x01ee98 to 0x014537 (0xa960 bytes) 000270 1776870122.822920 punch from 0x0027aa to 0x00c8a8 (0xa0ff bytes) 000271 1776870122.825136 mapwrite 0x017938 thru 0x0207a8 (0x8e71 bytes) 000272 1776870122.853269 trunc from 0x0207a9 to 0x01855f (0x8249 bytes) 000273 1776870122.861802 write 0x0259ae thru 0x02d8de (0x7f31 bytes) 000274 1776870122.863175 trunc from 0x02d8df to 0x036623 (0x8d45 bytes) 000275 1776870122.871168 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36624): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36624): Mirror 2: 0xf4dfa1ca (stale) Mirror 3: 0xd699b4cc (stale) CRC-32 checksum value for chunk [0, 0x36624): Mirror 1: 0xa24f22c Mirror 4: 0xa24f22c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36624: skipped 000276 1776870123.116812 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36624): Mirror 2: 0xf4dfa1ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36624): Mirror 2: 0xf4dfa1ca (stale) CRC-32 checksum value for chunk [0, 0x36624): Mirror 1: 0xa24f22c Mirror 4: 0xa24f22c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36624: skipped 000277 1776870123.290555 read 0x019afa thru 0x01e5a8 (0x4aaf bytes) 000278 1776870123.296696 trunc from 0x036624 to 0x0396fa (0x30d7 bytes) 000279 1776870123.314677 mapread 0x02706e thru 0x02d44d (0x63e0 bytes) 000280 1776870123.320324 trunc from 0x0396fb to 0x0199a4 (0x1fd56 bytes) 000282 1776870123.326553 trunc from 0x0199a5 to 0x034911 (0x1af6d bytes) 000284 1776870123.331648 punch from 0x03437f to 0x0345d6 (0x258 bytes) 000285 1776870123.333389 trunc from 0x034912 to 0x038cff (0x43ee bytes) 000286 1776870123.339696 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38d00): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d00): Mirror 2: 0x2e925b49 (stale) Mirror 4: 0xea909b49 (stale) CRC-32 checksum value for chunk [0, 0x38d00): Mirror 1: 0xc822b139 Mirror 5: 0xc822b139 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d00: skipped 000287 1776870123.580984 mapwrite 0x026b2b thru 0x02e837 (0x7d0d bytes) 000288 1776870123.632459 MIRROR_RESYNC 4 mirrors 000289 1776870123.947397 read 0x026223 thru 0x0292e8 (0x30c6 bytes) 000290 1776870123.950554 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000291 1776870124.070032 trunc from 0x038d00 to 0x03fe35 (0x7136 bytes) 000292 1776870124.084831 mapwrite 0x02ec4a thru 0x02ed3b (0xf2 bytes) 000293 1776870124.090850 write 0x01ab74 thru 0x024f8b (0xa418 bytes) 000294 1776870124.094352 trunc from 0x03fe36 to 0x02bbb4 (0x14281 bytes) 000295 1776870124.100066 read 0x00159a thru 0x00d942 (0xc3a9 bytes) 000296 1776870124.103297 mapwrite 0x0327b6 thru 0x03d56c (0xadb7 bytes) 000297 1776870124.119979 fallocate 0x03abac thru 0x03bf27 (0x137c bytes) 000298 1776870124.122040 fallocate 0x03a322 thru 0x03ffff (0x5cde bytes) 000299 1776870124.123770 write 0x029079 thru 0x02e5f5 (0x557d bytes) 000300 1776870124.126618 read 0x00d03a thru 0x01b2b6 (0xe27d bytes) 000301 1776870124.129312 trunc from 0x000d3e to 0x00b6a1 (0xa964 bytes) 000302 1776870124.131155 write 0x0347bc thru 0x0349af (0x1f4 bytes) 000303 1776870124.132003 trunc from 0x03f4ae to 0x03ffff (0xb52 bytes) 000304 1776870124.133850 punch from 0x0110ad to 0x01fedf (0xee33 bytes) 000305 1776870124.135587 trunc from 0x040000 to 0x02a305 (0x15cfa bytes) 000306 1776870124.139830 punch from 0x01c4bf to 0x02a305 (0xde47 bytes) 000307 1776870124.142898 trunc from 0x0181c8 to 0x01a8db (0x2714 bytes) 000308 1776870124.144307 trunc from 0x00406c to 0x00539f (0x1334 bytes) 000309 1776870124.145705 punch from 0x0117a5 to 0x016243 (0x4a9f bytes) 000310 1776870124.147258 trunc from 0x0008ec to 0x008b4e (0x8263 bytes) 000311 1776870124.148853 trunc from 0x02a306 to 0x00a602 (0x1fd03 bytes) 000312 1776870124.153401 trunc from 0x00a603 to 0x01e560 (0x13f5e bytes) 000314 1776870124.157698 trunc from 0x01e561 to 0x0083ab (0x161b5 bytes) 000315 1776870124.162538 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x83ac): Mirror 5: 0x5a3a14b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83ac): Mirror 5: 0x5a3a14b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83ac: skipped 000316 1776870124.231466 MIRROR_RESYNC 2 mirrors 000317 1776870124.379065 fallocate 0x00b70f thru 0x014456 (0x8d48 bytes) 000318 1776870124.390360 fallocate 0x006a39 thru 0x01088d (0x9e55 bytes) 000319 1776870124.398325 MIRROR_RESYNC 2 mirrors 000320 1776870124.563102 mapread 0x000424 thru 0x0083ab (0x7f88 bytes) 000321 1776870124.577995 mapread 0x00663b thru 0x007e9a (0x1860 bytes) 000322 1776870124.578660 read 0x002be7 thru 0x003e00 (0x121a bytes) 000323 1776870124.578995 read 0x00176e thru 0x0083ab (0x6c3e bytes) 000324 1776870124.579332 write 0x032751 thru 0x037887 (0x5137 bytes) 000325 1776870124.591012 mapread 0x0040cb thru 0x012920 (0xe856 bytes) 000326 1776870124.598984 trunc from 0x037888 to 0x01305f (0x24828 bytes) 000327 1776870124.616803 trunc from 0x003710 to 0x0099d1 (0x62c2 bytes) 000328 1776870124.618782 write 0x03adb7 thru 0x03ffff (0x5249 bytes) 000329 1776870124.619677 trunc from 0x007a8b to 0x00a9df (0x2f55 bytes) 000330 1776870124.623653 trunc from 0x02aeba to 0x02f906 (0x4a4d bytes) 000331 1776870124.627994 punch from 0x03d720 to 0x03f441 (0x1d22 bytes) 000333 1776870124.638276 fallocate 0x03a43c thru 0x03ffff (0x5bc4 bytes) 000334 1776870124.641728 mapread 0x0094b4 thru 0x00cf95 (0x3ae2 bytes) 000335 1776870124.652420 mapwrite 0x022ee0 thru 0x030b64 (0xdc85 bytes) 000336 1776870124.687298 fallocate 0x02f986 thru 0x03cde0 (0xd45b bytes) 000337 1776870124.689593 write 0x03d89a thru 0x03ffff (0x2766 bytes) 000338 1776870124.691704 MIRROR_RESYNC 2 mirrors 000339 1776870124.886353 read 0x01c04a thru 0x024a28 (0x89df bytes) 000340 1776870124.889707 fallocate 0x03fc4e thru 0x03ffff (0x3b2 bytes) 000341 1776870124.901002 mapread 0x033a79 thru 0x03d67d (0x9c05 bytes) 000342 1776870124.908775 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000344 1776870124.944529 trunc from 0x040000 to 0x033b6c (0xc493 bytes) 000346 1776870124.955146 trunc from 0x02c7eb to 0x033b6c (0x7382 bytes) 000348 1776870124.957796 trunc from 0x01acc0 to 0x01cdc3 (0x2104 bytes) 000349 1776870124.959485 write 0x01b455 thru 0x02ac3a (0xf7e6 bytes) 000350 1776870124.963277 trunc from 0x0211aa to 0x025edc (0x4d33 bytes) 000351 1776870124.966909 fallocate 0x0307a7 thru 0x03897b (0x81d5 bytes) 000352 1776870124.968405 fallocate 0x00508d thru 0x013e16 (0xed8a bytes) 000353 1776870124.969804 trunc from 0x03897c to 0x003b9c (0x34ddf bytes) 000356 1776870124.974697 mapwrite 0x03a2cd thru 0x03ffff (0x5d33 bytes) 000357 1776870124.985852 fallocate 0x004c8e thru 0x01033d (0xb6b0 bytes) 000359 1776870124.987997 trunc from 0x040000 to 0x03f18a (0xe75 bytes) 000360 1776870124.992137 fallocate 0x00327d thru 0x012644 (0xf3c8 bytes) 000361 1776870124.993896 write 0x00b087 thru 0x0175af (0xc529 bytes) 000363 1776870124.999042 mapwrite 0x0159ea thru 0x024913 (0xef2a bytes) 000365 1776870125.012749 read 0x024bf2 thru 0x02e2dd (0x96ec bytes) 000366 1776870125.015769 read 0x01fe11 thru 0x028c2b (0x8e1b bytes) 000367 1776870125.016350 mapread 0x01c8c5 thru 0x029dc6 (0xd502 bytes) 000368 1776870125.017461 mapwrite 0x013890 thru 0x01f339 (0xbaaa bytes) 000370 1776870125.025466 mapwrite 0x030676 thru 0x034eb9 (0x4844 bytes) 000371 1776870125.032999 trunc from 0x03f18b to 0x0390e4 (0x60a6 bytes) 000372 1776870125.037221 fallocate 0x02b10a thru 0x030dac (0x5ca3 bytes) 000373 1776870125.039002 mapwrite 0x01bb7a thru 0x02a53d (0xe9c4 bytes) 000374 1776870125.047132 fallocate 0x02b38b thru 0x032456 (0x70cc bytes) 000375 1776870125.049096 write 0x02edb4 thru 0x03ccd4 (0xdf21 bytes) 000376 1776870125.050594 mapread 0x0012fe thru 0x001632 (0x335 bytes) 000377 1776870125.051533 trunc from 0x015734 to 0x01bd3c (0x6609 bytes) 000378 1776870125.053026 mapread 0x00215b thru 0x00c385 (0xa22b bytes) 000379 1776870125.053818 trunc from 0x03ccd5 to 0x03943d (0x3897 bytes) 000380 1776870125.057803 trunc from 0x02fb5d to 0x03943d (0x98e1 bytes) 000381 1776870125.061291 mapwrite 0x01b721 thru 0x021b99 (0x6479 bytes) 000382 1776870125.067264 fallocate 0x019968 thru 0x01c81a (0x2eb3 bytes) 000383 1776870125.069007 fallocate 0x01701b thru 0x0176f4 (0x6da bytes) 000384 1776870125.070399 write 0x0022ed thru 0x004a92 (0x27a6 bytes) 000385 1776870125.071009 trunc from 0x02ec0d to 0x03943d (0xa831 bytes) 000386 1776870125.072546 trunc from 0x0188e7 to 0x020137 (0x7851 bytes) 000387 1776870125.074043 fallocate 0x03b9de thru 0x03ff0d (0x4530 bytes) 000388 1776870125.075541 punch from 0x011da4 to 0x01a547 (0x87a4 bytes) 000390 1776870125.077168 read 0x03e913 thru 0x03ff0d (0x15fb bytes) 000391 1776870125.080089 mapread 0x008cbe thru 0x014382 (0xb6c5 bytes) 000392 1776870125.081266 punch from 0x002a71 to 0x00aee4 (0x8474 bytes) 000394 1776870125.084967 mapread 0x03891c thru 0x038e50 (0x535 bytes) 000395 1776870125.085536 mapread 0x028b23 thru 0x02abf0 (0x20ce bytes) 000397 1776870125.086523 trunc from 0x001a14 to 0x00f7ae (0xdd9b bytes) 000398 1776870125.088186 fallocate 0x004e6c thru 0x0091cc (0x4361 bytes) 000399 1776870125.089984 read 0x03565f thru 0x03fb64 (0xa506 bytes) 000400 1776870125.090416 trunc from 0x03ff0e to 0x015977 (0x2a596 bytes) 000401 1776870125.095124 read 0x012c13 thru 0x015977 (0x2d65 bytes) 000402 1776870125.095773 trunc from 0x00eb7a to 0x015977 (0x6dfe bytes) 000403 1776870125.097653 trunc from 0x015978 to 0x01ba27 (0x60b0 bytes) 000404 1776870125.102460 trunc from 0x01ba28 to 0x00a578 (0x114af bytes) 000405 1776870125.107129 fallocate 0x00be2c thru 0x00f403 (0x35d8 bytes) 000406 1776870125.109195 read 0x00d208 thru 0x00f403 (0x21fc bytes) 000407 1776870125.113224 read 0x00c336 thru 0x00e32b (0x1ff6 bytes) 000408 1776870125.113915 read 0x00b97f thru 0x00f403 (0x3a85 bytes) 000409 1776870125.114488 fallocate 0x03125a thru 0x03f989 (0xe730 bytes) 000410 1776870125.116974 punch from 0x009e7a to 0x00f403 (0x558a bytes) 000411 1776870125.119013 trunc from 0x0087ad to 0x00f403 (0x6c57 bytes) 000412 1776870125.121005 punch from 0x0079e7 to 0x00ca08 (0x5022 bytes) 000413 1776870125.122968 mapwrite 0x033b0c thru 0x03812a (0x461f bytes) 000414 1776870125.141320 trunc from 0x010aa9 to 0x010f03 (0x45b bytes) 000415 1776870125.143558 write 0x02a63f thru 0x034347 (0x9d09 bytes) 000416 1776870125.144681 trunc from 0x014b52 to 0x018383 (0x3832 bytes) 000417 1776870125.146843 write 0x018162 thru 0x01a698 (0x2537 bytes) 000418 1776870125.148979 write 0x01c20f thru 0x01f7cc (0x35be bytes) 000419 1776870125.149684 mapwrite 0x01bd07 thru 0x0282f7 (0xc5f1 bytes) 000420 1776870125.158337 read 0x00e828 thru 0x014dbd (0x6596 bytes) 000422 1776870125.162194 trunc from 0x0149d2 to 0x022a93 (0xe0c2 bytes) 000423 1776870125.164362 write 0x032ba4 thru 0x035aba (0x2f17 bytes) 000425 1776870125.165604 write 0x012baa thru 0x018dde (0x6235 bytes) 000426 1776870125.168382 fallocate 0x0276d7 thru 0x02c86d (0x5197 bytes) 000427 1776870125.170363 read 0x0060b8 thru 0x00a7d7 (0x4720 bytes) 000429 1776870125.171158 mapread 0x01c201 thru 0x025a4e (0x984e bytes) 000430 1776870125.175417 mapwrite 0x029ce2 thru 0x02e9c6 (0x4ce5 bytes) 000431 1776870125.182253 fallocate 0x021629 thru 0x02e439 (0xce11 bytes) 000432 1776870125.184633 mapread 0x02c445 thru 0x03812a (0xbce6 bytes) 000433 1776870125.185769 mapread 0x019ac1 thru 0x02468f (0xabcf bytes) 000434 1776870125.186792 punch from 0x0102c2 to 0x019a0a (0x9749 bytes) 000436 1776870125.189713 write 0x008dcb thru 0x017852 (0xea88 bytes) 000437 1776870125.192346 mapread 0x017fa2 thru 0x01da65 (0x5ac4 bytes) 000439 1776870125.196094 read 0x0215c7 thru 0x02b5d0 (0xa00a bytes) 000440 1776870125.196754 mapread 0x00e638 thru 0x011dad (0x3776 bytes) 000441 1776870125.197661 read 0x02748b thru 0x029f90 (0x2b06 bytes) 000442 1776870125.198246 fallocate 0x036a22 thru 0x03ffff (0x95de bytes) 000443 1776870125.200619 trunc from 0x03812b to 0x03e03e (0x5f14 bytes) 000444 1776870125.205144 read 0x01771d thru 0x022156 (0xaa3a bytes) 000445 1776870125.205826 trunc from 0x03e03f to 0x02241c (0x1bc22 bytes) 000446 1776870125.210837 mapread 0x0064ce thru 0x010220 (0x9d53 bytes) 000447 1776870125.211718 mapread 0x0122a6 thru 0x0151d5 (0x2f30 bytes) 000449 1776870125.212774 mapwrite 0x02479e thru 0x02b60e (0x6e71 bytes) 000450 1776870125.229198 punch from 0x00bb2f to 0x015900 (0x9dd2 bytes) 000451 1776870125.231377 fallocate 0x0208be thru 0x02098f (0xd2 bytes) 000452 1776870125.233382 mapwrite 0x032160 thru 0x037704 (0x55a5 bytes) 000453 1776870125.255958 mapread 0x021ad7 thru 0x025382 (0x38ac bytes) 000455 1776870125.257687 punch from 0x01760b to 0x0255f7 (0xdfed bytes) 000456 1776870125.260300 mapread 0x0297f6 thru 0x037704 (0xdf0f bytes) 000457 1776870125.261301 mapwrite 0x03d4c7 thru 0x03ffff (0x2b39 bytes) 000458 1776870125.284943 mapread 0x00da2e thru 0x01cf95 (0xf568 bytes) 000459 1776870125.305811 trunc from 0x010a0c to 0x01d41d (0xca12 bytes) 000461 1776870125.308217 write 0x03e8d3 thru 0x03ffff (0x172d bytes) 000462 1776870125.308841 write 0x00530a thru 0x005e4c (0xb43 bytes) 000464 1776870125.310076 read 0x03b931 thru 0x03ffff (0x46cf bytes) 000465 1776870125.310597 mapread 0x0173a6 thru 0x018af1 (0x174c bytes) 000466 1776870125.314031 write 0x0374e7 thru 0x03ffff (0x8b19 bytes) 000469 1776870125.316167 fallocate 0x001f3d thru 0x00b2a8 (0x936c bytes) 000470 1776870125.318003 read 0x008530 thru 0x012d35 (0xa806 bytes) 000471 1776870125.318529 fallocate 0x013229 thru 0x02244e (0xf226 bytes) 000472 1776870125.320478 mapread 0x001fe0 thru 0x005f7b (0x3f9c bytes) 000473 1776870125.321421 mapread 0x00f124 thru 0x013121 (0x3ffe bytes) 000474 1776870125.322359 trunc from 0x040000 to 0x03d86c (0x2793 bytes) 000475 1776870125.327680 read 0x011154 thru 0x01a001 (0x8eae bytes) 000476 1776870125.328217 fallocate 0x004030 thru 0x01193d (0xd90e bytes) 000477 1776870125.330059 trunc from 0x03d86d to 0x020b6d (0x1ccff bytes) 000478 1776870125.334349 trunc from 0x00382e to 0x00e0ff (0xa8d2 bytes) 000480 1776870125.338383 trunc from 0x011a0c to 0x01691d (0x4f12 bytes) 000481 1776870125.339851 read 0x01aeed thru 0x020b6d (0x5c81 bytes) 000482 1776870125.340309 mapread 0x01b72c thru 0x01ffae (0x4883 bytes) 000483 1776870125.341017 punch from 0x011510 to 0x01afbc (0x9aad bytes) 000484 1776870125.342511 punch from 0x00bc75 to 0x01bc1a (0xffa6 bytes) 000485 1776870125.344162 write 0x03b6e7 thru 0x03ffff (0x4919 bytes) 000486 1776870125.345274 mapread 0x02bc0b thru 0x02e2bb (0x26b1 bytes) 000487 1776870125.349561 write 0x02afc9 thru 0x035b98 (0xabd0 bytes) 000489 1776870125.351542 mapread 0x01a575 thru 0x020b20 (0x65ac bytes) 000490 1776870125.352531 trunc from 0x01ffd7 to 0x02e8ac (0xe8d6 bytes) 000491 1776870125.357232 read 0x028f22 thru 0x02e1e5 (0x52c4 bytes) 000492 1776870125.360393 mapread 0x00974a thru 0x015998 (0xc24f bytes) 000493 1776870125.361207 write 0x03f691 thru 0x03ffff (0x96f bytes) 000494 1776870125.361856 punch from 0x00b90f to 0x01613a (0xa82c bytes) 000495 1776870125.363898 mapwrite 0x015c5e thru 0x01d626 (0x79c9 bytes) 000496 1776870125.374042 mapwrite 0x025d9e thru 0x02a28a (0x44ed bytes) 000497 1776870125.380716 trunc from 0x040000 to 0x033671 (0xc98e bytes) 000498 1776870125.385321 write 0x02097e thru 0x02396d (0x2ff0 bytes) 000499 1776870125.386104 mapwrite 0x02bfb6 thru 0x033dc7 (0x7e12 bytes) 000500 1776870125.399785 punch from 0x009641 to 0x016a27 (0xd3e7 bytes) 000501 1776870125.401787 mapread 0x01e4c7 thru 0x027cfd (0x9837 bytes) 000502 1776870125.402812 mapwrite 0x0345d1 thru 0x038c6d (0x469d bytes) 000503 1776870125.424021 trunc from 0x038c6e to 0x00fde4 (0x28e89 bytes) 000504 1776870125.428626 punch from 0x00bf66 to 0x00fde4 (0x3e7f bytes) 000505 1776870125.430129 mapwrite 0x03752f thru 0x03bb94 (0x4666 bytes) 000506 1776870125.451281 write 0x03658a thru 0x03a1cd (0x3c44 bytes) 000509 1776870125.454228 trunc from 0x03bb95 to 0x022421 (0x19773 bytes) 000510 1776870125.459182 write 0x025019 thru 0x029190 (0x4178 bytes) 000511 1776870125.460556 write 0x00eb49 thru 0x00edd4 (0x28c bytes) 000512 1776870125.462822 fallocate 0x01d7c5 thru 0x02cb02 (0xf33e bytes) 000513 1776870125.465342 fallocate 0x01e03a thru 0x027053 (0x901a bytes) 000514 1776870125.467597 punch from 0x01ebec to 0x027f4b (0x9360 bytes) 000515 1776870125.473266 punch from 0x01ebb3 to 0x022cb9 (0x4107 bytes) 000518 1776870125.476846 trunc from 0x02cb03 to 0x036552 (0x9a50 bytes) 000519 1776870125.482582 MIRROR_EXTEND to 2 mirrors 000520 1776870125.663892 MIRROR_RESYNC 2 mirrors 000521 1776870125.758886 mapread 0x02bbb8 thru 0x036552 (0xa99b bytes) 000522 1776870125.765194 read 0x02b7da thru 0x036552 (0xad79 bytes) 000523 1776870125.765569 trunc from 0x03618a to 0x036552 (0x3c9 bytes) 000524 1776870125.774180 read 0x00aee6 thru 0x010a73 (0x5b8e bytes) 000526 1776870125.777959 trunc from 0x036553 to 0x00cc5d (0x298f5 bytes) 000527 1776870125.788530 fallocate 0x0369a1 thru 0x037e25 (0x1485 bytes) 000528 1776870125.791161 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xcc5e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcc5e): Mirror 2: 0xc031dea1 (stale) CRC-32 checksum value for chunk [0, 0xcc5e): Mirror 1: 0xc031dea1 Mirror 3: 0xc031dea1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcc5e: skipped 000529 1776870126.006143 fallocate 0x0349e9 thru 0x035c34 (0x124c bytes) 000530 1776870126.020983 mapread 0x000600 thru 0x003acf (0x34d0 bytes) 000531 1776870126.028785 mapwrite 0x0080fd thru 0x0174c6 (0xf3ca bytes) 000532 1776870126.040378 trunc from 0x01aef7 to 0x0220fd (0x7207 bytes) 000533 1776870126.043250 mapwrite 0x03bd4f thru 0x03ffff (0x42b1 bytes) 000534 1776870126.062855 trunc from 0x009aad to 0x00a3e9 (0x93d bytes) 000535 1776870126.064488 trunc from 0x040000 to 0x01e958 (0x216a7 bytes) 000536 1776870126.071174 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e959): Mirror 2: 0xd24f362b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e959): Mirror 2: 0xd24f362b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e959: skipped 000537 1776870126.160061 fallocate 0x0004be thru 0x00c1b7 (0xbcfa bytes) 000538 1776870126.162122 read 0x002a12 thru 0x0116db (0xecca bytes) 000540 1776870126.166253 punch from 0x00f147 to 0x01310e (0x3fc8 bytes) 000541 1776870126.168446 read 0x002ed4 thru 0x00f1fb (0xc328 bytes) 000542 1776870126.172452 fallocate 0x02f436 thru 0x03bfaf (0xcb7a bytes) 000543 1776870126.174737 punch from 0x029842 to 0x02d169 (0x3928 bytes) 000544 1776870126.176508 mapread 0x00a338 thru 0x0140de (0x9da7 bytes) 000545 1776870126.177294 read 0x0045aa thru 0x0078af (0x3306 bytes) 000546 1776870126.177749 mapread 0x01116e thru 0x017d44 (0x6bd7 bytes) 000547 1776870126.178591 punch from 0x00ac6f to 0x00da5f (0x2df1 bytes) 000548 1776870126.181365 fallocate 0x037867 thru 0x03ffff (0x8799 bytes) 000549 1776870126.183566 punch from 0x0095db to 0x00a26a (0xc90 bytes) 000550 1776870126.185329 punch from 0x005b01 to 0x00b6ec (0x5bec bytes) 000552 1776870126.187769 mapread 0x025062 thru 0x0347e8 (0xf787 bytes) 000554 1776870126.192957 trunc from 0x040000 to 0x015add (0x2a522 bytes) 000555 1776870126.201356 MIRROR_RESYNC 2 mirrors 000556 1776870126.405309 read 0x00e415 thru 0x015add (0x76c9 bytes) 000557 1776870126.407769 read 0x0012fb thru 0x00ab94 (0x989a bytes) 000558 1776870126.409979 fallocate 0x015c8b thru 0x023305 (0xd67b bytes) 000559 1776870126.419486 mapread 0x00a366 thru 0x010108 (0x5da3 bytes) 000560 1776870126.428876 mapwrite 0x02a3a4 thru 0x032ea7 (0x8b04 bytes) 000561 1776870126.447217 mapwrite 0x014056 thru 0x021a43 (0xd9ee bytes) 000562 1776870126.457499 read 0x02c9b8 thru 0x032ea7 (0x64f0 bytes) 000563 1776870126.458157 mapread 0x01d38c thru 0x02c46b (0xf0e0 bytes) 000564 1776870126.459379 trunc from 0x032ea8 to 0x0093d3 (0x29ad4 bytes) 000565 1776870126.463936 trunc from 0x0093d4 to 0x00f0df (0x5d0c bytes) 000566 1776870126.468110 mapread 0x000fe6 thru 0x00f0df (0xe0fa bytes) 000567 1776870126.471596 punch from 0x003664 to 0x009ed3 (0x6870 bytes) 000569 1776870126.473764 mapwrite 0x02057b thru 0x022b14 (0x259a bytes) 000570 1776870126.485312 trunc from 0x0002a0 to 0x001f35 (0x1c96 bytes) 000571 1776870126.487213 read 0x0008a4 thru 0x0087ad (0x7f0a bytes) 000572 1776870126.490472 read 0x00279e thru 0x00295c (0x1bf bytes) 000573 1776870126.490929 mapwrite 0x015883 thru 0x01a646 (0x4dc4 bytes) 000574 1776870126.497228 trunc from 0x022b15 to 0x01832d (0xa7e7 bytes) 000575 1776870126.503172 fallocate 0x004797 thru 0x007869 (0x30d3 bytes) 000576 1776870126.504997 fallocate 0x00fe0b thru 0x01a83a (0xaa30 bytes) 000577 1776870126.506903 trunc from 0x00bb1c to 0x00f7d1 (0x3cb6 bytes) 000578 1776870126.508806 write 0x0335f0 thru 0x037437 (0x3e48 bytes) 000581 1776870126.512494 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37438): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37438): Mirror 2: 0x83117dd1 (stale) CRC-32 checksum value for chunk [0, 0x37438): Mirror 1: 0x17a44ac2 Mirror 3: 0x17a44ac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37438: skipped 000582 1776870126.733600 write 0x0111e3 thru 0x01641f (0x523d bytes) 000584 1776870126.748861 trunc from 0x0132be to 0x01f9ce (0xc711 bytes) 000585 1776870126.753792 mapread 0x034691 thru 0x036c31 (0x25a1 bytes) 000586 1776870126.756835 mapread 0x024bfa thru 0x02b32e (0x6735 bytes) 000587 1776870126.761609 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37438): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37438): Mirror 2: 0x83117dd1 (stale) Mirror 3: 0x17a44ac2 (stale) CRC-32 checksum value for chunk [0, 0x37438): Mirror 1: 0x2505e214 Mirror 4: 0x2505e214 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37438: skipped 000588 1776870127.005898 mapwrite 0x016094 thru 0x024bc9 (0xeb36 bytes) 000589 1776870127.050579 fallocate 0x029305 thru 0x032367 (0x9063 bytes) 000590 1776870127.052660 mapread 0x034192 thru 0x03592e (0x179d bytes) 000591 1776870127.057041 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37438): Mirror 3: 0x17a44ac2 (stale) Mirror 4: 0x2505e214 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37438): Mirror 3: 0x17a44ac2 (stale) Mirror 4: 0x2505e214 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37438: skipped 000592 1776870127.159189 mapwrite 0x0376f9 thru 0x03ffff (0x8907 bytes) 000593 1776870127.181026 punch from 0x0351c1 to 0x03a9e9 (0x5829 bytes) 000594 1776870127.183107 read 0x025004 thru 0x03321c (0xe219 bytes) 000595 1776870127.186326 mapwrite 0x01956d thru 0x027d08 (0xe79c bytes) 000596 1776870127.197977 trunc from 0x040000 to 0x015049 (0x2afb6 bytes) 000597 1776870127.202314 read 0x00c5c8 thru 0x0132ee (0x6d27 bytes) 000598 1776870127.206509 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1504a): Mirror 4: 0xd1f59a1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1504a): Mirror 4: 0xd1f59a1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1504a: skipped 000599 1776870127.275281 trunc from 0x000da8 to 0x00acec (0x9f45 bytes) 000600 1776870127.279552 mapwrite 0x015d4f thru 0x022789 (0xca3b bytes) 000601 1776870127.301568 mapread 0x014fbb thru 0x01c509 (0x754f bytes) 000603 1776870127.302666 fallocate 0x01bb11 thru 0x024e93 (0x9383 bytes) 000604 1776870127.304271 trunc from 0x002c85 to 0x005d89 (0x3105 bytes) 000605 1776870127.305779 trunc from 0x01592f to 0x01f850 (0x9f22 bytes) 000606 1776870127.308558 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000607 1776870127.342716 fallocate 0x0088c9 thru 0x00b476 (0x2bae bytes) 000608 1776870127.344966 read 0x01a6fd thru 0x022789 (0x808d bytes) 000609 1776870127.348965 mapwrite 0x00e76d thru 0x01a0be (0xb952 bytes) 000610 1776870127.365562 mapwrite 0x032f60 thru 0x0337af (0x850 bytes) 000611 1776870127.385642 fallocate 0x00506e thru 0x00d474 (0x8407 bytes) 000612 1776870127.388048 write 0x039627 thru 0x03983d (0x217 bytes) 000613 1776870127.388831 trunc from 0x01be66 to 0x0231cb (0x7366 bytes) 000615 1776870127.391986 punch from 0x030180 to 0x03983d (0x96be bytes) 000616 1776870127.395811 read 0x014081 thru 0x01d10f (0x908f bytes) 000617 1776870127.399514 fallocate 0x028bd6 thru 0x0291ee (0x619 bytes) 000618 1776870127.401714 mapwrite 0x0233a6 thru 0x02a8ab (0x7506 bytes) 000619 1776870127.410094 fallocate 0x015077 thru 0x017cf3 (0x2c7d bytes) 000620 1776870127.412113 trunc from 0x03983e to 0x01c1c7 (0x1d676 bytes) 000621 1776870127.418368 MIRROR_EXTEND to 2 mirrors 000622 1776870127.590406 mapwrite 0x01b070 thru 0x023b78 (0x8b09 bytes) 000623 1776870127.625626 mapread 0x01c7a4 thru 0x020042 (0x389f bytes) 000624 1776870127.626328 mapread 0x000e32 thru 0x005296 (0x4465 bytes) 000625 1776870127.631894 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23b79): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b79): Mirror 2: 0x56b8f95f (stale) CRC-32 checksum value for chunk [0, 0x23b79): Mirror 1: 0x43cfd0f2 Mirror 3: 0x43cfd0f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b79: skipped 000626 1776870127.907132 mapread 0x01102b thru 0x0134eb (0x24c1 bytes) 000627 1776870127.920152 trunc from 0x0047bc to 0x00e09a (0x98df bytes) 000628 1776870127.938423 trunc from 0x023b79 to 0x0088d4 (0x1b2a4 bytes) 000629 1776870127.962670 write 0x0292bf thru 0x02f740 (0x6482 bytes) 000630 1776870127.964503 write 0x0189f6 thru 0x01ea16 (0x6021 bytes) 000632 1776870127.969729 trunc from 0x02f741 to 0x038a07 (0x92c7 bytes) 000633 1776870127.979389 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38a08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a08): Mirror 2: 0x800c88be (stale) Mirror 3: 0xe9753619 (stale) CRC-32 checksum value for chunk [0, 0x38a08): Mirror 1: 0xc3222106 Mirror 4: 0xc3222106 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a08: skipped 000634 1776870128.275975 fallocate 0x02e05a thru 0x037530 (0x94d7 bytes) 000636 1776870128.290746 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a08): Mirror 3: 0xe9753619 (stale) Mirror 4: 0xc3222106 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a08): Mirror 3: 0xe9753619 (stale) Mirror 4: 0xc3222106 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a08: skipped 000637 1776870128.406324 punch from 0x025198 to 0x02e46b (0x92d4 bytes) 000638 1776870128.409919 punch from 0x019f5d to 0x0207cf (0x6873 bytes) 000639 1776870128.412095 mapread 0x034402 thru 0x038a07 (0x4606 bytes) 000640 1776870128.418135 trunc from 0x031c3c to 0x035535 (0x38fa bytes) 000641 1776870128.420229 fallocate 0x0269e7 thru 0x02f437 (0x8a51 bytes) 000642 1776870128.422282 fallocate 0x03e642 thru 0x03ffff (0x19be bytes) 000643 1776870128.423951 trunc from 0x038a08 to 0x0037da (0x3522d bytes) 000644 1776870128.431223 MIRROR_RESYNC 3 mirrors 000645 1776870128.630792 punch from 0x0007b1 to 0x0037da (0x302a bytes) 000646 1776870128.639581 mapread 0x002963 thru 0x0037da (0xe78 bytes) 000647 1776870128.643139 mapread 0x003278 thru 0x0037da (0x563 bytes) 000648 1776870128.643973 trunc from 0x0037db to 0x03e4d3 (0x3acf9 bytes) 000649 1776870128.650573 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 3: 0x3b690de6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 3: 0x3b690de6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d4: skipped 000650 1776870128.736567 MIRROR_RESYNC 2 mirrors 000651 1776870128.907714 MIRROR_EXTEND to 3 mirrors 000652 1776870129.125751 MIRROR_RESYNC 3 mirrors 000653 1776870129.301602 mapread 0x03130c thru 0x0325f7 (0x12ec bytes) 000654 1776870129.309780 punch from 0x033790 to 0x03e4d3 (0xad44 bytes) 000655 1776870129.322119 read 0x02bb3c thru 0x038fb6 (0xd47b bytes) 000657 1776870129.329574 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 4: 0x3b690de6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 4: 0x3b690de6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d4: skipped 000658 1776870129.437570 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 4: 0x3b690de6 (stale) CRC-32 checksum value for chunk [0, 0x3e4d4): Mirror 1: 0x3b690de6 Mirror 5: 0x3b690de6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d4: skipped 000659 1776870129.713890 fallocate 0x0058ad thru 0x00e927 (0x907b bytes) 000660 1776870129.723804 trunc from 0x03e4d4 to 0x005087 (0x3944c bytes) 000661 1776870129.730416 fallocate 0x02f4cc thru 0x0387b0 (0x92e5 bytes) 000663 1776870129.733759 punch from 0x00fc07 to 0x0132c5 (0x36bf bytes) 000665 1776870129.736161 fallocate 0x0384e8 thru 0x03e375 (0x5e8e bytes) 000666 1776870129.738481 fallocate 0x024d0c thru 0x02b727 (0x6a1c bytes) 000667 1776870129.742707 MIRROR_RESYNC 3 mirrors 000668 1776870129.963804 fallocate 0x03f274 thru 0x03ffff (0xd8c bytes) 000669 1776870129.982561 trunc from 0x03e376 to 0x0066e3 (0x37c92 bytes) 000670 1776870129.996335 mapread 0x00050e thru 0x0066e3 (0x61d6 bytes) 000672 1776870130.005212 read 0x00411e thru 0x0066a8 (0x258b bytes) 000673 1776870130.006033 punch from 0x0054cb to 0x0066e3 (0x1219 bytes) 000674 1776870130.007737 mapread 0x004eb7 thru 0x0066e3 (0x182d bytes) 000676 1776870130.010830 mapwrite 0x02425b thru 0x031bd0 (0xd976 bytes) 000677 1776870130.030255 trunc from 0x030f26 to 0x031bd0 (0xcab bytes) 000678 1776870130.032180 read 0x01ad87 thru 0x02a2f8 (0xf572 bytes) 000679 1776870130.042227 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31bd1): Mirror 4: 0x5f4ee17c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31bd1): Mirror 4: 0x5f4ee17c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31bd1: skipped 000680 1776870130.159867 MIRROR_RESYNC 2 mirrors 000681 1776870130.325108 mapread 0x02f268 thru 0x031bd0 (0x2969 bytes) 000682 1776870130.332991 MIRROR_RESYNC 2 mirrors 000683 1776870130.447143 mapwrite 0x0325f5 thru 0x033ea0 (0x18ac bytes) 000684 1776870130.472661 punch from 0x00b90f to 0x00f4e6 (0x3bd8 bytes) 000685 1776870130.475028 read 0x02de4b thru 0x033ea0 (0x6056 bytes) 000687 1776870130.482290 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33ea1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ea1): Mirror 4: 0x879aef54 (stale) CRC-32 checksum value for chunk [0, 0x33ea1): Mirror 1: 0xfbbd2d62 Mirror 5: 0xfbbd2d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ea1: skipped 000688 1776870130.720591 write 0x01ef70 thru 0x0229cd (0x3a5e bytes) 000689 1776870130.738865 mapwrite 0x0333bd thru 0x03dfa7 (0xabeb bytes) 000690 1776870130.767573 trunc from 0x03dfa8 to 0x03b491 (0x2b16 bytes) 000692 1776870130.773212 fallocate 0x00a5da thru 0x00db10 (0x3537 bytes) 000693 1776870130.775534 mapread 0x007e3a thru 0x007ec1 (0x88 bytes) 000694 1776870130.780940 MIRROR_RESYNC 3 mirrors 000695 1776870131.011977 read 0x022535 thru 0x02f17d (0xcc49 bytes) 000699 1776870131.015685 trunc from 0x02aec2 to 0x03a31c (0xf45b bytes) 000700 1776870131.028904 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b492): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b492): Mirror 4: 0x4fd8ba10 (stale) Mirror 5: 0x4fd8ba10 (stale) CRC-32 checksum value for chunk [0, 0x3b492): Mirror 1: 0x4ec0a5f2 Mirror 6: 0x4ec0a5f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b492: skipped 000701 1776870131.312567 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b492): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b492): Mirror 4: 0x4fd8ba10 (stale) Mirror 5: 0x4fd8ba10 (stale) CRC-32 checksum value for chunk [0, 0x3b492): Mirror 1: 0x4ec0a5f2 Mirror 6: 0x4ec0a5f2 Mirror 7: 0x4ec0a5f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b492: skipped 000702 1776870131.596011 read 0x00377f thru 0x0061fd (0x2a7f bytes) 000703 1776870131.598241 punch from 0x009954 to 0x019241 (0xf8ee bytes) 000704 1776870131.613755 mapread 0x025814 thru 0x02aed9 (0x56c6 bytes) 000705 1776870131.621077 mapread 0x00219b thru 0x007abf (0x5925 bytes) 000706 1776870131.625065 mapwrite 0x01b316 thru 0x0250ee (0x9dd9 bytes) 000707 1776870131.641981 MIRROR_RESYNC 5 mirrors 000708 1776870131.941841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000709 1776870132.042141 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000710 1776870132.160150 read 0x0105bb thru 0x01ae19 (0xa85f bytes) 000711 1776870132.163288 trunc from 0x016190 to 0x0238f8 (0xd769 bytes) 000712 1776870132.177206 read 0x01d32d thru 0x01ed81 (0x1a55 bytes) 000713 1776870132.183052 read 0x00a2f9 thru 0x01192d (0x7635 bytes) 000714 1776870132.183723 mapwrite 0x02d014 thru 0x037e84 (0xae71 bytes) 000715 1776870132.195882 write 0x01149c thru 0x01e6df (0xd244 bytes) 000716 1776870132.198515 fallocate 0x0086e8 thru 0x0093cd (0xce6 bytes) 000717 1776870132.202496 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b492): Mirror 6: 0x482da09a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b492): Mirror 6: 0x482da09a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b492: skipped 000718 1776870132.295977 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b492): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b492): Mirror 6: 0x482da09a (stale) CRC-32 checksum value for chunk [0, 0x3b492): Mirror 1: 0x120ad145 Mirror 7: 0x120ad145 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b492: skipped 000719 1776870132.549809 read 0x0332e1 thru 0x03b491 (0x81b1 bytes) 000720 1776870132.556600 MIRROR_RESYNC 3 mirrors 000721 1776870132.771059 trunc from 0x03b492 to 0x01c0ca (0x1f3c7 bytes) 000723 1776870132.797781 read 0x015f46 thru 0x01b385 (0x5440 bytes) 000724 1776870132.801532 mapread 0x011b60 thru 0x01c0ca (0xa56b bytes) 000725 1776870132.809644 read 0x01831f thru 0x01bbfb (0x38dd bytes) 000726 1776870132.810127 fallocate 0x004236 thru 0x0049b3 (0x77e bytes) 000727 1776870132.812165 mapread 0x007015 thru 0x011181 (0xa16d bytes) 000729 1776870132.816060 trunc from 0x01c0cb to 0x006abe (0x1560c bytes) 000730 1776870132.824003 MIRROR_RESYNC 3 mirrors 000732 1776870133.002634 mapwrite 0x02163d thru 0x02e264 (0xcc28 bytes) 000733 1776870133.031467 trunc from 0x02e265 to 0x00936f (0x24ef5 bytes) 000734 1776870133.036817 read 0x00573c thru 0x006109 (0x9ce bytes) 000736 1776870133.040517 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9370): Mirror 7: 0xedeae9a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9370): Mirror 7: 0xedeae9a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9370: skipped 000737 1776870133.113489 punch from 0x002008 to 0x007884 (0x587d bytes) 000738 1776870133.115141 read 0x000289 thru 0x00936f (0x90e7 bytes) 000739 1776870133.118499 fallocate 0x03fd69 thru 0x03ffff (0x297 bytes) 000740 1776870133.120857 mapwrite 0x03993d thru 0x03ffff (0x66c3 bytes) 000743 1776870133.142643 punch from 0x01e42b to 0x0267c1 (0x8397 bytes) 000744 1776870133.144795 mapwrite 0x02cef4 thru 0x03a54b (0xd658 bytes) 000745 1776870133.156200 read 0x03d4b7 thru 0x03ffff (0x2b49 bytes) 000746 1776870133.156768 read 0x018c9b thru 0x01e09c (0x5402 bytes) 000747 1776870133.161135 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb22b833f Mirror 8: 0xb22b833f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000748 1776870133.379726 MIRROR_RESYNC 3 mirrors 000749 1776870133.526355 read 0x02b9bb thru 0x02ec83 (0x32c9 bytes) 000751 1776870133.528735 punch from 0x01ba63 to 0x024e4c (0x93ea bytes) 000752 1776870133.541304 mapread 0x01787a thru 0x01ae89 (0x3610 bytes) 000753 1776870133.548604 mapwrite 0x031791 thru 0x03bf67 (0xa7d7 bytes) 000756 1776870133.563215 read 0x0037fe thru 0x00c624 (0x8e27 bytes) 000757 1776870133.568272 MIRROR_RESYNC 3 mirrors 000758 1776870133.747287 MIRROR_EXTEND to 4 mirrors 000759 1776870133.920621 mapwrite 0x00e106 thru 0x01324f (0x514a bytes) 000760 1776870133.965386 write 0x003cd2 thru 0x009370 (0x569f bytes) 000761 1776870133.974298 fallocate 0x0264d4 thru 0x02f042 (0x8b6f bytes) 000762 1776870133.977082 punch from 0x00196b to 0x009c13 (0x82a9 bytes) 000763 1776870133.981337 trunc from 0x040000 to 0x00241e (0x3dbe1 bytes) 000764 1776870133.993290 trunc from 0x001620 to 0x00241e (0xdff bytes) 000765 1776870133.995346 punch from 0x000812 to 0x00241e (0x1c0d bytes) 000766 1776870133.997267 trunc from 0x00100b to 0x00241e (0x1414 bytes) 000767 1776870133.998974 write 0x0200bb thru 0x022607 (0x254d bytes) 000768 1776870133.999868 trunc from 0x02172c to 0x022607 (0xedc bytes) 000769 1776870134.007496 mapwrite 0x00c111 thru 0x011e67 (0x5d57 bytes) 000770 1776870134.018761 mapwrite 0x031001 thru 0x038d78 (0x7d78 bytes) 000771 1776870134.034840 fallocate 0x029c67 thru 0x02b692 (0x1a2c bytes) 000772 1776870134.037108 fallocate 0x0050aa thru 0x013e7e (0xedd5 bytes) 000773 1776870134.039181 write 0x00dcff thru 0x01af60 (0xd262 bytes) 000774 1776870134.043872 trunc from 0x00c989 to 0x00f279 (0x28f1 bytes) 000775 1776870134.052134 mapwrite 0x00b445 thru 0x00dd93 (0x294f bytes) 000776 1776870134.059956 fallocate 0x00bb49 thru 0x0169dd (0xae95 bytes) 000779 1776870134.065238 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d79): Mirror 8: 0x319dc5da (stale) Mirror 9: 0x319dc5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d79): Mirror 8: 0x319dc5da (stale) Mirror 9: 0x319dc5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d79: skipped 000780 1776870134.150556 read 0x02ff3f thru 0x030c0d (0xccf bytes) 000781 1776870134.156401 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d79): Mirror 9: 0x319dc5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d79): Mirror 9: 0x319dc5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d79: skipped 000783 1776870134.237388 trunc from 0x038d79 to 0x001a6d (0x3730b bytes) 000784 1776870134.242864 read 0x00082e thru 0x001a6d (0x1240 bytes) 000786 1776870134.245310 fallocate 0x0329fb thru 0x0381f0 (0x57f6 bytes) 000787 1776870134.247279 punch from 0x000af7 to 0x001a6d (0xf77 bytes) 000788 1776870134.249022 mapwrite 0x004932 thru 0x005800 (0xecf bytes) 000789 1776870134.260296 write 0x00b1d7 thru 0x00b35f (0x189 bytes) 000790 1776870134.261027 write 0x031813 thru 0x032a8e (0x127c bytes) 000791 1776870134.261730 mapwrite 0x02c736 thru 0x030933 (0x41fe bytes) 000792 1776870134.269472 trunc from 0x032a8f to 0x01084b (0x22243 bytes) 000793 1776870134.273925 trunc from 0x002d1f to 0x006d5d (0x403f bytes) 000794 1776870134.275643 fallocate 0x013d16 thru 0x0222ef (0xe5da bytes) 000796 1776870134.278162 mapread 0x00278c thru 0x01084b (0xe0c0 bytes) 000797 1776870134.281749 trunc from 0x001022 to 0x009483 (0x8462 bytes) 000798 1776870134.283762 fallocate 0x0093b0 thru 0x01722c (0xde7d bytes) 000799 1776870134.285871 punch from 0x01714d to 0x01722c (0xe0 bytes) 000800 1776870134.287793 write 0x013092 thru 0x020977 (0xd8e6 bytes) 000801 1776870134.293143 mapread 0x01a4d0 thru 0x020977 (0x64a8 bytes) 000802 1776870134.294136 write 0x018e99 thru 0x01f7a2 (0x690a bytes) 000804 1776870134.295699 write 0x021b3f thru 0x030109 (0xe5cb bytes) 000805 1776870134.296871 write 0x009cc1 thru 0x00ccd6 (0x3016 bytes) 000806 1776870134.300579 trunc from 0x0048df to 0x00efe7 (0xa709 bytes) 000807 1776870134.306469 mapwrite 0x015d03 thru 0x01d68b (0x7989 bytes) 000808 1776870134.314605 trunc from 0x03010a to 0x02cd53 (0x33b6 bytes) 000809 1776870134.320625 trunc from 0x015141 to 0x0196bf (0x457f bytes) 000810 1776870134.322783 mapread 0x0044e0 thru 0x011427 (0xcf48 bytes) 000811 1776870134.327666 trunc from 0x007c7b to 0x0098df (0x1c65 bytes) 000813 1776870134.329982 fallocate 0x004c60 thru 0x00f7fb (0xab9c bytes) 000815 1776870134.332576 read 0x00d418 thru 0x01cfff (0xfbe8 bytes) 000816 1776870134.335832 read 0x026b61 thru 0x02cd53 (0x61f3 bytes) 000817 1776870134.336414 trunc from 0x02cd54 to 0x003381 (0x299d2 bytes) 000818 1776870134.344050 read 0x001441 thru 0x002c5a (0x181a bytes) 000819 1776870134.344707 trunc from 0x002323 to 0x002698 (0x376 bytes) 000820 1776870134.350260 trunc from 0x0031a1 to 0x003381 (0x1e1 bytes) 000821 1776870134.357283 MIRROR_RESYNC 2 mirrors 000822 1776870134.536523 write 0x00431a thru 0x013aea (0xf7d1 bytes) 000823 1776870134.547985 mapread 0x00fbc3 thru 0x013aea (0x3f28 bytes) 000824 1776870134.548984 write 0x034564 thru 0x03ffff (0xba9c bytes) 000825 1776870134.550336 trunc from 0x03e3aa to 0x03ffff (0x1c56 bytes) 000826 1776870134.556998 fallocate 0x02ad6d thru 0x033633 (0x88c7 bytes) 000827 1776870134.558994 write 0x037d46 thru 0x039cab (0x1f66 bytes) 000828 1776870134.559846 read 0x01cda6 thru 0x0221a2 (0x53fd bytes) 000829 1776870134.563054 trunc from 0x040000 to 0x00cb6f (0x33490 bytes) 000830 1776870134.574469 trunc from 0x00cb70 to 0x036a62 (0x29ef3 bytes) 000831 1776870134.578973 punch from 0x0290fd to 0x034f3e (0xbe42 bytes) 000832 1776870134.580655 trunc from 0x034fc7 to 0x036a62 (0x1a9c bytes) 000833 1776870134.582552 trunc from 0x01bb67 to 0x029684 (0xdb1e bytes) 000834 1776870134.584238 punch from 0x02f81f to 0x0348b2 (0x5094 bytes) 000835 1776870134.586002 trunc from 0x036a63 to 0x01a717 (0x1c34b bytes) 000836 1776870134.591169 write 0x027c63 thru 0x028f8b (0x1329 bytes) 000837 1776870134.592165 trunc from 0x028f8c to 0x035fff (0xd074 bytes) 000838 1776870134.597294 punch from 0x00056c to 0x000eeb (0x980 bytes) 000839 1776870134.599130 mapwrite 0x015086 thru 0x01b531 (0x64ac bytes) 000840 1776870134.612169 fallocate 0x01490e thru 0x015999 (0x108c bytes) 000841 1776870134.614683 fallocate 0x0060bc thru 0x014b8b (0xead0 bytes) 000842 1776870134.616923 write 0x00837f thru 0x016254 (0xded6 bytes) 000843 1776870134.618095 mapread 0x0357af thru 0x035fff (0x851 bytes) 000844 1776870134.622596 trunc from 0x036000 to 0x027d2f (0xe2d0 bytes) 000845 1776870134.628771 mapwrite 0x01608b thru 0x017dae (0x1d24 bytes) 000847 1776870134.634787 mapread 0x012486 thru 0x018718 (0x6293 bytes) 000848 1776870134.635513 mapread 0x011a23 thru 0x013ce1 (0x22bf bytes) 000849 1776870134.636130 trunc from 0x009ac0 to 0x016eca (0xd40b bytes) 000850 1776870134.638220 trunc from 0x012bb7 to 0x019c11 (0x705b bytes) 000851 1776870134.640137 mapread 0x0222f6 thru 0x027d2f (0x5a3a bytes) 000852 1776870134.643898 fallocate 0x019fcb thru 0x0280f7 (0xe12d bytes) 000853 1776870134.646027 write 0x031655 thru 0x03c7b5 (0xb161 bytes) 000854 1776870134.647235 trunc from 0x01aab3 to 0x020a39 (0x5f87 bytes) 000855 1776870134.649416 trunc from 0x00a135 to 0x00b433 (0x12ff bytes) 000856 1776870134.651020 fallocate 0x005dc9 thru 0x00612a (0x362 bytes) 000857 1776870134.652792 mapwrite 0x017b69 thru 0x020590 (0x8a28 bytes) 000858 1776870134.670569 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c7b6): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7b6): Mirror 9: 0xe2da08d3 (stale) CRC-32 checksum value for chunk [0, 0x3c7b6): Mirror 1: 0xd653057f Mirror 10: 0xd653057f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7b6: skipped 000859 1776870134.944932 write 0x032b16 thru 0x034af3 (0x1fde bytes) 000860 1776870134.967409 trunc from 0x00f56a to 0x01c8ef (0xd386 bytes) 000861 1776870134.969904 write 0x032fe7 thru 0x03a2ba (0x72d4 bytes) 000862 1776870134.973052 trunc from 0x03309d to 0x03b431 (0x8395 bytes) 000864 1776870134.982855 write 0x0143a6 thru 0x01e710 (0xa36b bytes) 000865 1776870134.989769 punch from 0x0174a9 to 0x0184fa (0x1052 bytes) 000866 1776870134.997751 trunc from 0x022b40 to 0x02c4db (0x999c bytes) 000867 1776870134.999767 trunc from 0x03c7b6 to 0x03ea69 (0x22b4 bytes) 000868 1776870135.012289 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea6a): Mirror 9: 0x1e4cc4c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea6a): Mirror 9: 0x1e4cc4c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea6a: skipped 000869 1776870135.101408 punch from 0x022b38 to 0x029aa2 (0x6f6b bytes) 000870 1776870135.105079 trunc from 0x03ea6a to 0x017311 (0x27758 bytes) 000871 1776870135.113060 trunc from 0x0040a9 to 0x00fd8d (0xbce5 bytes) 000873 1776870135.117418 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000874 1776870135.163368 write 0x00e348 thru 0x00f72a (0x13e3 bytes) 000875 1776870135.174781 fallocate 0x036d04 thru 0x03eef1 (0x81ee bytes) 000876 1776870135.177601 punch from 0x029744 to 0x032d45 (0x9602 bytes) 000877 1776870135.179881 trunc from 0x03eef2 to 0x00634c (0x38ba5 bytes) 000878 1776870135.186283 mapread 0x0000b5 thru 0x00634c (0x6298 bytes) 000879 1776870135.190480 read 0x003734 thru 0x00634c (0x2c19 bytes) 000881 1776870135.191592 read 0x00002c thru 0x00634c (0x6321 bytes) 000882 1776870135.192151 mapread 0x00146a thru 0x00634c (0x4ee3 bytes) 000883 1776870135.192998 write 0x00b57e thru 0x00b72f (0x1b2 bytes) 000884 1776870135.193769 mapread 0x005ccb thru 0x00b72f (0x5a65 bytes) 000885 1776870135.197483 trunc from 0x00230b to 0x00b72f (0x9425 bytes) 000886 1776870135.201377 punch from 0x002bec to 0x006a4d (0x3e62 bytes) 000887 1776870135.204319 fallocate 0x00a983 thru 0x019b8b (0xf209 bytes) 000888 1776870135.206382 fallocate 0x0382c1 thru 0x03ffff (0x7d3f bytes) 000889 1776870135.208089 trunc from 0x008ef5 to 0x017f63 (0xf06f bytes) 000891 1776870135.210514 fallocate 0x0090fa thru 0x017aec (0xe9f3 bytes) 000892 1776870135.212320 mapread 0x0035fa thru 0x00ddea (0xa7f1 bytes) 000893 1776870135.226980 trunc from 0x019b8c to 0x011cb2 (0x7ed9 bytes) 000894 1776870135.231378 trunc from 0x00c52d to 0x011cb2 (0x5786 bytes) 000895 1776870135.233173 punch from 0x010922 to 0x011cb2 (0x1391 bytes) 000897 1776870135.237873 mapwrite 0x006bc4 thru 0x00b50a (0x4947 bytes) 000898 1776870135.244785 mapwrite 0x0119e1 thru 0x020d37 (0xf357 bytes) 000899 1776870135.269488 punch from 0x01b70a to 0x01ce3f (0x1736 bytes) 000901 1776870135.272204 write 0x035cce thru 0x038716 (0x2a49 bytes) 000902 1776870135.273180 read 0x02f4dd thru 0x031646 (0x216a bytes) 000904 1776870135.278248 fallocate 0x0185ef thru 0x01a583 (0x1f95 bytes) 000905 1776870135.280268 trunc from 0x021b0f to 0x023c18 (0x210a bytes) 000906 1776870135.281976 mapread 0x01639e thru 0x02176a (0xb3cd bytes) 000907 1776870135.284120 punch from 0x021549 to 0x026c23 (0x56db bytes) 000908 1776870135.286004 punch from 0x028b49 to 0x0349a1 (0xbe59 bytes) 000909 1776870135.287769 trunc from 0x02b3e8 to 0x038716 (0xd32f bytes) 000910 1776870135.290938 mapread 0x0314a2 thru 0x038716 (0x7275 bytes) 000911 1776870135.303070 mapread 0x012bab thru 0x0135c2 (0xa18 bytes) 000912 1776870135.304006 fallocate 0x016ea6 thru 0x01843c (0x1597 bytes) 000914 1776870135.307937 mapwrite 0x01d586 thru 0x02c5c9 (0xf044 bytes) 000915 1776870135.332376 write 0x010746 thru 0x01d0c4 (0xc97f bytes) 000916 1776870135.333512 read 0x01071c thru 0x014839 (0x411e bytes) 000917 1776870135.335606 MIRROR_EXTEND to 2 mirrors 000918 1776870135.565243 mapread 0x021890 thru 0x027894 (0x6005 bytes) 000919 1776870135.570348 mapwrite 0x001404 thru 0x00fef9 (0xeaf6 bytes) 000920 1776870135.604509 read 0x02edb5 thru 0x036b5e (0x7daa bytes) 000922 1776870135.608122 read 0x01c2d2 thru 0x02ad52 (0xea81 bytes) 000923 1776870135.611531 trunc from 0x038717 to 0x0069f3 (0x31d23 bytes) 000924 1776870135.624116 write 0x02a02a thru 0x033ad3 (0x9aaa bytes) 000925 1776870135.625376 trunc from 0x033ad4 to 0x02b8ab (0x8228 bytes) 000926 1776870135.632424 punch from 0x014460 to 0x01a47e (0x601f bytes) 000927 1776870135.634358 mapwrite 0x011e7c thru 0x01f8d7 (0xda5c bytes) 000928 1776870135.670133 punch from 0x0197b0 to 0x022efb (0x974c bytes) 000929 1776870135.673390 read 0x017667 thru 0x01e38a (0x6d24 bytes) 000930 1776870135.677613 read 0x0188a4 thru 0x01d077 (0x47d4 bytes) 000931 1776870135.679821 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000932 1776870135.722830 punch from 0x01b15f to 0x01cdb3 (0x1c55 bytes) 000933 1776870135.726292 mapread 0x00c423 thru 0x013b03 (0x76e1 bytes) 000934 1776870135.733514 fallocate 0x0351da thru 0x03ffff (0xae26 bytes) 000935 1776870135.735981 fallocate 0x00cb0d thru 0x01121a (0x470e bytes) 000936 1776870135.738711 trunc from 0x02b8ac to 0x034f7a (0x96cf bytes) 000937 1776870135.746827 trunc from 0x026643 to 0x02793f (0x12fd bytes) 000938 1776870135.748849 trunc from 0x034f7b to 0x012ffb (0x21f7f bytes) 000939 1776870135.757193 mapwrite 0x0089f9 thru 0x00f443 (0x6a4b bytes) 000940 1776870135.766065 trunc from 0x012ffc to 0x037eaf (0x24eb4 bytes) 000941 1776870135.771174 mapwrite 0x00f71b thru 0x0102a0 (0xb86 bytes) 000942 1776870135.777447 mapread 0x006857 thru 0x0152c4 (0xea6e bytes) 000943 1776870135.784133 trunc from 0x001e55 to 0x011149 (0xf2f5 bytes) 000945 1776870135.786866 read 0x02a99c thru 0x037eaf (0xd514 bytes) 000946 1776870135.789781 write 0x034c7d thru 0x03ffff (0xb383 bytes) 000947 1776870135.791016 punch from 0x0222e5 to 0x0295e0 (0x72fc bytes) 000949 1776870135.793494 punch from 0x023eb3 to 0x0330db (0xf229 bytes) 000950 1776870135.795774 trunc from 0x02b152 to 0x035ce5 (0xab94 bytes) 000951 1776870135.802633 trunc from 0x00df13 to 0x0150c5 (0x71b3 bytes) 000952 1776870135.807848 MIRROR_EXTEND to 2 mirrors 000953 1776870135.991403 mapread 0x0392f3 thru 0x03ffff (0x6d0d bytes) 000954 1776870135.999166 punch from 0x031766 to 0x03ec9d (0xd538 bytes) 000955 1776870136.017002 mapwrite 0x0055b4 thru 0x008393 (0x2de0 bytes) 000956 1776870136.036560 read 0x01014b thru 0x0124d6 (0x238c bytes) 000957 1776870136.039079 mapread 0x005a94 thru 0x012b9e (0xd10b bytes) 000958 1776870136.041839 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000959 1776870136.078800 trunc from 0x040000 to 0x029077 (0x16f88 bytes) 000960 1776870136.095537 fallocate 0x00e8a2 thru 0x011ce2 (0x3441 bytes) 000961 1776870136.097914 mapread 0x01dc83 thru 0x029077 (0xb3f5 bytes) 000964 1776870136.111322 trunc from 0x01a9d2 to 0x024167 (0x9796 bytes) 000965 1776870136.113783 fallocate 0x005111 thru 0x0119a9 (0xc899 bytes) 000966 1776870136.115961 punch from 0x01bde3 to 0x025144 (0x9362 bytes) 000967 1776870136.117962 fallocate 0x00587a thru 0x0157eb (0xff72 bytes) 000968 1776870136.122408 mapwrite 0x03c5f3 thru 0x03ffff (0x3a0d bytes) 000969 1776870136.137176 punch from 0x024142 to 0x029885 (0x5744 bytes) 000970 1776870136.140915 MIRROR_EXTEND to 2 mirrors 000971 1776870136.328820 read 0x034339 thru 0x03ffff (0xbcc7 bytes) 000972 1776870136.332323 write 0x02abc5 thru 0x0341ae (0x95ea bytes) 000973 1776870136.347816 trunc from 0x040000 to 0x00053b (0x3fac4 bytes) 000974 1776870136.361978 mapwrite 0x006dc7 thru 0x00c438 (0x5672 bytes) 000975 1776870136.376251 fallocate 0x02ac56 thru 0x03738f (0xc73a bytes) 000976 1776870136.378483 read 0x025bc1 thru 0x025fcb (0x40b bytes) 000978 1776870136.383319 mapwrite 0x0060c2 thru 0x00a1a8 (0x40e7 bytes) 000979 1776870136.389944 fallocate 0x036ee8 thru 0x03ffff (0x9118 bytes) 000980 1776870136.391940 read 0x029896 thru 0x031525 (0x7c90 bytes) 000982 1776870136.392902 trunc from 0x01d4b5 to 0x028535 (0xb081 bytes) 000983 1776870136.395068 mapwrite 0x013954 thru 0x0197a4 (0x5e51 bytes) 000984 1776870136.401803 write 0x00e846 thru 0x0128a1 (0x405c bytes) 000985 1776870136.402732 read 0x01b78b thru 0x0248c0 (0x9136 bytes) 000986 1776870136.406119 read 0x00cb11 thru 0x00d343 (0x833 bytes) 000987 1776870136.408341 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x695:0x0]: Invalid argument (22) 000988 1776870136.460568 mapread 0x00928f thru 0x00ecf1 (0x5a63 bytes) 000989 1776870136.466864 mapwrite 0x01d9ec thru 0x027fe1 (0xa5f6 bytes) 000990 1776870136.478993 trunc from 0x037390 to 0x03a432 (0x30a3 bytes) 000991 1776870136.484982 fallocate 0x011659 thru 0x01f863 (0xe20b bytes) 000992 1776870136.486961 trunc from 0x03a433 to 0x03d4ab (0x3079 bytes) 000993 1776870136.491751 mapwrite 0x02fa9f thru 0x03739e (0x7900 bytes) 000994 1776870136.500598 mapwrite 0x03d768 thru 0x03de1d (0x6b6 bytes) 000995 1776870136.510632 mapwrite 0x000143 thru 0x007d2e (0x7bec bytes) 000996 1776870136.518274 read 0x0194de thru 0x02254a (0x906d bytes) 000997 1776870136.518782 read 0x0025ea thru 0x004f2f (0x2946 bytes) 000998 1776870136.519261 mapread 0x00eb54 thru 0x00eecc (0x379 bytes) 000999 1776870136.519900 read 0x00fdac thru 0x01ce1c (0xd071 bytes) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 11:02:18 (1776870138) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x6d4:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 11:02:21 (1776870141) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0757639 s, 27.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385236 s, 27.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 11:02:24 (1776870144) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7572416 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1336288 .. failed Extending file size to 2125440 .. Extending file size to 7728192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7986336 .. Extending file size to 7731168 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8659424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8945184 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8049376 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8266688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8858048 .. failed Extending file size to 9001760 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8976640 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9105696 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63572 63573 63575 63577 63580 done 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:03:30 (1776870210) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6188256 .. Extending file size to 7317216 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7497280 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8676544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7928192 .. Extending file size to 8588000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8789792 .. Extending file size to 8899776 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9102144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9314144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 66737 66738 66739 66741 66744 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:04:34 (1776870274) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2816 1268488 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17976 3589044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26104 3580916 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44080 7169960 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2816 1268488 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17976 3589044 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 26104 3580916 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44080 7169960 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0408484 s, 51.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0529299 s, 39.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5a07e826 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:04:41 (1776870281) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0538781 s, 38.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 11:04:43 (1776870283) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:04:46 (1776870286) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0265645 s, 78.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x98c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x98e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x98f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:04:48 (1776870288) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0204919 s, 102 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:04:50 (1776870290) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.021464 s, 97.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:05:14 (1776870314) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162244 s, 64.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:05:48 (1776870348) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0219568 s, 95.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:06:23 (1776870383) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg129-server: Pool lustre.test_204f created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0222266 s, 94.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg129-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:07:10 (1776870430) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:07:12 (1776870432) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:07:14 (1776870434) oleg129-server: Pool lustre.test_206 created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg129-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:07:28 (1776870448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262186 s, 40.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116597 s, 89.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdb8388c2 Mirror 3: 0xdb8388c2 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:07:31 (1776870451) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:07:32 (1776870452) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:07:33 (1776870453) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.011522 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1710000 Read this (4 bytes): BBBB 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:07:48 (1776870468) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00346017 s, 2.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:07:50 (1776870470) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147816 s, 70.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 11:07:51.000000000 -0400 Modify: 2026-04-22 11:07:51.000000000 -0400 Change: 2026-04-22 11:07:51.000000000 -0400 Birth: 2026-04-22 11:07:50.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:07:53 (1776870473) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137692 s, 76.2 MB/s oleg129-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg129-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg129-server: O/280000401/d31/bf: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:08:07 (1776870487) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0384434 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00402237 s, 1.0 MB/s size after second write -rw-r--r-- 1 root root 4096 Apr 22 11:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 22 11:08 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:08:09 (1776870489) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:08:11 (1776870491) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 819 sec ============ 11:08:13 (1776870493) === sanity-flr: start cleanup 11:08:14 (1776870494) === === sanity-flr: finish cleanup 11:08:16 (1776870496) ===