-----============= acceptance-small: sanity-flr ============----- Thu Jan 1 19:51:22 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg254-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:51:46 (1767315106) === oleg254-client.virtnet: executing check_config_client /mnt/lustre oleg254-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg254-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92cd42f9f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92cd42f9f000.idle_timeout=debug disable quota as required oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:52:50 (1767315170) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f9345] == sanity-flr test 0a: lfs mirror create with -N option == 19:52:52 (1767315172) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:53:08 (1767315188) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:53:12 (1767315192) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:53:16 (1767315196) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:53:44 (1767315224) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:53:49 (1767315229) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:53:53 (1767315233) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92cd42f9f000.stats=clear osc.lustre-OST0001-osc-ffff92cd42f9f000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:54:01 (1767315241) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:54:13 (1767315253) PASS 0j (13s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:54:27 (1767315267) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:54:32 (1767315272) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (16s) == sanity-flr test 2: create components from existing files ========================================================== 19:54:49 (1767315289) PASS 2 (12s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:55:00 (1767315300) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:55:06 (1767315306) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:55:16 (1767315316) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:55:28 (1767315328) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:55:36 (1767315336) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.29702 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222206 s, 4.7 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:55:48 (1767315348) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.615727 s, 34.1 MB/s PASS 22 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:56:00 (1767315360) Stopping /mnt/lustre-ost1 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg254-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost2 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg254-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (87s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:57:26 (1767315446) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.738615 s, 14.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg254-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:58:16 (1767315496) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg254-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost2 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg254-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (92s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:59:48 (1767315588) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.8106 s, 9.0 MB/s -rw-r--r-- 1 root root 52428800 Jan 1 19:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.461s user 0m0.001s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.670s user 0m0.001s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg254-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg254-server oleg254-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.690s user 0m0.002s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 affected facets: ost2 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg254-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92cd42f9f000.ost_server_uuid in FULL state after 0 sec PASS 33b (121s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:01:49 (1767315709) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:01:53 (1767315713) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:01:57 (1767315717) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 20:02:01 (1767315721) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 20:02:10 (1767315730) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.738838 s, 17.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:02:26 (1767315746) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620798 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472809 s, 22.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:02:34 (1767315754) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:02:42 (1767315762) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00389509 s, 263 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113024 s, 906 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00277812 s, 369 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119703 s, 855 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 20:02:55 (1767315775) create test files with size 15749821 .. a931294aa05d39a1844b02585af12b15 - fd471235502e66a70ef8564012606818 - aabbda2768b1bb57b898bc096a85f298 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 20:03:23 (1767315803) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:03:42 (1767315822) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:03:50 (1767315830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188819 s, 5.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402178 s, 26.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 20:04:02 (1767315842) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234949 s, 8.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.554228 s, 7.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259501 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 20:04:20 (1767315860) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:04:23 (1767315863) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:04:27 (1767315867) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250894 s, 41.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:04:35 (1767315875) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:04:39 (1767315879) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x55:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x54:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x54:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:04:51 (1767315891) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.483054 s, 21.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:05:05 (1767315905) llite.lustre-ffff92cd4b3a5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299784 s, 35.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff92cd4c4b9800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:05:22 (1767315922) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:06:03 (1767315963) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:06:44 (1767316004) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:06:48 (1767316008) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.284951 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.332005 s, 9.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd54c2006 (stale) Mirror 2: 0xd54c2006 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 49025b4a86ebe56d26b7d89c1ada0641 - mirror 1 checksum: cc43970896c33ad3719966b5625efe5f - mirror 2 checksum: 49025b4a86ebe56d26b7d89c1ada0641 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 20:06:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 20:06:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 49025b4a86ebe56d26b7d89c1ada0641 - mirror 1 checksum: cc43970896c33ad3719966b5625efe5f - mirror 2 checksum: cc43970896c33ad3719966b5625efe5f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 20:06:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 20:06:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:07:04 (1767316024) Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 20:07:12 (1767316032) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 20:07:47 (1767316067) targets are mounted 20:07:47 (1767316067) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353649 s, 29.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:08:07 (1767316087) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:08:11 (1767316091) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:08:14 (1767316094) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00902414 s, 1.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:08:23 (1767316103) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:08:27 (1767316107) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:08:31 (1767316111) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:08:57 (1767316137) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:09:26 (1767316166) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:09:51 (1767316191) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:10:16 (1767316216) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:10:24 (1767316224) mirror create pid 54867 mirror split pid 54869 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7808: 54869 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 54867 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:11:31 (1767316291) Seed set to 6293 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0xaa90 000004 1767316293.926933 trunc from 00000000 to 0x00aa8f (0xaa90 bytes) 000006 1767316293.945883 mapread 0x009888 thru 0x00aa8f (0x1208 bytes) 000007 1767316293.962773 write 0x010b09 thru 0x019ee4 (0x93dc bytes) 000008 1767316293.965194 mapwrite 0x026e38 thru 0x03228f (0xb458 bytes) 000009 1767316294.154736 read 0x0007a5 thru 0x004722 (0x3f7e bytes) 000010 1767316294.187833 mapwrite 0x01986e thru 0x01dacb (0x425e bytes) 000012 1767316294.301596 write 0x02246c thru 0x025806 (0x339b bytes) truncating to largest ever: 0x22181 000014 1767316294.305003 trunc from 0x032290 to 0x022180 (0x1010f bytes) 000016 1767316294.324533 mapread 0x0105cc thru 0x01c937 (0xc36c bytes) truncating to largest ever: 0x28ba6 000018 1767316294.331713 trunc from 0x022181 to 0x028ba5 (0x6a25 bytes) 000019 1767316294.364763 MIRROR_EXTEND to 2 mirrors 000020 1767316295.133201 write 0x008d80 thru 0x00b45e (0x26df bytes) 000021 1767316295.217637 read 0x009ab2 thru 0x00edc0 (0x530f bytes) 000022 1767316295.230506 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 2: 0x68ad3055 (stale) CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 1: 0xbf6aa7ef Mirror 3: 0xbf6aa7ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ba6: skipped 000025 1767316296.366577 write 0x009a27 thru 0x012300 (0x88da bytes) 000027 1767316296.517464 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 2: 0x68ad3055 (stale) Mirror 3: 0xbf6aa7ef (stale) CRC-32 checksum value for chunk [0, 0x28ba6): Mirror 1: 0xca24fd0f Mirror 4: 0xca24fd0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ba6: skipped 000028 1767316298.190265 read 0x004726 thru 0x00ab7f (0x645a bytes) 000029 1767316298.219916 read 0x01f0f6 thru 0x028ba5 (0x9ab0 bytes) truncating to largest ever: 0x377f6 000032 1767316298.266935 trunc from 0x028ba6 to 0x0377f5 (0xec50 bytes) 000035 1767316298.468387 mapread 0x003a14 thru 0x00b72c (0x7d19 bytes) 000037 1767316298.563833 trunc from 0x0377f6 to 0x01e139 (0x196bc bytes) 000038 1767316298.597109 mapread 0x01964b thru 0x01dbea (0x45a0 bytes) 000040 1767316298.672089 write 0x008cd7 thru 0x017a37 (0xed61 bytes) 000041 1767316298.682262 read 0x00cb94 thru 0x01a8b9 (0xdd26 bytes) 000042 1767316298.698452 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e13a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e13a): Mirror 2: 0xf3ea5f16 (stale) Mirror 3: 0xf5520752 (stale) Mirror 4: 0xcbf7814 (stale) CRC-32 checksum value for chunk [0, 0x1e13a): Mirror 1: 0xc372e604 Mirror 5: 0xc372e604 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e13a: skipped 000043 1767316299.757937 write 0x03505c thru 0x03f86e (0xa813 bytes) 000046 1767316299.853867 mapread 0x03d427 thru 0x03f86e (0x2448 bytes) 000047 1767316299.855783 mapread 0x02b928 thru 0x02dc35 (0x230e bytes) 000048 1767316299.884458 MIRROR_RESYNC 5 mirrors 000049 1767316301.001503 read 0x005f48 thru 0x00e041 (0x80fa bytes) 000050 1767316301.012425 mapwrite 0x03108a thru 0x03f9ad (0xe924 bytes) 000051 1767316301.165168 write 0x014d42 thru 0x0168f4 (0x1bb3 bytes) 000054 1767316301.182439 mapwrite 0x02e84e thru 0x02f440 (0xbf3 bytes) 000055 1767316301.230184 write 0x038e5b thru 0x03ffff (0x71a5 bytes) 000056 1767316301.231524 read 0x002248 thru 0x00f76f (0xd528 bytes) 000057 1767316301.243858 write 0x010d9a thru 0x013a63 (0x2cca bytes) 000058 1767316301.245125 mapread 0x03b452 thru 0x03ffff (0x4bae bytes) 000059 1767316301.248496 mapwrite 0x033628 thru 0x03ed0d (0xb6e6 bytes) 000060 1767316301.295148 mapwrite 0x00c13a thru 0x00eb0d (0x29d4 bytes) 000061 1767316301.353811 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x677a3bbf (stale) Mirror 3: 0x677a3bbf (stale) Mirror 4: 0x677a3bbf (stale) Mirror 5: 0x677a3bbf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0f1db2d Mirror 6: 0xc0f1db2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000066 1767316302.213546 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x677a3bbf (stale) Mirror 3: 0x677a3bbf (stale) Mirror 4: 0x677a3bbf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x677a3bbf (stale) Mirror 3: 0x677a3bbf (stale) Mirror 4: 0x677a3bbf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0f1db2d Mirror 6: 0xc0f1db2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1767316303.001710 write 0x01a263 thru 0x022225 (0x7fc3 bytes) 000069 1767316303.103656 mapwrite 0x003805 thru 0x00385b (0x57 bytes) 000071 1767316303.175198 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x677a3bbf (stale) Mirror 3: 0x677a3bbf (stale) Mirror 4: 0x677a3bbf (stale) Mirror 6: 0xc0f1db2d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83c61a6b Mirror 7: 0x83c61a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000072 1767316304.226674 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x677a3bbf (stale) Mirror 3: 0x677a3bbf (stale) Mirror 4: 0x677a3bbf (stale) Mirror 6: 0xc0f1db2d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83c61a6b Mirror 7: 0x83c61a6b Mirror 8: 0x83c61a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000073 1767316305.044136 mapread 0x02e961 thru 0x03aebe (0xc55e bytes) 000074 1767316305.067632 trunc from 0x040000 to 0x007596 (0x38a69 bytes) 000076 1767316305.170136 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x7597): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7597): Mirror 2: 0x6230e542 (stale) Mirror 3: 0x6230e542 (stale) Mirror 4: 0x6230e542 (stale) Mirror 6: 0x6230e542 (stale) Mirror 7: 0x43d4ff85 (stale) Mirror 8: 0x43d4ff85 (stale) CRC-32 checksum value for chunk [0, 0x7597): Mirror 1: 0x43d4ff85 Mirror 9: 0x43d4ff85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7597: skipped 000077 1767316306.013466 mapwrite 0x0015e5 thru 0x0090d1 (0x7aed bytes) 000078 1767316306.202521 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x90d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x90d2): Mirror 2: 0x8041ee20 (stale) Mirror 3: 0x8041ee20 (stale) Mirror 4: 0x8041ee20 (stale) Mirror 6: 0x8041ee20 (stale) Mirror 7: 0x6fbae92 (stale) Mirror 8: 0x6fbae92 (stale) Mirror 9: 0xfebf3e78 (stale) CRC-32 checksum value for chunk [0, 0x90d2): Mirror 1: 0x6f3b9d03 Mirror 10: 0x6f3b9d03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x90d2: skipped 000079 1767316307.063241 mapwrite 0x027e85 thru 0x0377ed (0xf969 bytes) 000080 1767316307.465728 mapwrite 0x00b964 thru 0x01783c (0xbed9 bytes) 000081 1767316307.569731 mapread 0x00e976 thru 0x017661 (0x8cec bytes) 000083 1767316307.570709 trunc from 0x0377ee to 0x01ee46 (0x189a7 bytes) 000085 1767316307.584874 write 0x022941 thru 0x028a62 (0x6122 bytes) 000087 1767316307.586471 mapwrite 0x011f52 thru 0x01ef31 (0xcfe0 bytes) 000090 1767316307.671744 mapread 0x0251ec thru 0x028a62 (0x3877 bytes) 000091 1767316307.674666 mapwrite 0x018cf6 thru 0x025825 (0xcb30 bytes) 000092 1767316307.759827 MIRROR_RESYNC 9 mirrors 000093 1767316309.151011 mapwrite 0x030485 thru 0x039905 (0x9481 bytes) 000096 1767316309.353752 read 0x0212f7 thru 0x02d4a0 (0xc1aa bytes) 000097 1767316309.375480 write 0x02d15f thru 0x0356c7 (0x8569 bytes) 000098 1767316309.377031 trunc from 0x039906 to 0x00114c (0x387b9 bytes) 000100 1767316309.411477 mapread 0x000109 thru 0x00114c (0x1044 bytes) 000101 1767316309.423213 trunc from 0x00114d to 0x0370d8 (0x35f8c bytes) 000102 1767316309.462091 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x370d9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370d9): Mirror 2: 0xda7043f (stale) Mirror 3: 0xda7043f (stale) Mirror 4: 0xda7043f (stale) Mirror 6: 0xda7043f (stale) Mirror 7: 0xda7043f (stale) Mirror 8: 0xda7043f (stale) Mirror 9: 0xda7043f (stale) Mirror 10: 0xda7043f (stale) CRC-32 checksum value for chunk [0, 0x370d9): Mirror 1: 0x1809cf90 Mirror 11: 0x1809cf90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370d9: skipped truncating to largest ever: 0x3c75b 000105 1767316310.518590 trunc from 0x0370d9 to 0x03c75a (0x5682 bytes) 000109 1767316310.584128 mapwrite 0x02386c thru 0x030034 (0xc7c9 bytes) 000111 1767316310.665635 MIRROR_RESYNC 10 mirrors 000113 1767316312.141644 mapwrite 0x02c3f5 thru 0x033517 (0x7123 bytes) 000115 1767316312.316081 mapwrite 0x0198d5 thru 0x02946b (0xfb97 bytes) 000117 1767316312.376609 mapread 0x0152d4 thru 0x01f434 (0xa161 bytes) 000118 1767316312.377581 mapwrite 0x022adb thru 0x0245a7 (0x1acd bytes) 000119 1767316312.413072 read 0x026277 thru 0x0331bb (0xcf45 bytes) 000120 1767316312.413792 write 0x0067b5 thru 0x011763 (0xafaf bytes) 000121 1767316312.426735 mapwrite 0x0005a2 thru 0x00284a (0x22a9 bytes) 000123 1767316312.485039 read 0x02d0a7 thru 0x036181 (0x90db bytes) 000127 1767316312.495271 write 0x016b96 thru 0x01d4b7 (0x6922 bytes) 000130 1767316312.497633 read 0x024709 thru 0x02a027 (0x591f bytes) 000131 1767316312.498084 trunc from 0x03c75b to 0x0159b0 (0x26daa bytes) 000132 1767316312.524018 trunc from 0x0159b1 to 0x03bf72 (0x265c2 bytes) 000133 1767316312.538948 trunc from 0x03bf73 to 0x021b0c (0x1a466 bytes) 000134 1767316312.548263 trunc from 0x021b0d to 0x02acac (0x91a0 bytes) 000136 1767316312.563443 read 0x01b5d3 thru 0x022113 (0x6b41 bytes) 000137 1767316312.573786 write 0x0295b5 thru 0x02d7f2 (0x423e bytes) 000138 1767316312.580977 MIRROR_RESYNC 10 mirrors 000139 1767316314.039078 write 0x0325f4 thru 0x035982 (0x338f bytes) 000143 1767316314.121622 mapread 0x02d7fd thru 0x02ea20 (0x1224 bytes) 000144 1767316314.133643 mapwrite 0x02f324 thru 0x03da9d (0xe77a bytes) 000146 1767316314.268608 mapwrite 0x03dfaa thru 0x03ffff (0x2056 bytes) 000147 1767316314.342824 mapread 0x036967 thru 0x03726e (0x908 bytes) 000148 1767316314.343658 trunc from 0x040000 to 0x015198 (0x2ae67 bytes) 000151 1767316314.374686 read 0x003580 thru 0x00b0d4 (0x7b55 bytes) 000152 1767316314.382704 write 0x02762c thru 0x03411a (0xcaef bytes) 000156 1767316314.386054 read 0x017905 thru 0x024ae7 (0xd1e3 bytes) 000157 1767316314.394890 mapwrite 0x03d130 thru 0x03d4ae (0x37f bytes) 000158 1767316314.518777 trunc from 0x03d4af to 0x018ee9 (0x245c5 bytes) 000159 1767316314.541790 mapwrite 0x037b92 thru 0x03c2d2 (0x4741 bytes) 000160 1767316314.667791 write 0x03df43 thru 0x03ffff (0x20bd bytes) 000161 1767316314.668814 mapwrite 0x006ba1 thru 0x00dece (0x732e bytes) 000162 1767316314.758813 mapwrite 0x024fa7 thru 0x03223f (0xd299 bytes) 000163 1767316315.016844 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88a10fe4 (stale) Mirror 3: 0x88a10fe4 (stale) Mirror 4: 0x88a10fe4 (stale) Mirror 6: 0x88a10fe4 (stale) Mirror 7: 0x88a10fe4 (stale) Mirror 8: 0x88a10fe4 (stale) Mirror 9: 0x88a10fe4 (stale) Mirror 10: 0x88a10fe4 (stale) Mirror 11: 0x88a10fe4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b423201 Mirror 12: 0x4b423201 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000165 1767316316.052085 trunc from 0x040000 to 0x014788 (0x2b877 bytes) 000167 1767316316.143463 read 0x00dac9 thru 0x014788 (0x6cc0 bytes) 000169 1767316316.152121 mapwrite 0x039c11 thru 0x03dfb1 (0x43a1 bytes) 000172 1767316316.284993 MIRROR_RESYNC 11 mirrors 000173 1767316318.187009 mapwrite 0x03c133 thru 0x03d94c (0x181a bytes) 000174 1767316318.385278 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfb2): Mirror 2: 0x51dd12d2 (stale) Mirror 3: 0x51dd12d2 (stale) Mirror 6: 0x51dd12d2 (stale) Mirror 7: 0x51dd12d2 (stale) Mirror 8: 0x51dd12d2 (stale) Mirror 9: 0x51dd12d2 (stale) Mirror 10: 0x51dd12d2 (stale) Mirror 11: 0x51dd12d2 (stale) Mirror 12: 0x51dd12d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfb2): Mirror 2: 0x51dd12d2 (stale) Mirror 3: 0x51dd12d2 (stale) Mirror 6: 0x51dd12d2 (stale) Mirror 7: 0x51dd12d2 (stale) Mirror 8: 0x51dd12d2 (stale) Mirror 9: 0x51dd12d2 (stale) Mirror 10: 0x51dd12d2 (stale) Mirror 11: 0x51dd12d2 (stale) Mirror 12: 0x51dd12d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfb2: skipped 000175 1767316319.211198 trunc from 0x03dfb2 to 0x0366b4 (0x78fd bytes) 000176 1767316319.243054 write 0x019ecb thru 0x01f259 (0x538f bytes) 000177 1767316319.257062 read 0x0348ed thru 0x0366b4 (0x1dc8 bytes) 000178 1767316319.279927 mapwrite 0x00e2bd thru 0x015fa1 (0x7ce5 bytes) 000179 1767316319.374438 mapwrite 0x0192b8 thru 0x020be0 (0x7929 bytes) 000180 1767316319.432530 mapread 0x01243b thru 0x01aeb0 (0x8a76 bytes) 000181 1767316319.438891 mapread 0x00b203 thru 0x0181da (0xcfd8 bytes) 000183 1767316319.445463 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x366b5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b5): Mirror 2: 0x1823c0cd (stale) Mirror 3: 0x1823c0cd (stale) Mirror 6: 0x1823c0cd (stale) Mirror 7: 0x1823c0cd (stale) Mirror 8: 0x1823c0cd (stale) Mirror 9: 0x1823c0cd (stale) Mirror 10: 0x1823c0cd (stale) Mirror 11: 0x1823c0cd (stale) Mirror 12: 0x1823c0cd (stale) CRC-32 checksum value for chunk [0, 0x366b5): Mirror 1: 0x7847766e Mirror 13: 0x7847766e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b5: skipped 000184 1767316320.341988 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366b5): Mirror 3: 0x1823c0cd (stale) Mirror 6: 0x1823c0cd (stale) Mirror 7: 0x1823c0cd (stale) Mirror 8: 0x1823c0cd (stale) Mirror 9: 0x1823c0cd (stale) Mirror 10: 0x1823c0cd (stale) Mirror 11: 0x1823c0cd (stale) Mirror 12: 0x1823c0cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b5): Mirror 3: 0x1823c0cd (stale) Mirror 6: 0x1823c0cd (stale) Mirror 7: 0x1823c0cd (stale) Mirror 8: 0x1823c0cd (stale) Mirror 9: 0x1823c0cd (stale) Mirror 10: 0x1823c0cd (stale) Mirror 11: 0x1823c0cd (stale) Mirror 12: 0x1823c0cd (stale) CRC-32 checksum value for chunk [0, 0x366b5): Mirror 1: 0x7847766e Mirror 13: 0x7847766e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b5: skipped 000185 1767316321.569681 read 0x021e41 thru 0x029042 (0x7202 bytes) 000186 1767316321.604467 mapread 0x0008ae thru 0x005925 (0x5078 bytes) 000187 1767316321.665604 write 0x02bb78 thru 0x0386d6 (0xcb5f bytes) 000189 1767316321.819588 write 0x019296 thru 0x01a0ad (0xe18 bytes) 000191 1767316321.842673 mapread 0x01d96d thru 0x02445d (0x6af1 bytes) 000192 1767316321.873758 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x386d7): Mirror 3: 0x3e47d3ae (stale) Mirror 6: 0x3e47d3ae (stale) Mirror 7: 0x3e47d3ae (stale) Mirror 8: 0x3e47d3ae (stale) Mirror 9: 0x3e47d3ae (stale) Mirror 10: 0x3e47d3ae (stale) Mirror 11: 0x3e47d3ae (stale) Mirror 12: 0x3e47d3ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x386d7): Mirror 3: 0x3e47d3ae (stale) Mirror 6: 0x3e47d3ae (stale) Mirror 7: 0x3e47d3ae (stale) Mirror 8: 0x3e47d3ae (stale) Mirror 9: 0x3e47d3ae (stale) Mirror 10: 0x3e47d3ae (stale) Mirror 11: 0x3e47d3ae (stale) Mirror 12: 0x3e47d3ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x386d7: skipped 000193 1767316322.534610 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x386d7): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x386d7): Mirror 3: 0x3e47d3ae (stale) Mirror 6: 0x3e47d3ae (stale) Mirror 7: 0x3e47d3ae (stale) Mirror 8: 0x3e47d3ae (stale) Mirror 9: 0x3e47d3ae (stale) Mirror 10: 0x3e47d3ae (stale) Mirror 11: 0x3e47d3ae (stale) Mirror 12: 0x3e47d3ae (stale) CRC-32 checksum value for chunk [0, 0x386d7): Mirror 1: 0xded803df Mirror 13: 0xded803df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x386d7: skipped 000197 1767316323.753441 trunc from 0x0386d7 to 0x016fea (0x216ec bytes) 000199 1767316323.847699 mapwrite 0x029322 thru 0x035353 (0xc032 bytes) 000200 1767316323.946746 mapwrite 0x022a2c thru 0x024469 (0x1a3e bytes) 000202 1767316324.020523 write 0x0071f5 thru 0x010c93 (0x9a9f bytes) 000203 1767316324.034177 mapread 0x0207b2 thru 0x02ed54 (0xe5a3 bytes) 000205 1767316324.039740 read 0x02be64 thru 0x0327e3 (0x6980 bytes) 000206 1767316324.041155 mapwrite 0x01f611 thru 0x029fd5 (0xa9c5 bytes) 000207 1767316324.182698 write 0x038028 thru 0x03ffff (0x7fd8 bytes) 000208 1767316324.184131 mapwrite 0x027345 thru 0x0368d4 (0xf590 bytes) 000210 1767316324.284752 read 0x038382 thru 0x03ffff (0x7c7e bytes) 000213 1767316324.286641 write 0x037f78 thru 0x03ffff (0x8088 bytes) 000214 1767316324.287896 mapread 0x02360e thru 0x02d33e (0x9d31 bytes) 000215 1767316324.289023 read 0x02ace3 thru 0x030d5a (0x6078 bytes) 000216 1767316324.289590 read 0x03d2df thru 0x03f04b (0x1d6d bytes) 000217 1767316324.290655 trunc from 0x040000 to 0x018da7 (0x27258 bytes) 000219 1767316324.315485 trunc from 0x018da8 to 0x024865 (0xbabe bytes) truncating to largest ever: 0x3f54d 000220 1767316324.331972 trunc from 0x024866 to 0x03f54c (0x1ace7 bytes) 000223 1767316324.358602 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f54d): Mirror 6: 0xb17c0158 (stale) Mirror 7: 0xb17c0158 (stale) Mirror 8: 0xb17c0158 (stale) Mirror 9: 0xb17c0158 (stale) Mirror 10: 0xb17c0158 (stale) Mirror 11: 0xb17c0158 (stale) Mirror 12: 0xb17c0158 (stale) Mirror 13: 0x9e58931d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f54d): Mirror 6: 0xb17c0158 (stale) Mirror 7: 0xb17c0158 (stale) Mirror 8: 0xb17c0158 (stale) Mirror 9: 0xb17c0158 (stale) Mirror 10: 0xb17c0158 (stale) Mirror 11: 0xb17c0158 (stale) Mirror 12: 0xb17c0158 (stale) Mirror 13: 0x9e58931d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f54d: skipped 000224 1767316324.889459 write 0x0060bf thru 0x00aefa (0x4e3c bytes) 000225 1767316324.904995 mapread 0x02d8f4 thru 0x035c56 (0x8363 bytes) 000226 1767316324.936362 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f54d): Mirror 7: 0xb17c0158 (stale) Mirror 8: 0xb17c0158 (stale) Mirror 9: 0xb17c0158 (stale) Mirror 10: 0xb17c0158 (stale) Mirror 11: 0xb17c0158 (stale) Mirror 12: 0xb17c0158 (stale) Mirror 13: 0x9e58931d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f54d): Mirror 7: 0xb17c0158 (stale) Mirror 8: 0xb17c0158 (stale) Mirror 9: 0xb17c0158 (stale) Mirror 10: 0xb17c0158 (stale) Mirror 11: 0xb17c0158 (stale) Mirror 12: 0xb17c0158 (stale) Mirror 13: 0x9e58931d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f54d: skipped 000227 1767316325.559294 write 0x00759f thru 0x01355e (0xbfc0 bytes) 000228 1767316325.572281 mapwrite 0x00be7e thru 0x0177e7 (0xb96a bytes) 000230 1767316325.635039 read 0x0395d3 thru 0x03f54c (0x5f7a bytes) 000231 1767316325.648119 trunc from 0x03f54d to 0x0250b2 (0x1a49a bytes) 000232 1767316325.674697 MIRROR_RESYNC 8 mirrors 000234 1767316327.062361 mapwrite 0x01fe63 thru 0x020778 (0x916 bytes) 000237 1767316327.220574 mapwrite 0x01c5e9 thru 0x01d127 (0xb3f bytes) 000238 1767316327.313462 write 0x017cbd thru 0x020056 (0x839a bytes) 000239 1767316327.339353 trunc from 0x0250b3 to 0x032e63 (0xddb1 bytes) 000241 1767316327.382649 mapwrite 0x00acec thru 0x0112ea (0x65ff bytes) 000242 1767316327.458438 mapwrite 0x03f44f thru 0x03ffff (0xbb1 bytes) 000244 1767316327.543040 write 0x02d0b9 thru 0x03438a (0x72d2 bytes) 000245 1767316327.557503 mapread 0x0134ab thru 0x015a16 (0x256c bytes) 000246 1767316327.559607 mapwrite 0x00575d thru 0x014b42 (0xf3e6 bytes) 000249 1767316327.630887 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x39faf3bb (stale) Mirror 9: 0x39faf3bb (stale) Mirror 10: 0x39faf3bb (stale) Mirror 11: 0x39faf3bb (stale) Mirror 12: 0x39faf3bb (stale) Mirror 13: 0x39faf3bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x39faf3bb (stale) Mirror 9: 0x39faf3bb (stale) Mirror 10: 0x39faf3bb (stale) Mirror 11: 0x39faf3bb (stale) Mirror 12: 0x39faf3bb (stale) Mirror 13: 0x39faf3bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000250 1767316328.064851 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x39faf3bb (stale) Mirror 9: 0x39faf3bb (stale) Mirror 10: 0x39faf3bb (stale) Mirror 11: 0x39faf3bb (stale) Mirror 12: 0x39faf3bb (stale) Mirror 13: 0x39faf3bb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x11ed101a Mirror 14: 0x11ed101a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000251 1767316328.995425 trunc from 0x040000 to 0x01ea0a (0x215f5 bytes) 000255 1767316329.111453 trunc from 0x01ea0b to 0x0259ae (0x6fa4 bytes) 000257 1767316329.147803 MIRROR_RESYNC 8 mirrors 000261 1767316330.465606 mapread 0x008de7 thru 0x00b5b7 (0x27d1 bytes) 000262 1767316330.486762 write 0x03c37a thru 0x03ffff (0x3c86 bytes) 000263 1767316330.588302 mapread 0x00db92 thru 0x01069c (0x2b0b bytes) 000264 1767316330.625178 mapwrite 0x02324e thru 0x029c51 (0x6a04 bytes) 000265 1767316330.708611 mapwrite 0x02a1e4 thru 0x0362b7 (0xc0d4 bytes) 000267 1767316330.822828 mapread 0x02d421 thru 0x038fac (0xbb8c bytes) 000269 1767316330.824197 read 0x001773 thru 0x00ec46 (0xd4d4 bytes) 000270 1767316330.833198 trunc from 0x040000 to 0x03c96f (0x3690 bytes) 000272 1767316330.857141 write 0x02ddf4 thru 0x03cc3f (0xee4c bytes) 000275 1767316330.860717 mapwrite 0x00f478 thru 0x014c5e (0x57e7 bytes) 000276 1767316330.935716 MIRROR_RESYNC 8 mirrors 000278 1767316332.284127 write 0x02e6fa thru 0x03762a (0x8f31 bytes) 000279 1767316332.421349 mapwrite 0x02f592 thru 0x0310ef (0x1b5e bytes) 000281 1767316332.473862 read 0x01204d thru 0x0135e7 (0x159b bytes) 000282 1767316332.484923 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 8: 0x1b8d04b1 (stale) Mirror 9: 0x1b8d04b1 (stale) Mirror 10: 0x1b8d04b1 (stale) Mirror 11: 0x1b8d04b1 (stale) Mirror 12: 0x1b8d04b1 (stale) Mirror 13: 0x1b8d04b1 (stale) Mirror 14: 0x1b8d04b1 (stale) CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 1: 0x2a787970 Mirror 15: 0x2a787970 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc40: skipped 000283 1767316333.470041 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 8: 0x1b8d04b1 (stale) Mirror 9: 0x1b8d04b1 (stale) Mirror 10: 0x1b8d04b1 (stale) Mirror 11: 0x1b8d04b1 (stale) Mirror 12: 0x1b8d04b1 (stale) Mirror 13: 0x1b8d04b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 8: 0x1b8d04b1 (stale) Mirror 9: 0x1b8d04b1 (stale) Mirror 10: 0x1b8d04b1 (stale) Mirror 11: 0x1b8d04b1 (stale) Mirror 12: 0x1b8d04b1 (stale) Mirror 13: 0x1b8d04b1 (stale) CRC-32 checksum value for chunk [0, 0x3cc40): Mirror 1: 0x2a787970 Mirror 15: 0x2a787970 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc40: skipped 000284 1767316334.060572 mapread 0x03c719 thru 0x03cc3f (0x527 bytes) 000285 1767316334.084048 mapwrite 0x03a9f1 thru 0x03ff4b (0x555b bytes) 000288 1767316334.300574 trunc from 0x03ff4c to 0x019f31 (0x2601a bytes) 000292 1767316334.328907 trunc from 0x019f32 to 0x0032d8 (0x16c59 bytes) 000295 1767316334.349580 mapread 0x003035 thru 0x0032d8 (0x2a4 bytes) 000296 1767316334.363331 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32d9): Mirror 9: 0xfe680af (stale) Mirror 10: 0xfe680af (stale) Mirror 11: 0xfe680af (stale) Mirror 12: 0xfe680af (stale) Mirror 13: 0xfe680af (stale) Mirror 15: 0xfe680af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d9): Mirror 9: 0xfe680af (stale) Mirror 10: 0xfe680af (stale) Mirror 11: 0xfe680af (stale) Mirror 12: 0xfe680af (stale) Mirror 13: 0xfe680af (stale) Mirror 15: 0xfe680af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d9: skipped 000299 1767316334.774981 read 0x002f2b thru 0x0032d8 (0x3ae bytes) 000300 1767316334.785148 read 0x001879 thru 0x0032d8 (0x1a60 bytes) 000301 1767316334.789632 mapwrite 0x035948 thru 0x03e858 (0x8f11 bytes) 000302 1767316334.884401 mapread 0x019626 thru 0x025a4c (0xc427 bytes) 000303 1767316334.885610 read 0x032255 thru 0x03e858 (0xc604 bytes) 000304 1767316334.886119 write 0x03f98f thru 0x03fb03 (0x175 bytes) 000305 1767316334.892987 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb04): Mirror 9: 0xe3411d90 (stale) Mirror 11: 0xe3411d90 (stale) Mirror 12: 0xe3411d90 (stale) Mirror 13: 0xe3411d90 (stale) Mirror 15: 0x727249c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb04): Mirror 9: 0xe3411d90 (stale) Mirror 11: 0xe3411d90 (stale) Mirror 12: 0xe3411d90 (stale) Mirror 13: 0xe3411d90 (stale) Mirror 15: 0x727249c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb04: skipped 000307 1767316335.629126 trunc from 0x03fb04 to 0x024d5a (0x1ada9 bytes) 000308 1767316335.645743 mapwrite 0x02a557 thru 0x03618b (0xbc35 bytes) 000309 1767316335.756133 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3618c): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3618c): Mirror 9: 0x89f580b7 (stale) Mirror 11: 0x89f580b7 (stale) Mirror 12: 0x89f580b7 (stale) Mirror 13: 0x89f580b7 (stale) Mirror 15: 0xac538705 (stale) CRC-32 checksum value for chunk [0, 0x3618c): Mirror 1: 0x83fa4a Mirror 16: 0x83fa4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3618c: skipped 000310 1767316337.289087 trunc from 0x03618c to 0x02d98a (0x8801 bytes) 000311 1767316337.426509 mapwrite 0x013c26 thru 0x0224f4 (0xe8cf bytes) 000312 1767316337.559800 read 0x0286cb thru 0x02d98a (0x52c0 bytes) 000314 1767316337.576420 mapwrite 0x016dce thru 0x01ca79 (0x5cac bytes) 000315 1767316337.639404 write 0x03722d thru 0x03ffff (0x8dd3 bytes) 000318 1767316337.644342 write 0x016ae5 thru 0x01e56c (0x7a88 bytes) 000319 1767316337.647210 read 0x00f71c thru 0x01f013 (0xf8f8 bytes) 000320 1767316337.648467 mapwrite 0x037b12 thru 0x03eaba (0x6fa9 bytes) 000321 1767316337.739954 mapread 0x018250 thru 0x01f0fb (0x6eac bytes) 000322 1767316337.741044 mapread 0x036472 thru 0x03ffff (0x9b8e bytes) 000323 1767316337.760523 MIRROR_RESYNC 7 mirrors 000324 1767316339.375596 mapread 0x01faf1 thru 0x029342 (0x9852 bytes) 000327 1767316339.395735 mapwrite 0x02af05 thru 0x034f40 (0xa03c bytes) 000328 1767316339.621545 read 0x017d15 thru 0x01c225 (0x4511 bytes) 000329 1767316339.634298 trunc from 0x040000 to 0x009b8f (0x36470 bytes) 000330 1767316339.662192 write 0x0010cd thru 0x005a0b (0x493f bytes) 000333 1767316339.675142 mapwrite 0x03a877 thru 0x03d50d (0x2c97 bytes) 000334 1767316339.751804 mapwrite 0x03d2b0 thru 0x03e0ac (0xdfd bytes) 000335 1767316339.821460 read 0x03b824 thru 0x03e0ac (0x2889 bytes) 000336 1767316339.824095 trunc from 0x03e0ad to 0x00fd9a (0x2e312 bytes) 000337 1767316339.845871 read 0x0032e4 thru 0x00ba58 (0x8775 bytes) 000338 1767316339.856407 write 0x02a7f9 thru 0x02db4a (0x3352 bytes) 000339 1767316339.863483 MIRROR_RESYNC 7 mirrors 000341 1767316341.280487 read 0x019e3e thru 0x022882 (0x8a45 bytes) 000342 1767316341.290893 write 0x00503b thru 0x00dff5 (0x8fbb bytes) 000343 1767316341.383034 read 0x003117 thru 0x0117bd (0xe6a7 bytes) 000344 1767316341.399903 mapread 0x02ae3a thru 0x02db4a (0x2d11 bytes) 000345 1767316341.402086 mapwrite 0x02dbba thru 0x0393e2 (0xb829 bytes) 000346 1767316341.518786 write 0x031218 thru 0x03ffff (0xede8 bytes) 000349 1767316341.525553 read 0x03b855 thru 0x03ffff (0x47ab bytes) 000350 1767316341.526681 mapwrite 0x007c2a thru 0x010749 (0x8b20 bytes) 000353 1767316341.595898 mapwrite 0x014bed thru 0x01d6bc (0x8ad0 bytes) 000355 1767316341.657762 mapwrite 0x03e493 thru 0x03ffff (0x1b6d bytes) 000356 1767316341.705988 trunc from 0x040000 to 0x010fcb (0x2f034 bytes) 000359 1767316341.730321 trunc from 0x010fcc to 0x021db0 (0x10de5 bytes) 000360 1767316341.751578 read 0x00e070 thru 0x0120d6 (0x4067 bytes) 000361 1767316341.763449 mapwrite 0x02d83b thru 0x03d5ff (0xfdc5 bytes) 000362 1767316341.874767 mapread 0x03c2c6 thru 0x03d5ff (0x133a bytes) 000363 1767316341.879113 mapwrite 0x018c75 thru 0x01d63e (0x49ca bytes) 000364 1767316341.936112 read 0x000c46 thru 0x0036d9 (0x2a94 bytes) 000365 1767316341.938831 mapwrite 0x01faf0 thru 0x02b06d (0xb57e bytes) 000366 1767316342.013178 write 0x0327da thru 0x03e2da (0xbb01 bytes) 000369 1767316342.035931 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 9: 0x72ad252 (stale) Mirror 11: 0x72ad252 (stale) Mirror 12: 0x72ad252 (stale) Mirror 13: 0x72ad252 (stale) Mirror 15: 0x72ad252 (stale) Mirror 16: 0x72ad252 (stale) CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 1: 0x7065119c Mirror 17: 0x7065119c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2db: skipped 000371 1767316343.124271 mapwrite 0x0138d7 thru 0x01d9f0 (0xa11a bytes) 000375 1767316343.318787 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 9: 0x72ad252 (stale) Mirror 11: 0x72ad252 (stale) Mirror 12: 0x72ad252 (stale) Mirror 13: 0x72ad252 (stale) Mirror 15: 0x72ad252 (stale) Mirror 16: 0x72ad252 (stale) Mirror 17: 0x7065119c (stale) CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 1: 0x3196def9 Mirror 18: 0x3196def9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2db: skipped 000376 1767316344.732719 read 0x00f9f5 thru 0x0192f7 (0x9903 bytes) 000377 1767316344.767317 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 11: 0x72ad252 (stale) Mirror 12: 0x72ad252 (stale) Mirror 13: 0x72ad252 (stale) Mirror 15: 0x72ad252 (stale) Mirror 16: 0x72ad252 (stale) Mirror 17: 0x7065119c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 11: 0x72ad252 (stale) Mirror 12: 0x72ad252 (stale) Mirror 13: 0x72ad252 (stale) Mirror 15: 0x72ad252 (stale) Mirror 16: 0x72ad252 (stale) Mirror 17: 0x7065119c (stale) CRC-32 checksum value for chunk [0, 0x3e2db): Mirror 1: 0x3196def9 Mirror 18: 0x3196def9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2db: skipped 000378 1767316345.830829 mapwrite 0x037131 thru 0x03ffff (0x8ecf bytes) 000379 1767316346.073513 trunc from 0x040000 to 0x0303bd (0xfc42 bytes) 000382 1767316346.120510 trunc from 0x0303be to 0x01d9bd (0x12a00 bytes) 000383 1767316346.181153 trunc from 0x01d9be to 0x002e4a (0x1ab73 bytes) 000385 1767316346.244513 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2e4b): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e4b): Mirror 11: 0xbf657416 (stale) Mirror 12: 0xbf657416 (stale) Mirror 13: 0xbf657416 (stale) Mirror 15: 0xbf657416 (stale) Mirror 16: 0xbf657416 (stale) Mirror 17: 0xbf657416 (stale) Mirror 18: 0xbf657416 (stale) CRC-32 checksum value for chunk [0, 0x2e4b): Mirror 1: 0xbf657416 Mirror 19: 0xbf657416 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e4b: skipped 000386 1767316347.487563 write 0x007b52 thru 0x009c8c (0x213b bytes) 000387 1767316347.614478 read 0x006aa5 thru 0x007b1b (0x1077 bytes) 000388 1767316347.631696 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9c8d): Mirror 11: 0x6d8f7668 (stale) Mirror 12: 0x6d8f7668 (stale) Mirror 13: 0x6d8f7668 (stale) Mirror 16: 0x6d8f7668 (stale) Mirror 17: 0x57ef77 (stale) Mirror 18: 0x57ef77 (stale) Mirror 19: 0x96cc30db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9c8d): Mirror 11: 0x6d8f7668 (stale) Mirror 12: 0x6d8f7668 (stale) Mirror 13: 0x6d8f7668 (stale) Mirror 16: 0x6d8f7668 (stale) Mirror 17: 0x57ef77 (stale) Mirror 18: 0x57ef77 (stale) Mirror 19: 0x96cc30db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9c8d: skipped 000389 1767316348.107777 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9c8d): Mirror 12: 0x6d8f7668 (stale) Mirror 13: 0x6d8f7668 (stale) Mirror 16: 0x6d8f7668 (stale) Mirror 17: 0x57ef77 (stale) Mirror 18: 0x57ef77 (stale) Mirror 19: 0x96cc30db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9c8d): Mirror 12: 0x6d8f7668 (stale) Mirror 13: 0x6d8f7668 (stale) Mirror 16: 0x6d8f7668 (stale) Mirror 17: 0x57ef77 (stale) Mirror 18: 0x57ef77 (stale) Mirror 19: 0x96cc30db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9c8d: skipped 000392 1767316348.571246 write 0x00ccf8 thru 0x00e1bb (0x14c4 bytes) 000394 1767316348.580650 mapread 0x0046a5 thru 0x006162 (0x1abe bytes) 000396 1767316348.606490 trunc from 0x00e1bc to 0x032c9d (0x24ae2 bytes) 000397 1767316348.634729 trunc from 0x032c9e to 0x02c63e (0x665f bytes) 000398 1767316348.664477 mapread 0x027c55 thru 0x029cc3 (0x206f bytes) 000399 1767316348.678339 trunc from 0x02c63f to 0x017fa9 (0x14695 bytes) 000401 1767316348.721077 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x17faa): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17faa): Mirror 12: 0x2c594bff (stale) Mirror 13: 0x2c594bff (stale) Mirror 16: 0x2c594bff (stale) Mirror 17: 0xa1154899 (stale) Mirror 18: 0xbf9da772 (stale) Mirror 19: 0xde4fa53a (stale) CRC-32 checksum value for chunk [0, 0x17faa): Mirror 1: 0xc8a0e52c Mirror 20: 0xc8a0e52c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17faa: skipped 000402 1767316349.988447 write 0x00a9b5 thru 0x00c85e (0x1eaa bytes) 000403 1767316350.097969 read 0x017208 thru 0x017fa9 (0xda2 bytes) 000405 1767316350.105078 trunc from 0x017faa to 0x011f38 (0x6071 bytes) 000406 1767316350.132445 mapread 0x00aec0 thru 0x00d940 (0x2a81 bytes) 000407 1767316350.143157 write 0x01c3c1 thru 0x0277a4 (0xb3e4 bytes) 000408 1767316350.149347 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x277a5): Mirror 12: 0xd8f4cb41 (stale) Mirror 13: 0xd8f4cb41 (stale) Mirror 16: 0xd8f4cb41 (stale) Mirror 17: 0x7df71322 (stale) Mirror 18: 0xf474303c (stale) Mirror 19: 0xb235e017 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x277a5): Mirror 12: 0xd8f4cb41 (stale) Mirror 13: 0xd8f4cb41 (stale) Mirror 16: 0xd8f4cb41 (stale) Mirror 17: 0x7df71322 (stale) Mirror 18: 0xf474303c (stale) Mirror 19: 0xb235e017 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x277a5: skipped 000412 1767316350.654996 MIRROR_RESYNC 7 mirrors 000413 1767316351.850737 trunc from 0x0277a5 to 0x0318ef (0xa14b bytes) 000414 1767316351.908916 read 0x025b80 thru 0x028bee (0x306f bytes) 000415 1767316351.917127 mapwrite 0x02b92b thru 0x02c913 (0xfe9 bytes) 000416 1767316351.973188 write 0x01edf0 thru 0x02c3ce (0xd5df bytes) 000419 1767316351.986084 mapread 0x01792d thru 0x0257d2 (0xdea6 bytes) 000420 1767316351.987514 read 0x00dc20 thru 0x01677f (0x8b60 bytes) 000421 1767316351.998104 mapwrite 0x03013c thru 0x033652 (0x3517 bytes) 000423 1767316352.077759 mapwrite 0x001876 thru 0x00b59b (0x9d26 bytes) 000424 1767316352.120700 mapwrite 0x018d95 thru 0x027110 (0xe37c bytes) 000425 1767316352.177691 mapread 0x0093ec thru 0x012e56 (0x9a6b bytes) 000426 1767316352.178746 mapwrite 0x035b96 thru 0x037ea9 (0x2314 bytes) 000428 1767316352.266069 trunc from 0x037eaa to 0x034543 (0x3966 bytes) 000430 1767316352.295412 read 0x034387 thru 0x034543 (0x1bd bytes) 000431 1767316352.296705 read 0x02ea1b thru 0x034543 (0x5b29 bytes) 000437 1767316352.305975 MIRROR_RESYNC 7 mirrors 000438 1767316353.255115 MIRROR_RESYNC 7 mirrors 000439 1767316353.966951 MIRROR_RESYNC 7 mirrors 000443 1767316354.504677 mapread 0x001103 thru 0x00e698 (0xd596 bytes) 000444 1767316354.553751 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) 000447 1767316355.026618 mapwrite 0x026a4e thru 0x029855 (0x2e08 bytes) 000448 1767316355.265180 trunc from 0x034544 to 0x0096b0 (0x2ae93 bytes) 000449 1767316355.364689 mapwrite 0x020d48 thru 0x0293a9 (0x8662 bytes) 000450 1767316355.502665 write 0x018192 thru 0x01ea5c (0x68cb bytes) 000451 1767316355.520139 read 0x012fb2 thru 0x01a66b (0x76ba bytes) 000453 1767316355.524137 write 0x01f1ed thru 0x0215ae (0x23c2 bytes) 000454 1767316355.534017 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x293aa): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293aa): Mirror 12: 0xb3410023 (stale) Mirror 13: 0xb3410023 (stale) Mirror 17: 0xb3410023 (stale) Mirror 18: 0xb3410023 (stale) Mirror 19: 0xb3410023 (stale) CRC-32 checksum value for chunk [0, 0x293aa): Mirror 1: 0xaf4b7399 Mirror 20: 0xaf4b7399 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293aa: skipped 000456 1767316356.484569 read 0x01d0e8 thru 0x01ebab (0x1ac4 bytes) 000457 1767316356.494663 mapread 0x00a46a thru 0x013ea2 (0x9a39 bytes) 000459 1767316356.534610 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293aa): Mirror 12: 0xb3410023 (stale) Mirror 17: 0xb3410023 (stale) Mirror 18: 0xb3410023 (stale) Mirror 19: 0xb3410023 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293aa): Mirror 12: 0xb3410023 (stale) Mirror 17: 0xb3410023 (stale) Mirror 18: 0xb3410023 (stale) Mirror 19: 0xb3410023 (stale) CRC-32 checksum value for chunk [0, 0x293aa): Mirror 1: 0xaf4b7399 Mirror 20: 0xaf4b7399 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293aa: skipped 000460 1767316357.165063 mapread 0x0284d7 thru 0x0293a9 (0xed3 bytes) 000462 1767316357.203128 write 0x03504f thru 0x03a62f (0x55e1 bytes) 000463 1767316357.297822 MIRROR_RESYNC 6 mirrors 000467 1767316358.532046 read 0x029a12 thru 0x02fdd2 (0x63c1 bytes) 000469 1767316358.556737 mapread 0x00e555 thru 0x012fae (0x4a5a bytes) 000470 1767316358.590294 trunc from 0x03a630 to 0x003b59 (0x36ad6 bytes) 000471 1767316358.683801 trunc from 0x003b5a to 0x008d34 (0x51db bytes) 000474 1767316358.712681 mapwrite 0x0318a0 thru 0x03d70e (0xbe6f bytes) 000477 1767316358.790124 write 0x03125b thru 0x0357cc (0x4572 bytes) 000478 1767316358.793372 write 0x00ef20 thru 0x01763c (0x871d bytes) 000480 1767316358.825597 mapread 0x00f7b7 thru 0x010bdf (0x1429 bytes) 000482 1767316358.837114 read 0x021124 thru 0x029a42 (0x891f bytes) 000483 1767316358.868964 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d70f): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d70f): Mirror 12: 0x179866a6 (stale) Mirror 17: 0x179866a6 (stale) Mirror 18: 0x179866a6 (stale) Mirror 19: 0x179866a6 (stale) Mirror 20: 0x179866a6 (stale) CRC-32 checksum value for chunk [0, 0x3d70f): Mirror 1: 0x3d104b22 Mirror 21: 0x3d104b22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d70f: skipped 000488 1767316360.231029 mapwrite 0x030637 thru 0x03a247 (0x9c11 bytes) 000489 1767316360.399814 write 0x020e94 thru 0x024b83 (0x3cf0 bytes) 000490 1767316360.412870 mapread 0x036306 thru 0x03a7db (0x44d6 bytes) 000491 1767316360.414184 read 0x030957 thru 0x031a99 (0x1143 bytes) 000493 1767316360.419658 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d70f): Mirror 12: 0x179866a6 (stale) Mirror 18: 0x179866a6 (stale) Mirror 19: 0x179866a6 (stale) Mirror 20: 0x179866a6 (stale) Mirror 21: 0x3d104b22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d70f): Mirror 12: 0x179866a6 (stale) Mirror 18: 0x179866a6 (stale) Mirror 19: 0x179866a6 (stale) Mirror 20: 0x179866a6 (stale) Mirror 21: 0x3d104b22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d70f: skipped 000494 1767316360.869806 MIRROR_RESYNC 6 mirrors 000496 1767316361.904860 MIRROR_RESYNC 6 mirrors 000497 1767316362.566787 mapwrite 0x01b8ea thru 0x020676 (0x4d8d bytes) 000498 1767316362.735642 trunc from 0x03d70f to 0x01b898 (0x21e76 bytes) 000499 1767316362.763829 mapwrite 0x030694 thru 0x032fc6 (0x2933 bytes) truncating to largest ever: 0x3fcc1 000503 1767316362.855066 trunc from 0x032fc7 to 0x03fcc0 (0xccfa bytes) 000504 1767316362.871736 write 0x007996 thru 0x00a0af (0x271a bytes) 000505 1767316362.906039 write 0x02b268 thru 0x030926 (0x56bf bytes) 000506 1767316362.907149 write 0x00ab63 thru 0x00b943 (0xde1 bytes) 000508 1767316362.908399 trunc from 0x03fcc1 to 0x015200 (0x2aac0 bytes) 000511 1767316362.939282 mapwrite 0x03d1af thru 0x03ffff (0x2e51 bytes) 000512 1767316363.034317 trunc from 0x040000 to 0x03f3aa (0xc55 bytes) 000513 1767316363.063337 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f3ab): Mirror 12: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3ab): Mirror 12: 0x1cd7039e (stale) Mirror 18: 0x1cd7039e (stale) Mirror 19: 0x1cd7039e (stale) Mirror 20: 0x1cd7039e (stale) Mirror 21: 0x1cd7039e (stale) CRC-32 checksum value for chunk [0, 0x3f3ab): Mirror 1: 0x47acb31 Mirror 22: 0x47acb31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3ab: skipped 000514 1767316364.203280 mapread 0x003d06 thru 0x00a629 (0x6924 bytes) 000517 1767316364.297170 mapwrite 0x026eb5 thru 0x03038a (0x94d6 bytes) 000518 1767316364.662658 read 0x0332ac thru 0x03cfdb (0x9d30 bytes) 000520 1767316364.679282 trunc from 0x03f3ab to 0x026cd8 (0x186d2 bytes) 000521 1767316364.717165 read 0x0098a9 thru 0x01440f (0xab67 bytes) 000523 1767316364.765278 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26cd9): Mirror 12: 0x7679b7c (stale) Mirror 19: 0x7679b7c (stale) Mirror 20: 0x7679b7c (stale) Mirror 21: 0x7679b7c (stale) Mirror 22: 0xb2b4ea99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26cd9): Mirror 12: 0x7679b7c (stale) Mirror 19: 0x7679b7c (stale) Mirror 20: 0x7679b7c (stale) Mirror 21: 0x7679b7c (stale) Mirror 22: 0xb2b4ea99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26cd9: skipped 000525 1767316365.197535 read 0x00685e thru 0x012bfa (0xc39d bytes) 000526 1767316365.207662 mapwrite 0x010d3c thru 0x01f178 (0xe43d bytes) 000527 1767316365.295082 trunc from 0x026cd9 to 0x03cd2b (0x16053 bytes) 000529 1767316365.313781 write 0x009fd2 thru 0x00cbb3 (0x2be2 bytes) 000530 1767316365.314747 read 0x01e795 thru 0x01fef9 (0x1765 bytes) 000531 1767316365.315310 mapread 0x01ce76 thru 0x0284ab (0xb636 bytes) 000533 1767316365.340878 trunc from 0x03cd2c to 0x01fd99 (0x1cf92 bytes) 000534 1767316365.367946 mapread 0x004ea2 thru 0x014ad6 (0xfc35 bytes) 000535 1767316365.368983 write 0x01cdae thru 0x01fe16 (0x3069 bytes) 000536 1767316365.379504 MIRROR_RESYNC 6 mirrors 000537 1767316366.472848 mapwrite 0x01c555 thru 0x021d83 (0x582f bytes) 000541 1767316366.656549 MIRROR_RESYNC 6 mirrors 000542 1767316367.552705 mapwrite 0x0125a2 thru 0x01ca8b (0xa4ea bytes) 000543 1767316367.726769 read 0x01f237 thru 0x021d83 (0x2b4d bytes) 000545 1767316367.727876 write 0x016947 thru 0x02521f (0xe8d9 bytes) 000546 1767316367.734202 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25220): Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25220): Mirror 12: 0xd867c195 (stale) Mirror 19: 0xd867c195 (stale) Mirror 20: 0xd867c195 (stale) Mirror 21: 0xd867c195 (stale) Mirror 22: 0xd867c195 (stale) CRC-32 checksum value for chunk [0, 0x25220): Mirror 1: 0x93e83b0 Mirror 23: 0x93e83b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25220: skipped 000547 1767316368.577984 mapread 0x0142fc thru 0x0234fb (0xf200 bytes) 000548 1767316368.630821 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25220): Mirror 12: 0xd867c195 (stale) Mirror 19: 0xd867c195 (stale) Mirror 20: 0xd867c195 (stale) Mirror 21: 0xd867c195 (stale) Mirror 22: 0xd867c195 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25220): Mirror 12: 0xd867c195 (stale) Mirror 19: 0xd867c195 (stale) Mirror 20: 0xd867c195 (stale) Mirror 21: 0xd867c195 (stale) Mirror 22: 0xd867c195 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25220: skipped 000549 1767316369.017071 mapread 0x00f4bb thru 0x012494 (0x2fda bytes) 000550 1767316369.033564 mapread 0x003790 thru 0x009697 (0x5f08 bytes) 000552 1767316369.048632 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25220): Mirror 19: 0xd867c195 (stale) Mirror 20: 0xd867c195 (stale) Mirror 21: 0xd867c195 (stale) Mirror 22: 0xd867c195 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25220): Mirror 19: 0xd867c195 (stale) Mirror 20: 0xd867c195 (stale) Mirror 21: 0xd867c195 (stale) Mirror 22: 0xd867c195 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25220: skipped 000553 1767316369.489968 mapwrite 0x024abb thru 0x02511c (0x662 bytes) 000558 1767316369.572511 trunc from 0x025220 to 0x00c5a7 (0x18c78 bytes) 000559 1767316369.595791 write 0x01e335 thru 0x021946 (0x3612 bytes) 000560 1767316369.604418 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21947): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21947): Mirror 19: 0x3865a973 (stale) Mirror 20: 0x3865a973 (stale) Mirror 21: 0x3865a973 (stale) Mirror 22: 0x3865a973 (stale) CRC-32 checksum value for chunk [0, 0x21947): Mirror 1: 0x3d1ba9a7 Mirror 23: 0x3d1ba9a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21947: skipped 000563 1767316370.370857 write 0x037c25 thru 0x03ffff (0x83db bytes) 000564 1767316370.461879 MIRROR_RESYNC 6 mirrors 000565 1767316371.662065 trunc from 0x040000 to 0x03204a (0xdfb5 bytes) 000569 1767316371.743840 write 0x039bba thru 0x03ab23 (0xf6a bytes) 000572 1767316371.747813 read 0x009fe5 thru 0x017740 (0xd75c bytes) 000573 1767316371.760037 read 0x002bcf thru 0x003b39 (0xf6b bytes) 000576 1767316371.769564 trunc from 0x03ab24 to 0x01df6b (0x1cbb8 bytes) 000578 1767316371.792266 mapread 0x00a1cb thru 0x00d9a9 (0x37df bytes) 000581 1767316371.793398 mapread 0x00bad5 thru 0x01837b (0xc8a7 bytes) 000584 1767316371.798014 MIRROR_RESYNC 6 mirrors 000585 1767316372.661026 write 0x0292d5 thru 0x02dbd3 (0x48ff bytes) 000587 1767316372.694515 mapread 0x0023c1 thru 0x008ed4 (0x6b14 bytes) 000588 1767316372.706369 mapread 0x026c77 thru 0x029b8d (0x2f17 bytes) 000589 1767316372.711608 read 0x02abd0 thru 0x02dbd3 (0x3004 bytes) 000590 1767316372.711998 trunc from 0x02dbd4 to 0x02e9ad (0xdda bytes) 000591 1767316372.724657 mapwrite 0x00fad2 thru 0x019b24 (0xa053 bytes) 000592 1767316372.777973 mapread 0x002900 thru 0x00be45 (0x9546 bytes) 000594 1767316372.780308 mapread 0x01a3e7 thru 0x0230cd (0x8ce7 bytes) 000595 1767316372.790937 mapwrite 0x000026 thru 0x001b2c (0x1b07 bytes) 000596 1767316372.852307 write 0x0016d7 thru 0x002549 (0xe73 bytes) 000597 1767316372.853578 read 0x003c8a thru 0x00f858 (0xbbcf bytes) 000599 1767316372.856151 trunc from 0x02e9ae to 0x0100be (0x1e8ef bytes) 000600 1767316372.885148 mapread 0x003493 thru 0x0100be (0xcc2c bytes) 000601 1767316372.887243 mapwrite 0x00f25a thru 0x013b49 (0x48f0 bytes) 000606 1767316372.987734 mapread 0x009e90 thru 0x00df2b (0x409c bytes) 000607 1767316372.989049 trunc from 0x013b4a to 0x02a2ba (0x16771 bytes) 000608 1767316373.008846 read 0x01b9da thru 0x024944 (0x8f6b bytes) 000609 1767316373.025908 mapread 0x00896e thru 0x0149cc (0xc05f bytes) 000614 1767316373.029929 mapread 0x01c765 thru 0x02a2ba (0xdb56 bytes) 000616 1767316373.031331 trunc from 0x02a2bb to 0x033e0d (0x9b53 bytes) 000618 1767316373.064146 write 0x010b10 thru 0x012dfb (0x22ec bytes) 000619 1767316373.069881 trunc from 0x033e0e to 0x003f32 (0x2fedb bytes) 000620 1767316373.096180 write 0x01833e thru 0x01f382 (0x7045 bytes) 000621 1767316373.097515 read 0x01a0c1 thru 0x01f382 (0x52c2 bytes) 000622 1767316373.098041 trunc from 0x01f383 to 0x025075 (0x5cf3 bytes) 000623 1767316373.121783 mapwrite 0x01aaa0 thru 0x021585 (0x6ae6 bytes) 000624 1767316373.194046 trunc from 0x025076 to 0x02c0ae (0x7039 bytes) 000625 1767316373.223117 write 0x03f417 thru 0x03ffff (0xbe9 bytes) 000629 1767316373.227776 mapwrite 0x015208 thru 0x018cd9 (0x3ad2 bytes) 000630 1767316373.304116 mapread 0x02f5d0 thru 0x02fb19 (0x54a bytes) 000631 1767316373.318097 read 0x015a31 thru 0x01edf0 (0x93c0 bytes) 000633 1767316373.319207 write 0x015b10 thru 0x01b40c (0x58fd bytes) 000635 1767316373.321608 mapread 0x024e11 thru 0x028ee6 (0x40d6 bytes) 000636 1767316373.322849 mapwrite 0x01dde4 thru 0x027a0d (0x9c2a bytes) 000637 1767316373.386848 write 0x01bfef thru 0x01d831 (0x1843 bytes) 000638 1767316373.388036 mapread 0x030a74 thru 0x039dc1 (0x934e bytes) 000639 1767316373.389942 write 0x02985b thru 0x0359b8 (0xc15e bytes) 000640 1767316373.391926 mapread 0x00410d thru 0x00bf7a (0x7e6e bytes) 000641 1767316373.400922 MIRROR_RESYNC 6 mirrors 000647 1767316374.467101 trunc from 0x040000 to 0x016bf3 (0x2940c bytes) 000648 1767316374.541473 write 0x002f40 thru 0x00f98a (0xca4b bytes) 000650 1767316374.551574 mapwrite 0x0143f9 thru 0x016836 (0x243e bytes) 000651 1767316374.616692 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16bf4): Mirror 19: 0x7a19579b (stale) Mirror 21: 0x7a19579b (stale) Mirror 22: 0x7a19579b (stale) Mirror 23: 0x7a19579b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16bf4): Mirror 19: 0x7a19579b (stale) Mirror 21: 0x7a19579b (stale) Mirror 22: 0x7a19579b (stale) Mirror 23: 0x7a19579b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16bf4: skipped 000653 1767316374.978054 mapread 0x015bf9 thru 0x016bf3 (0xffb bytes) 000656 1767316374.991294 trunc from 0x016bf4 to 0x036396 (0x1f7a3 bytes) 000661 1767316375.012753 mapread 0x024df0 thru 0x02b5f8 (0x6809 bytes) 000665 1767316375.030411 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36397): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36397): Mirror 19: 0x5e045854 (stale) Mirror 21: 0x5e045854 (stale) Mirror 22: 0x5e045854 (stale) Mirror 23: 0x5e045854 (stale) CRC-32 checksum value for chunk [0, 0x36397): Mirror 1: 0xa1b23be5 Mirror 24: 0xa1b23be5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36397: skipped 000667 1767316376.179750 read 0x004d4d thru 0x008642 (0x38f6 bytes) 000668 1767316376.207853 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36397): Mirror 19: 0x5e045854 (stale) Mirror 21: 0x5e045854 (stale) Mirror 22: 0x5e045854 (stale) Mirror 23: 0x5e045854 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36397): Mirror 19: 0x5e045854 (stale) Mirror 21: 0x5e045854 (stale) Mirror 22: 0x5e045854 (stale) Mirror 23: 0x5e045854 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36397: skipped 000670 1767316376.708045 trunc from 0x036397 to 0x00341f (0x32f77 bytes) 000671 1767316376.756506 read 0x001963 thru 0x002118 (0x7b6 bytes) 000672 1767316376.764952 read 0x000084 thru 0x00341f (0x339c bytes) 000675 1767316376.777260 mapread 0x0018ce thru 0x00341f (0x1b52 bytes) 000676 1767316376.781250 write 0x0298f9 thru 0x037d73 (0xe47b bytes) 000677 1767316376.785457 write 0x032409 thru 0x033e4b (0x1a43 bytes) 000679 1767316376.796093 trunc from 0x037d74 to 0x007447 (0x3092c bytes) 000683 1767316376.837592 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7448): Mirror 21: 0xb2637c31 (stale) Mirror 22: 0xb2637c31 (stale) Mirror 23: 0xb2637c31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7448): Mirror 21: 0xb2637c31 (stale) Mirror 22: 0xb2637c31 (stale) Mirror 23: 0xb2637c31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7448: skipped 000689 1767316377.249591 mapwrite 0x00dc50 thru 0x01c8aa (0xec5b bytes) 000690 1767316377.361450 write 0x01b787 thru 0x0278d1 (0xc14b bytes) 000694 1767316377.366025 write 0x00e707 thru 0x013fe2 (0x58dc bytes) 000695 1767316377.367435 mapread 0x01c459 thru 0x01dd46 (0x18ee bytes) 000696 1767316377.369855 trunc from 0x0278d2 to 0x03146a (0x9b99 bytes) 000698 1767316377.391889 write 0x030b3a thru 0x03c426 (0xb8ed bytes) 000699 1767316377.402524 read 0x037c63 thru 0x03c426 (0x47c4 bytes) 000700 1767316377.403885 trunc from 0x03c427 to 0x031185 (0xb2a1 bytes) 000701 1767316377.427985 mapread 0x00a3e5 thru 0x012fde (0x8bfa bytes) 000705 1767316377.430211 mapread 0x0008ff thru 0x002fc1 (0x26c3 bytes) 000706 1767316377.443823 trunc from 0x031186 to 0x01148b (0x1fcfa bytes) 000707 1767316377.467968 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1148c): Mirror 22: 0xf75a1168 (stale) Mirror 23: 0xf75a1168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1148c): Mirror 22: 0xf75a1168 (stale) Mirror 23: 0xf75a1168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1148c: skipped 000708 1767316377.805778 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1148c): Mirror 23: 0xf75a1168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1148c): Mirror 23: 0xf75a1168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1148c: skipped 000710 1767316378.104075 mapread 0x00f5ae thru 0x01148b (0x1ede bytes) 000712 1767316378.124689 mapread 0x0068df thru 0x00e00d (0x772f bytes) 000713 1767316378.133468 mapwrite 0x036858 thru 0x0376f9 (0xea2 bytes) 000715 1767316378.212929 write 0x035d09 thru 0x03656a (0x862 bytes) 000716 1767316378.215057 read 0x0341f2 thru 0x0376f9 (0x3508 bytes) 000717 1767316378.216004 mapwrite 0x0036a9 thru 0x0091ca (0x5b22 bytes) 000718 1767316378.261993 read 0x003bce thru 0x012788 (0xebbb bytes) 000720 1767316378.273825 mapread 0x036faa thru 0x0376f9 (0x750 bytes) 000721 1767316378.274961 mapwrite 0x008d4e thru 0x014934 (0xbbe7 bytes) 000722 1767316378.327927 read 0x01a80e thru 0x024269 (0x9a5c bytes) 000723 1767316378.329050 trunc from 0x0376fa to 0x012e39 (0x248c0 bytes) 000725 1767316378.350999 read 0x001062 thru 0x0049ed (0x398c bytes) 000729 1767316378.355570 write 0x003689 thru 0x008386 (0x4cfe bytes) 000730 1767316378.357108 mapread 0x00f78a thru 0x012e39 (0x36b0 bytes) 000731 1767316378.358986 mapread 0x007a88 thru 0x010e2e (0x93a7 bytes) 000734 1767316378.362199 trunc from 0x012e3a to 0x0137fb (0x9c2 bytes) 000737 1767316378.380965 mapread 0x00be7b thru 0x00c339 (0x4bf bytes) 000739 1767316378.383185 read 0x010c53 thru 0x0137fb (0x2ba9 bytes) 000742 1767316378.394708 write 0x009091 thru 0x00b6c3 (0x2633 bytes) 000743 1767316378.397078 read 0x008fa4 thru 0x00c0d0 (0x312d bytes) 000744 1767316378.403583 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x137fc): Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x137fc): Mirror 23: 0xfd0edac1 (stale) CRC-32 checksum value for chunk [0, 0x137fc): Mirror 1: 0x23a67d09 Mirror 24: 0x23a67d09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x137fc: skipped 000745 1767316379.148146 trunc from 0x0137fc to 0x00430c (0xf4ef bytes) 000747 1767316379.253838 write 0x03f58d thru 0x03ffff (0xa73 bytes) 000748 1767316379.255509 mapwrite 0x02c05f thru 0x034b48 (0x8aea bytes) 000754 1767316379.331048 read 0x024fd7 thru 0x02ab0d (0x5b37 bytes) 000755 1767316379.338693 trunc from 0x040000 to 0x025360 (0x1ac9f bytes) 000756 1767316379.374042 write 0x03d086 thru 0x03ffff (0x2f7a bytes) 000758 1767316379.375833 write 0x028617 thru 0x0373c0 (0xedaa bytes) 000759 1767316379.396712 write 0x02ef2c thru 0x030b9f (0x1c74 bytes) 000760 1767316379.401918 MIRROR_RESYNC 3 mirrors 000764 1767316380.548028 MIRROR_RESYNC 3 mirrors 000765 1767316381.019780 mapwrite 0x0389d8 thru 0x03ffff (0x7628 bytes) 000766 1767316381.197291 read 0x020f55 thru 0x02d0e3 (0xc18f bytes) 000767 1767316381.211366 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x703dc55f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x703dc55f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000768 1767316381.511422 write 0x00fd8e thru 0x01adaa (0xb01d bytes) 000770 1767316381.524545 mapread 0x00e9b3 thru 0x015493 (0x6ae1 bytes) 000771 1767316381.533964 read 0x01c20b thru 0x01fbe3 (0x39d9 bytes) 000772 1767316381.541882 trunc from 0x040000 to 0x03b47f (0x4b80 bytes) 000774 1767316381.576542 read 0x016085 thru 0x01f898 (0x9814 bytes) 000775 1767316381.589900 mapwrite 0x01fc9c thru 0x026dd5 (0x713a bytes) 000776 1767316381.648474 trunc from 0x03b480 to 0x024131 (0x1734e bytes) 000778 1767316381.674746 mapwrite 0x0068aa thru 0x010b6b (0xa2c2 bytes) 000779 1767316381.724603 MIRROR_RESYNC 2 mirrors 000781 1767316382.426648 trunc from 0x024132 to 0x01156b (0x12bc6 bytes) 000782 1767316382.524400 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1156c): Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1156c): Mirror 24: 0x3eb59316 (stale) CRC-32 checksum value for chunk [0, 0x1156c): Mirror 1: 0x3eb59316 Mirror 25: 0x3eb59316 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1156c: skipped 000783 1767316383.810552 mapwrite 0x031085 thru 0x03d63e (0xc5ba bytes) 000787 1767316384.001485 write 0x00a981 thru 0x011d26 (0x73a6 bytes) 000788 1767316384.014989 trunc from 0x03d63f to 0x03499d (0x8ca1 bytes) 000792 1767316384.041639 mapwrite 0x02c690 thru 0x034b73 (0x84e4 bytes) 000794 1767316384.136951 trunc from 0x034b74 to 0x01e287 (0x168ec bytes) 000795 1767316384.166016 write 0x028867 thru 0x0339ae (0xb148 bytes) 000800 1767316384.173969 mapread 0x01e562 thru 0x02477e (0x621d bytes) 000804 1767316384.188117 mapwrite 0x0338a1 thru 0x0366b3 (0x2e13 bytes) 000805 1767316384.292887 trunc from 0x0366b4 to 0x0090f8 (0x2d5bb bytes) 000806 1767316384.323790 write 0x008f97 thru 0x011888 (0x88f2 bytes) 000807 1767316384.326929 trunc from 0x011889 to 0x01d4e4 (0xbc5c bytes) 000808 1767316384.358930 trunc from 0x01d4e5 to 0x029399 (0xbeb5 bytes) 000809 1767316384.382892 trunc from 0x02939a to 0x015637 (0x13d62 bytes) 000811 1767316384.419279 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15638): Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15638): Mirror 24: 0xd168edf3 (stale) Mirror 25: 0x8f2c24a1 (stale) CRC-32 checksum value for chunk [0, 0x15638): Mirror 1: 0xcd055036 Mirror 26: 0xcd055036 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15638: skipped 000814 1767316385.483514 trunc from 0x015638 to 0x00b6b5 (0x9f82 bytes) 000815 1767316385.558451 read 0x00ae58 thru 0x00b6b5 (0x85e bytes) 000816 1767316385.564767 trunc from 0x00b6b6 to 0x002ed9 (0x87dc bytes) 000818 1767316385.594708 mapwrite 0x0098e5 thru 0x013107 (0x9823 bytes) 000820 1767316385.696472 trunc from 0x013108 to 0x02d0ed (0x19fe6 bytes) 000821 1767316385.723810 write 0x027190 thru 0x028b0a (0x197b bytes) 000823 1767316385.735394 trunc from 0x02d0ee to 0x0023fa (0x2acf3 bytes) 000825 1767316385.758754 read 0x0019ac thru 0x0023fa (0xa4f bytes) 000826 1767316385.766519 mapwrite 0x01ac28 thru 0x02883a (0xdc13 bytes) 000830 1767316385.868827 write 0x009394 thru 0x00d494 (0x4101 bytes) 000832 1767316385.871135 mapread 0x00c588 thru 0x01667c (0xa0f5 bytes) 000833 1767316385.874031 mapwrite 0x01d8b3 thru 0x02b4a1 (0xdbef bytes) 000837 1767316385.973697 trunc from 0x02b4a2 to 0x03c58e (0x110ed bytes) 000838 1767316385.995889 read 0x0199d5 thru 0x020b18 (0x7144 bytes) 000839 1767316385.996875 read 0x01235e thru 0x018315 (0x5fb8 bytes) 000843 1767316385.999861 mapwrite 0x032481 thru 0x03ff0f (0xda8f bytes) 000844 1767316386.121074 mapread 0x00cdf6 thru 0x0112ff (0x450a bytes) 000846 1767316386.125061 read 0x00f5a4 thru 0x017bb8 (0x8615 bytes) 000848 1767316386.131015 MIRROR_RESYNC 4 mirrors 000850 1767316387.144053 read 0x02c655 thru 0x037580 (0xaf2c bytes) 000853 1767316387.154733 trunc from 0x03ff10 to 0x01b7be (0x24751 bytes) 000854 1767316387.258695 write 0x035be5 thru 0x036109 (0x525 bytes) 000857 1767316387.261869 mapwrite 0x01489b thru 0x021b7a (0xd2e0 bytes) 000858 1767316387.317208 mapread 0x02c2d0 thru 0x03219b (0x5ecc bytes) 000865 1767316387.325958 read 0x016499 thru 0x01f1f3 (0x8d5b bytes) 000866 1767316387.330017 write 0x008952 thru 0x00c52c (0x3bdb bytes) 000868 1767316387.342771 mapwrite 0x02ae04 thru 0x02b25d (0x45a bytes) 000869 1767316387.379161 read 0x013ef1 thru 0x02129c (0xd3ac bytes) 000870 1767316387.379909 trunc from 0x03610a to 0x0203fd (0x15d0c bytes) 000871 1767316387.402507 read 0x00ec8d thru 0x019e97 (0xb20b bytes) 000873 1767316387.404639 mapread 0x0172ad thru 0x01c1a0 (0x4ef4 bytes) 000874 1767316387.406067 trunc from 0x0203fe to 0x021ea9 (0x1aac bytes) 000879 1767316387.423974 read 0x0072ba thru 0x00fa7a (0x87c1 bytes) 000881 1767316387.425140 trunc from 0x021eaa to 0x023fa4 (0x20fb bytes) 000882 1767316387.437394 mapread 0x019f6c thru 0x023fa4 (0xa039 bytes) 000887 1767316387.448903 write 0x00e3f5 thru 0x011b44 (0x3750 bytes) 000888 1767316387.454963 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23fa5): Mirror 24: 0x307414ca (stale) Mirror 26: 0x307414ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fa5): Mirror 24: 0x307414ca (stale) Mirror 26: 0x307414ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fa5: skipped 000891 1767316387.888132 read 0x011098 thru 0x0111d8 (0x141 bytes) 000892 1767316387.899136 trunc from 0x023fa5 to 0x03134d (0xd3a9 bytes) 000893 1767316387.917398 write 0x035a6e thru 0x03ed7a (0x930d bytes) 000894 1767316387.918867 write 0x02d2d1 thru 0x02df68 (0xc98 bytes) 000895 1767316387.927062 mapwrite 0x00ddea thru 0x01514d (0x7364 bytes) 000897 1767316388.059306 read 0x02afbb thru 0x031036 (0x607c bytes) 000899 1767316388.096918 mapread 0x028e3e thru 0x032aca (0x9c8d bytes) 000900 1767316388.098162 mapread 0x01fcbf thru 0x028ceb (0x902d bytes) 000902 1767316388.100770 trunc from 0x03ed7b to 0x012c4d (0x2c12d bytes) 000904 1767316388.136545 trunc from 0x012c4e to 0x010020 (0x2c2d bytes) 000905 1767316388.169020 read 0x00e222 thru 0x010020 (0x1dff bytes) 000909 1767316388.180021 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x199:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10021): Mirror 26: 0x8c9a5a80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10021): Mirror 26: 0x8c9a5a80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10021: skipped 000910 1767316388.662698 mapwrite 0x01250e thru 0x01e402 (0xbef5 bytes) 000911 1767316388.805355 write 0x025c89 thru 0x032efb (0xd273 bytes) 000913 1767316388.809274 write 0x024c9b thru 0x030322 (0xb688 bytes) 000915 1767316388.820417 mapread 0x02d535 thru 0x032efb (0x59c7 bytes) 000916 1767316388.821574 mapread 0x0222df thru 0x0225eb (0x30d bytes) 000917 1767316388.822747 mapwrite 0x024840 thru 0x029c1f (0x53e0 bytes) 000918 1767316388.873034 write 0x033d27 thru 0x03ffff (0xc2d9 bytes) 000920 1767316388.876759 read 0x03d215 thru 0x03ffff (0x2deb bytes) 000923 1767316388.879126 trunc from 0x040000 to 0x01ca6b (0x23594 bytes) 000924 1767316388.899663 read 0x018ea1 thru 0x019476 (0x5d6 bytes) 000925 1767316388.900521 mapwrite 0x02e484 thru 0x02f924 (0x14a1 bytes) 000926 1767316389.005923 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f925): Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f925): Mirror 26: 0xb7a1b9e1 (stale) CRC-32 checksum value for chunk [0, 0x2f925): Mirror 1: 0x8087b862 Mirror 27: 0x8087b862 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f925: skipped 000928 1767316389.933042 mapwrite 0x0257f2 thru 0x027e9f (0x26ae bytes) 000930 1767316390.107569 read 0x01e8bd thru 0x02114c (0x2890 bytes) 000931 1767316390.116119 trunc from 0x02f925 to 0x0183b1 (0x17573 bytes) 000932 1767316390.141864 mapwrite 0x037b87 thru 0x03b759 (0x3bd3 bytes) 000933 1767316390.218763 mapread 0x006f48 thru 0x00865f (0x1718 bytes) 000934 1767316390.230089 trunc from 0x03b75a to 0x0250e8 (0x16671 bytes) 000935 1767316390.259886 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x250e9): Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x250e9): Mirror 26: 0x99c5f4ac (stale) Mirror 27: 0x3c9bb935 (stale) CRC-32 checksum value for chunk [0, 0x250e9): Mirror 1: 0x12b9361b Mirror 28: 0x12b9361b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x250e9: skipped 000937 1767316391.171445 read 0x0223bd thru 0x0250e8 (0x2d2c bytes) 000938 1767316391.183399 trunc from 0x0250e9 to 0x03a97c (0x15894 bytes) 000940 1767316391.327968 trunc from 0x03a97d to 0x01f219 (0x1b763 bytes) 000941 1767316391.354021 write 0x027656 thru 0x0313b4 (0x9d5f bytes) 000943 1767316391.355856 mapread 0x025fed thru 0x0313b4 (0xb3c8 bytes) 000944 1767316391.366962 mapwrite 0x00f860 thru 0x019330 (0x9ad1 bytes) 000948 1767316391.444754 mapwrite 0x0135af thru 0x0208a8 (0xd2fa bytes) 000949 1767316391.508337 read 0x00cc73 thru 0x01079c (0x3b2a bytes) 000950 1767316391.509136 trunc from 0x0313b5 to 0x012cc8 (0x1e6ec bytes) 000951 1767316391.530881 mapwrite 0x03a671 thru 0x03ac1d (0x5ad bytes) 000952 1767316391.600357 mapwrite 0x0075d8 thru 0x00dc94 (0x66bd bytes) 000953 1767316391.652440 read 0x006b91 thru 0x00bae5 (0x4f55 bytes) 000954 1767316391.653297 write 0x01c230 thru 0x01d0f0 (0xec1 bytes) 000956 1767316391.656086 read 0x00bde5 thru 0x00d957 (0x1b73 bytes) 000958 1767316391.657809 read 0x0023e3 thru 0x010270 (0xde8e bytes) 000960 1767316391.663781 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ac1e): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac1e): Mirror 26: 0xff4c5f94 (stale) Mirror 27: 0x577c5ee0 (stale) Mirror 28: 0xbaf2ffc8 (stale) CRC-32 checksum value for chunk [0, 0x3ac1e): Mirror 1: 0xe362c8ef Mirror 29: 0xe362c8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac1e: skipped 000962 1767316392.399670 write 0x02cdd5 thru 0x03b563 (0xe78f bytes) 000963 1767316392.513995 write 0x005eb4 thru 0x012fee (0xd13b bytes) 000965 1767316392.529460 mapwrite 0x02aab7 thru 0x02c309 (0x1853 bytes) 000967 1767316392.629797 trunc from 0x03b564 to 0x0306e1 (0xae82 bytes) 000969 1767316392.683129 write 0x00c486 thru 0x017099 (0xac14 bytes) 000970 1767316392.691806 write 0x006477 thru 0x013278 (0xce02 bytes) 000971 1767316392.693901 trunc from 0x0306e2 to 0x02db86 (0x2b5b bytes) 000972 1767316392.722181 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2db87): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db87): Mirror 26: 0xc3a197c0 (stale) Mirror 27: 0x5c1129bf (stale) Mirror 28: 0x4f4d6e6c (stale) Mirror 29: 0x62e95fba (stale) CRC-32 checksum value for chunk [0, 0x2db87): Mirror 1: 0xc7489300 Mirror 30: 0xc7489300 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db87: skipped 000975 1767316393.975086 MIRROR_RESYNC 6 mirrors 000976 1767316395.730069 trunc from 0x02db87 to 0x01041d (0x1d769 bytes) 000977 1767316395.835617 write 0x030950 thru 0x03dd4b (0xd3fc bytes) 000979 1767316395.837975 trunc from 0x03dd4c to 0x00757d (0x367ce bytes) 000980 1767316395.858017 mapread 0x001e3c thru 0x00757d (0x5742 bytes) 000981 1767316395.884319 write 0x02dc42 thru 0x031d13 (0x40d2 bytes) 000982 1767316395.885663 mapwrite 0x026b23 thru 0x02eb0d (0x7feb bytes) 000983 1767316395.970823 read 0x023204 thru 0x0244c6 (0x12c3 bytes) 000984 1767316395.971754 mapwrite 0x00cc62 thru 0x01aada (0xde79 bytes) 000987 1767316396.079733 read 0x008bce thru 0x009174 (0x5a7 bytes) 000989 1767316396.087080 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31d14): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d14): Mirror 26: 0xa82f5608 (stale) Mirror 27: 0xa82f5608 (stale) Mirror 28: 0xa82f5608 (stale) Mirror 29: 0xa82f5608 (stale) Mirror 30: 0xa82f5608 (stale) CRC-32 checksum value for chunk [0, 0x31d14): Mirror 1: 0x9eb98b33 Mirror 31: 0x9eb98b33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d14: skipped 000990 1767316397.486812 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31d14): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d14): Mirror 26: 0xa82f5608 (stale) Mirror 27: 0xa82f5608 (stale) Mirror 28: 0xa82f5608 (stale) Mirror 29: 0xa82f5608 (stale) Mirror 30: 0xa82f5608 (stale) CRC-32 checksum value for chunk [0, 0x31d14): Mirror 1: 0x9eb98b33 Mirror 31: 0x9eb98b33 Mirror 32: 0x9eb98b33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d14: skipped 000991 1767316399.092364 mapread 0x01d209 thru 0x0202fd (0x30f5 bytes) 000992 1767316399.126146 mapread 0x025c58 thru 0x0272a7 (0x1650 bytes) 000993 1767316399.127990 read 0x00c74e thru 0x0199ad (0xd260 bytes) 000994 1767316399.141651 trunc from 0x031d14 to 0x002cdb (0x2f038 bytes) 000995 1767316399.292872 mapread 0x00034a thru 0x002cdb (0x2992 bytes) 000997 1767316399.302973 read 0x002335 thru 0x002cdb (0x9a7 bytes) 000998 1767316399.304705 write 0x0067a0 thru 0x013a18 (0xd279 bytes) 000999 1767316399.306636 write 0x038f44 thru 0x03ffff (0x70bc bytes) 001000 1767316399.318392 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0xea409d0 (stale) Mirror 27: 0xea409d0 (stale) Mirror 28: 0xea409d0 (stale) Mirror 29: 0xea409d0 (stale) Mirror 30: 0xea409d0 (stale) Mirror 31: 0xa32fffd8 (stale) Mirror 32: 0xa32fffd8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x135050d5 Mirror 33: 0x135050d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 20:13:26 (1767316406) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:13:29 (1767316409) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x253:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232316 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x253:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x254:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158186 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x253:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x254:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x258:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x253:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x254:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x258:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x255:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 20:13:37 (1767316417) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to Extending file size to 8408736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 623744 .. Extending file size to 3293088 .. Extending file size to 8144480 .. Extending file size to 8745472 .. Extending file size to 8607424 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9102464 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 58051 58052 58053 58057 58059 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) 192.168.202.154@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:14:56 (1767316496) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8099232 .. Extending file size to 753184 .. Extending file size to 5083936 .. Extending file size to 8108576 .. Extending file size to 8147200 .. Extending file size to 8539392 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8640416 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8391456 .. Extending file size to 9233088 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8652032 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59679 59680 59682 59683 59685 done 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) 192.168.202.154@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:16:13 (1767316573) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124063 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33858 s, 6.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8274d1ca (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:16:32 (1767316592) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0727765 s, 28.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 20:16:40 (1767316600) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:16:47 (1767316607) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184637 s, 11.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x270:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:16:56 (1767316616) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.384419 s, 5.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x271:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x271:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:17:03 (1767316623) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308394 s, 6.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:17:37 (1767316657) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105485 s, 9.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:18:38 (1767316718) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.374981 s, 5.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:19:33 (1767316773) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. Creating new pool test_204f oleg254-server: Pool lustre.test_204f created Adding targets to pool oleg254-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg254-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420698 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg254-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg254-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg254-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:20:52 (1767316852) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:20:59 (1767316859) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:21:07 (1767316867) oleg254-server: Pool lustre.test_206 created Adding targets to pool oleg254-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg254-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg254-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg254-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg254-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:21:31 (1767316891) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505497 s, 20.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513279 s, 20.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4e242e3b Mirror 3: 0x4e242e3b PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:21:42 (1767316902) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:21:46 (1767316906) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:21:50 (1767316910) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.152607 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf50000 Read this (4 bytes): BBBB 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:22:28 (1767316948) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0551134 s, 149 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:22:35 (1767316955) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465421 s, 22.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834447 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 20:22:37.000000000 -0500 Modify: 2026-01-01 20:22:37.000000000 -0500 Change: 2026-01-01 20:22:38.000000000 -0500 Birth: 2026-01-01 20:22:36.000000000 -0500 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:22:42 (1767316962) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:22:45 (1767316965) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.679617 s, 60.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0554926 s, 73.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 1 20:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 1 20:22 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:22:55 (1767316975) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:23:03 (1767316983) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1908 sec =========== 20:23:13 (1767316993) === sanity-flr: start cleanup 20:23:15 (1767316995) === === sanity-flr: finish cleanup 20:23:21 (1767317001) ===