-----============= acceptance-small: sanity-flr ============----- Fri May 8 20:30:06 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg104-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:30:26 (1778286626) === oleg104-client.virtnet: executing check_config_client /mnt/lustre oleg104-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg104-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff915fca439800.idle_timeout=debug osc.lustre-OST0001-osc-ffff915fca439800.idle_timeout=debug disable quota as required oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:31:09 (1778286669) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8235] == sanity-flr test 0a: lfs mirror create with -N option == 20:31:13 (1778286673) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:31:35 (1778286695) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:31:39 (1778286699) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:31:44 (1778286704) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:32:12 (1778286732) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:32:17 (1778286737) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:32:20 (1778286740) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff915fca439800.stats=clear osc.lustre-OST0001-osc-ffff915fca439800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:32:31 (1778286751) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:32:46 (1778286766) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:32:54 (1778286774) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:32:58 (1778286778) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:33:05 (1778286785) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 20:33:16 (1778286796) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:33:23 (1778286803) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:33:29 (1778286809) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:33:37 (1778286817) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:33:46 (1778286826) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:33:54 (1778286834) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.715188 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113365 s, 9.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:34:03 (1778286843) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.376173 s, 27.9 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:34:10 (1778286850) Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:35:16 (1778286916) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.62526 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:35:57 (1778286957) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:37:20 (1778287040) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 9.16496 s, 5.7 MB/s -rw-r--r-- 1 root root 52428800 May 8 20:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m2.502s user 0m0.002s sys 0m0.154s Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.072s user 0m0.000s sys 0m0.163s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.474s user 0m0.000s sys 0m0.192s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff915fca439800.ost_server_uuid in FULL state after 0 sec PASS 33b (116s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:39:16 (1778287156) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:39:20 (1778287160) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:39:24 (1778287164) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:39:27 (1778287167) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 20:39:36 (1778287176) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.68424 s, 18.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:39:52 (1778287192) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721798 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361801 s, 29.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:40:00 (1778287200) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:40:09 (1778287209) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00193955 s, 528 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000950835 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00732062 s, 140 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142526 s, 718 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 20:40:21 (1778287221) create test files with size 15750273 .. 2e7ad88410a30a21e7bbb2d5f4324331 - 39145d6f64ef66d13626fe49e273c0fa - 44613920b1badf682d941f6aaef85021 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 20:40:48 (1778287248) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:41:01 (1778287261) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:41:08 (1778287268) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100036 s, 10.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359677 s, 29.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 20:41:18 (1778287278) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0881554 s, 23.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.420637 s, 10.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17769 s, 11.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 20:41:36 (1778287296) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 20:41:40 (1778287300) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:41:44 (1778287304) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314054 s, 33.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:41:53 (1778287313) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.388917 s, 8.1 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:42:04 (1778287324) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:42:15 (1778287335) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.338474 s, 31.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:42:28 (1778287348) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff915fe4cb5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.525293 s, 20.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff915fca1df800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:42:46 (1778287366) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:43:25 (1778287405) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:44:11 (1778287451) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:44:15 (1778287455) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.414422 s, 7.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.18749 s, 16.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x19ed0c9e (stale) Mirror 2: 0x19ed0c9e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f059a36a6d8021c8b005a311a91817ab - mirror 1 checksum: 9b6966819a275e69d6c4088ea0a3001f - mirror 2 checksum: f059a36a6d8021c8b005a311a91817ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 8 20:44:19 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 8 20:44:19 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f059a36a6d8021c8b005a311a91817ab - mirror 1 checksum: 9b6966819a275e69d6c4088ea0a3001f - mirror 2 checksum: 9b6966819a275e69d6c4088ea0a3001f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 8 20:44:19 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 8 20:44:19 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:44:28 (1778287468) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249921 s, 42.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:44:36 (1778287476) Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 20:44:48 (1778287488) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 20:45:20 (1778287520) targets are mounted 20:45:20 (1778287520) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432082 s, 24.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:45:39 (1778287539) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0743141 s, 276 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0175771 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (16s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:45:56 (1778287556) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.107496 s, 191 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139994 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:46:08 (1778287568) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0075806 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:46:17 (1778287577) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.114786 s, 178 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:46:27 (1778287587) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.013982 s, 293 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0401141 s, 102 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:46:37 (1778287597) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:47:03 (1778287623) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:47:33 (1778287653) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:47:59 (1778287679) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:48:23 (1778287703) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:48:31 (1778287711) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 56549 mirror split pid 56551 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6895: 56551 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6895: 56551 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56549 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56551 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 20:49:41 (1778287781) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7785 truncating to largest ever: 0x5bc 000002 1778287786.077020 trunc from 00000000 to 0x0005bb (0x5bc bytes) truncating to largest ever: 0x3caed 000003 1778287786.112615 trunc from 0x0005bc to 0x03caec (0x3c531 bytes) 000004 1778287786.141574 mapwrite 0x0138d0 thru 0x01eab7 (0xb1e8 bytes) 000007 1778287786.288724 trunc from 0x034b55 to 0x035ef0 (0x139c bytes) 000008 1778287786.296844 punch from 0x02de78 to 0x037a18 (0x9ba1 bytes) 000009 1778287786.314787 mapwrite 0x03c08a thru 0x03ffff (0x3f76 bytes) 000010 1778287786.405925 fallocate 0x01a9ab thru 0x02184f (0x6ea5 bytes) 000012 1778287786.428858 punch from 0x00bc3f to 0x013582 (0x7944 bytes) 000013 1778287786.439878 fallocate 0x01f364 thru 0x02ceb4 (0xdb51 bytes) 000014 1778287786.450951 punch from 0x000037 to 0x007e20 (0x7dea bytes) 000016 1778287786.457328 punch from 0x001d6b to 0x011b13 (0xfda9 bytes) zero_range to largest ever: 0x3df49 000017 1778287786.462753 trunc from 0x037ef0 to 0x03df48 (0x6059 bytes) 000018 1778287786.468676 trunc from 0x007bcf to 0x012953 (0xad85 bytes) 000020 1778287786.481644 mapwrite 0x00f80e thru 0x01090c (0x10ff bytes) 000021 1778287786.521812 trunc from 0x012ab9 to 0x01333a (0x882 bytes) 000022 1778287786.545096 fallocate 0x008f7c thru 0x01490e (0xb993 bytes) 000023 1778287786.571843 fallocate 0x01a41b thru 0x02895a (0xe540 bytes) 000024 1778287786.582600 read 0x00e670 thru 0x0103d9 (0x1d6a bytes) 000025 1778287786.593505 trunc from 0x00fd74 to 0x01532f (0x55bc bytes) 000027 1778287786.602596 fallocate 0x001b3e thru 0x00aa67 (0x8f2a bytes) 000028 1778287786.646597 MIRROR_EXTEND to 2 mirrors 000029 1778287788.238394 write 0x02fbc5 thru 0x034f98 (0x53d4 bytes) 000030 1778287788.483446 trunc from 0x040000 to 0x0222fe (0x1dd01 bytes) 000032 1778287788.595367 trunc from 0x01b69b to 0x0222fe (0x6c64 bytes) 000033 1778287788.633859 mapread 0x001d0e thru 0x00b4a2 (0x9795 bytes) 000034 1778287788.697394 trunc from 0x020b5d to 0x0222fe (0x17a2 bytes) 000035 1778287788.721098 punch from 0x01b38b to 0x01e36c (0x2fe2 bytes) 000036 1778287788.769287 MIRROR_RESYNC 2 mirrors 000037 1778287790.196738 write 0x0050a1 thru 0x00eb02 (0x9a62 bytes) 000038 1778287790.267226 fallocate 0x021388 thru 0x022f65 (0x1bde bytes) 000039 1778287790.278698 fallocate 0x01f811 thru 0x020bb4 (0x13a4 bytes) 000040 1778287790.289368 punch from 0x01c97f to 0x0205bc (0x3c3e bytes) 000041 1778287790.297652 mapwrite 0x0180f3 thru 0x01e801 (0x670f bytes) 000042 1778287790.360438 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x222ff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222ff): Mirror 2: 0xf471471a (stale) CRC-32 checksum value for chunk [0, 0x222ff): Mirror 1: 0xe3e05948 Mirror 3: 0xe3e05948 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222ff: skipped 000043 1778287791.572741 write 0x016ea3 thru 0x026c58 (0xfdb6 bytes) 000044 1778287791.637716 fallocate 0x019752 thru 0x023be2 (0xa491 bytes) 000045 1778287791.653917 trunc from 0x0262b8 to 0x026c58 (0x9a1 bytes) 000046 1778287791.681881 trunc from 0x01abdb to 0x023390 (0x87b6 bytes) 000047 1778287791.692530 write 0x00ca3c thru 0x015f17 (0x94dc bytes) 000048 1778287791.715635 mapread 0x024489 thru 0x024cb7 (0x82f bytes) 000049 1778287791.721010 trunc from 0x026c59 to 0x025325 (0x1933 bytes) 000050 1778287791.769311 mapread 0x022f3f thru 0x025325 (0x23e7 bytes) 000051 1778287791.787763 trunc from 0x025326 to 0x0103cb (0x14f5a bytes) 000052 1778287791.816742 read 0x0072f7 thru 0x00f229 (0x7f33 bytes) 000054 1778287791.830778 read 0x0053a5 thru 0x00fa2a (0xa686 bytes) 000055 1778287791.843267 trunc from 0x0103cc to 0x0103c0 (0xb bytes) 000056 1778287791.863236 read 0x004fb7 thru 0x0103c0 (0xb40a bytes) 000057 1778287791.865306 trunc from 0x0103c1 to 0x004d89 (0xb637 bytes) 000058 1778287791.897600 trunc from 0x004d8a to 0x039a36 (0x34cad bytes) 000059 1778287791.935962 read 0x004db4 thru 0x00a19b (0x53e8 bytes) 000060 1778287791.962927 read 0x00ff78 thru 0x017544 (0x75cd bytes) 000061 1778287791.963681 fallocate 0x006ad6 thru 0x01290e (0xbe39 bytes) fallocating to largest ever: 0x40000 000062 1778287791.971860 fallocate 0x03e9eb thru 0x03ffff (0x1615 bytes) 000063 1778287791.985612 read 0x011d0a thru 0x01954d (0x7844 bytes) 000064 1778287791.988850 mapwrite 0x034440 thru 0x03f3df (0xafa0 bytes) 000065 1778287792.041690 mapwrite 0x006475 thru 0x00cdf5 (0x6981 bytes) 000066 1778287792.095417 mapwrite 0x004acb thru 0x00d6e1 (0x8c17 bytes) 000067 1778287792.151490 read 0x0245c0 thru 0x0278c8 (0x3309 bytes) 000068 1778287792.159555 mapread 0x00aa45 thru 0x01869d (0xdc59 bytes) 000069 1778287792.166983 fallocate 0x0251aa thru 0x02faf3 (0xa94a bytes) 000070 1778287792.186321 read 0x011bad thru 0x01c610 (0xaa64 bytes) 000071 1778287792.187060 write 0x014a5e thru 0x022912 (0xdeb5 bytes) 000072 1778287792.195989 trunc from 0x040000 to 0x036641 (0x99be bytes) 000073 1778287792.233049 fallocate 0x023609 thru 0x031c31 (0xe629 bytes) 000074 1778287792.263913 trunc from 0x036642 to 0x00ec26 (0x27a1b bytes) 000075 1778287792.376245 punch from 0x0028c4 to 0x00cc12 (0xa34f bytes) 000076 1778287792.399626 mapread 0x0037a3 thru 0x00ec26 (0xb484 bytes) 000077 1778287792.461315 punch from 0x00e400 to 0x00ec26 (0x827 bytes) 000078 1778287792.481949 fallocate 0x01d6d1 thru 0x01e3cf (0xcff bytes) 000079 1778287792.503897 fallocate 0x029257 thru 0x038c7b (0xfa25 bytes) 000080 1778287792.520613 punch from 0x036de3 to 0x038c7b (0x1e99 bytes) 000081 1778287792.545573 punch from 0x011bcb to 0x0153c4 (0x37fa bytes) 000082 1778287792.566464 mapread 0x004f78 thru 0x0098d3 (0x495c bytes) 000083 1778287792.567704 punch from 0x034b38 to 0x038c7b (0x4144 bytes) 000084 1778287792.605545 trunc from 0x0311d4 to 0x037e8c (0x6cb9 bytes) 000085 1778287792.625319 trunc from 0x038c7c to 0x03a516 (0x189b bytes) 000086 1778287792.778641 fallocate 0x028752 thru 0x0340ae (0xb95d bytes) 000088 1778287792.851281 read 0x03910d thru 0x03a516 (0x140a bytes) 000091 1778287792.933050 mapwrite 0x00abdf thru 0x01308b (0x84ad bytes) 000092 1778287793.068113 fallocate 0x002b6f thru 0x00be34 (0x92c6 bytes) 000093 1778287793.102444 fallocate 0x006910 thru 0x00bafd (0x51ee bytes) 000094 1778287793.136493 MIRROR_RESYNC 3 mirrors 000095 1778287795.078633 mapread 0x0373ef thru 0x03a516 (0x3128 bytes) 000096 1778287795.132406 read 0x03406c thru 0x03a516 (0x64ab bytes) 000097 1778287795.132843 mapwrite 0x00e3c5 thru 0x013861 (0x549d bytes) 000098 1778287795.481597 mapwrite 0x026193 thru 0x02633d (0x1ab bytes) 000099 1778287795.613245 trunc from 0x03a517 to 0x00e021 (0x2c4f5 bytes) 000100 1778287795.667445 mapwrite 0x033527 thru 0x03ffff (0xcad9 bytes) 000103 1778287795.847172 mapwrite 0x01e041 thru 0x02b590 (0xd550 bytes) 000104 1778287796.010843 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe17eaa72 (stale) Mirror 3: 0xe17eaa72 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x592f9870 Mirror 4: 0x592f9870 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000106 1778287797.028879 mapwrite 0x036123 thru 0x03ffff (0x9edd bytes) 000107 1778287797.124571 trunc from 0x040000 to 0x012b31 (0x2d4ce bytes) 000108 1778287797.188947 trunc from 0x012b32 to 0x002da5 (0xfd8c bytes) 000109 1778287797.221494 trunc from 0x002da6 to 0x005650 (0x28ab bytes) 000110 1778287797.253422 MIRROR_RESYNC 4 mirrors 000111 1778287798.310795 read 0x0009a8 thru 0x005650 (0x4ca9 bytes) 000112 1778287798.331139 trunc from 0x005602 to 0x005650 (0x4f bytes) 000113 1778287798.385228 mapwrite 0x0057d6 thru 0x00c5fa (0x6e25 bytes) 000115 1778287798.503224 trunc from 0x00c5fb to 0x031f36 (0x2593c bytes) 000116 1778287798.566371 mapread 0x00c23f thru 0x01833f (0xc101 bytes) 000117 1778287798.652180 mapwrite 0x03c7af thru 0x03ffff (0x3851 bytes) 000118 1778287798.745302 write 0x00ce09 thru 0x010a9b (0x3c93 bytes) 000119 1778287798.757194 read 0x00846f thru 0x00be81 (0x3a13 bytes) 000120 1778287798.774911 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xda33ad72 Mirror 5: 0xda33ad72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000121 1778287800.053714 trunc from 0x040000 to 0x033c21 (0xc3de bytes) 000122 1778287800.147051 read 0x01862d thru 0x01cb91 (0x4565 bytes) 000124 1778287800.159317 fallocate 0x00a65c thru 0x01a54c (0xfef1 bytes) 000125 1778287800.166312 punch from 0x002a58 to 0x00553b (0x2ae4 bytes) 000126 1778287800.194316 trunc from 0x013c3b to 0x01fd65 (0xc12b bytes) 000127 1778287800.202260 mapwrite 0x015d63 thru 0x01ea7f (0x8d1d bytes) 000128 1778287800.297867 punch from 0x01a517 to 0x0284f3 (0xdfdd bytes) 000129 1778287800.310462 trunc from 0x02723d to 0x02c352 (0x5116 bytes) 000130 1778287800.324856 trunc from 0x033c22 to 0x02f4b2 (0x476f bytes) 000132 1778287800.371224 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f4b3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4b3): Mirror 2: 0xc5c8c8c4 (stale) Mirror 3: 0xc5c8c8c4 (stale) Mirror 4: 0xc5c8c8c4 (stale) Mirror 5: 0xdd0adbdb (stale) CRC-32 checksum value for chunk [0, 0x2f4b3): Mirror 1: 0xf039ccd1 Mirror 6: 0xf039ccd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4b3: skipped 000133 1778287802.130635 trunc from 0x02b517 to 0x02f4b2 (0x3f9c bytes) 000134 1778287802.162496 MIRROR_RESYNC 6 mirrors 000135 1778287804.298316 trunc from 0x0248f5 to 0x02f4b2 (0xabbe bytes) 000136 1778287804.373286 trunc from 0x000109 to 0x0055f3 (0x54eb bytes) 000137 1778287804.380198 trunc from 0x02ed34 to 0x02f4b2 (0x77f bytes) 000139 1778287804.408045 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f4b3): Mirror 2: 0xf039ccd1 (stale) Mirror 3: 0xf039ccd1 (stale) Mirror 4: 0xf039ccd1 (stale) Mirror 5: 0xf039ccd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4b3): Mirror 2: 0xf039ccd1 (stale) Mirror 3: 0xf039ccd1 (stale) Mirror 4: 0xf039ccd1 (stale) Mirror 5: 0xf039ccd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4b3: skipped 000140 1778287805.101143 trunc from 0x01b793 to 0x02804b (0xc8b9 bytes) 000141 1778287805.115492 read 0x002bb8 thru 0x008253 (0x569c bytes) 000142 1778287805.141296 write 0x03e32b thru 0x03ffff (0x1cd5 bytes) 000143 1778287805.148735 fallocate 0x00f4c5 thru 0x01ee33 (0xf96f bytes) 000144 1778287805.159491 fallocate 0x025ee1 thru 0x031e50 (0xbf70 bytes) 000145 1778287805.169210 fallocate 0x033916 thru 0x03ffff (0xc6ea bytes) 000146 1778287805.183270 mapread 0x014162 thru 0x01d0d8 (0x8f77 bytes) 000147 1778287805.238317 trunc from 0x005c25 to 0x00a19e (0x457a bytes) 000148 1778287805.250584 mapwrite 0x030683 thru 0x03cd32 (0xc6b0 bytes) 000150 1778287805.361944 write 0x03897e thru 0x03ffff (0x7682 bytes) 000151 1778287805.368298 read 0x006042 thru 0x008726 (0x26e5 bytes) 000152 1778287805.389457 mapread 0x02cd7c thru 0x03513e (0x83c3 bytes) 000153 1778287805.392169 mapread 0x03db5f thru 0x03ffff (0x24a1 bytes) 000154 1778287805.406116 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce7b9916 (stale) Mirror 3: 0xce7b9916 (stale) Mirror 4: 0xce7b9916 (stale) Mirror 5: 0xce7b9916 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6299faed Mirror 6: 0x6299faed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1778287807.355717 read 0x0174f4 thru 0x024f15 (0xda22 bytes) 000156 1778287807.380923 mapwrite 0x02e721 thru 0x03e4f3 (0xfdd3 bytes) 000157 1778287807.512179 trunc from 0x040000 to 0x01a633 (0x259cc bytes) 000158 1778287807.535689 write 0x003d90 thru 0x010480 (0xc6f1 bytes) 000159 1778287807.546689 trunc from 0x01a634 to 0x026a7b (0xc448 bytes) 000160 1778287807.569142 trunc from 0x026a7c to 0x02a4fc (0x3a81 bytes) 000161 1778287807.585534 mapwrite 0x006a7d thru 0x00a061 (0x35e5 bytes) 000162 1778287807.602021 read 0x0005d4 thru 0x00cc78 (0xc6a5 bytes) 000163 1778287807.614163 trunc from 0x0048af to 0x00de1b (0x956d bytes) 000164 1778287807.620110 punch from 0x01579f to 0x016da4 (0x1606 bytes) 000165 1778287807.626557 mapwrite 0x032f03 thru 0x03ffff (0xd0fd bytes) 000166 1778287807.691752 read 0x01a5d9 thru 0x024a82 (0xa4aa bytes) 000167 1778287807.692589 trunc from 0x029d09 to 0x02f223 (0x551b bytes) 000168 1778287807.698927 punch from 0x0241d2 to 0x0301e3 (0xc012 bytes) 000169 1778287807.705155 read 0x01524c thru 0x01ca20 (0x77d5 bytes) 000170 1778287807.727075 write 0x0041a5 thru 0x013fb8 (0xfe14 bytes) 000171 1778287807.734697 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce7b9916 (stale) Mirror 3: 0xce7b9916 (stale) Mirror 4: 0xce7b9916 (stale) Mirror 5: 0xce7b9916 (stale) Mirror 6: 0x6299faed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7b2acfa Mirror 7: 0xf7b2acfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000172 1778287809.107671 mapwrite 0x009fdd thru 0x0155fc (0xb620 bytes) 000173 1778287809.297127 punch from 0x00fd37 to 0x01d57b (0xd845 bytes) 000174 1778287809.311184 read 0x02b408 thru 0x02e79a (0x3393 bytes) 000175 1778287809.320106 trunc from 0x038ed6 to 0x03cdb6 (0x3ee1 bytes) 000176 1778287809.329980 fallocate 0x033cc5 thru 0x03faad (0xbde9 bytes) 000177 1778287809.345415 trunc from 0x01f6dd to 0x021942 (0x2266 bytes) 000178 1778287809.364439 punch from 0x037246 to 0x03eb17 (0x78d2 bytes) 000179 1778287809.376924 mapread 0x0335fa thru 0x03acdb (0x76e2 bytes) 000180 1778287809.407442 punch from 0x026999 to 0x02ff94 (0x95fc bytes) 000182 1778287809.423120 fallocate 0x00b414 thru 0x01435f (0x8f4c bytes) 000183 1778287809.438032 trunc from 0x040000 to 0x03a390 (0x5c6f bytes) 000184 1778287809.492502 mapwrite 0x00c359 thru 0x01902c (0xccd4 bytes) 000185 1778287809.558486 trunc from 0x0325ba to 0x03a390 (0x7dd7 bytes) 000187 1778287809.565952 fallocate 0x02c873 thru 0x02fe43 (0x35d1 bytes) 000188 1778287809.571057 write 0x0060f4 thru 0x00a48d (0x439a bytes) 000189 1778287809.572578 mapwrite 0x026668 thru 0x02cf81 (0x691a bytes) 000190 1778287809.634872 mapread 0x027888 thru 0x032e62 (0xb5db bytes) 000191 1778287809.660329 write 0x013bfc thru 0x0151a4 (0x15a9 bytes) 000192 1778287809.670030 MIRROR_RESYNC 7 mirrors 000193 1778287812.973924 punch from 0x003394 to 0x005c22 (0x288f bytes) 000194 1778287813.057209 mapread 0x008a7d thru 0x010790 (0x7d14 bytes) 000195 1778287813.104764 mapwrite 0x030a68 thru 0x032d48 (0x22e1 bytes) 000196 1778287813.161968 mapread 0x019395 thru 0x0234c9 (0xa135 bytes) 000197 1778287813.218916 read 0x029939 thru 0x02f14d (0x5815 bytes) 000198 1778287813.221510 read 0x012c2a thru 0x01b77b (0x8b52 bytes) 000199 1778287813.228998 trunc from 0x03a391 to 0x01c8bc (0x1dad4 bytes) 000200 1778287813.288060 fallocate 0x03241c thru 0x03ffff (0xdbe4 bytes) 000201 1778287813.296025 punch from 0x00d574 to 0x0188a6 (0xb333 bytes) 000202 1778287813.311833 MIRROR_RESYNC 7 mirrors 000204 1778287816.353728 trunc from 0x040000 to 0x03ac59 (0x53a6 bytes) 000205 1778287816.433395 read 0x036cfb thru 0x03ac59 (0x3f5f bytes) 000206 1778287816.462999 trunc from 0x032910 to 0x03ac59 (0x834a bytes) 000207 1778287816.488442 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ac5a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac5a): Mirror 2: 0xabba6b06 (stale) Mirror 3: 0xabba6b06 (stale) Mirror 4: 0xabba6b06 (stale) Mirror 5: 0xabba6b06 (stale) Mirror 6: 0xabba6b06 (stale) Mirror 7: 0xabba6b06 (stale) CRC-32 checksum value for chunk [0, 0x3ac5a): Mirror 1: 0xabba6b06 Mirror 8: 0xabba6b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac5a: skipped 000208 1778287818.594265 MIRROR_RESYNC 8 mirrors 000209 1778287822.037465 trunc from 0x02becf to 0x034a6c (0x8b9e bytes) 000210 1778287822.133872 read 0x0125e0 thru 0x0192b7 (0x6cd8 bytes) 000211 1778287822.162438 read 0x02cb4e thru 0x02e179 (0x162c bytes) 000212 1778287822.192260 trunc from 0x03ac5a to 0x01dfe9 (0x1cc70 bytes) 000213 1778287822.224554 punch from 0x01198a to 0x014299 (0x2910 bytes) 000214 1778287822.241925 write 0x005e52 thru 0x0114c5 (0xb674 bytes) 000215 1778287822.259039 mapwrite 0x015488 thru 0x0210bb (0xbc34 bytes) 000216 1778287822.398994 write 0x0376c4 thru 0x03ffff (0x893c bytes) 000217 1778287822.400659 mapwrite 0x00606d thru 0x0075e5 (0x1579 bytes) 000218 1778287822.444898 write 0x012cc0 thru 0x015ee7 (0x3228 bytes) 000219 1778287822.476778 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf54dfaa8 (stale) Mirror 3: 0xf54dfaa8 (stale) Mirror 4: 0xf54dfaa8 (stale) Mirror 5: 0xf54dfaa8 (stale) Mirror 6: 0xf54dfaa8 (stale) Mirror 7: 0xf54dfaa8 (stale) Mirror 8: 0xf54dfaa8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xade55bb8 Mirror 9: 0xade55bb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000220 1778287823.860233 mapread 0x03eb14 thru 0x03ffff (0x14ec bytes) 000221 1778287823.899736 mapread 0x0330cd thru 0x03cab9 (0x99ed bytes) 000222 1778287823.949753 mapwrite 0x037f1a thru 0x03bbde (0x3cc5 bytes) 000223 1778287824.097651 trunc from 0x033e36 to 0x03ffff (0xc1ca bytes) 000224 1778287824.119665 read 0x013206 thru 0x015bf5 (0x29f0 bytes) 000225 1778287824.127603 write 0x00b95b thru 0x011af5 (0x619b bytes) 000226 1778287824.152140 punch from 0x033668 to 0x037c9a (0x4633 bytes) 000227 1778287824.164937 punch from 0x016cf3 to 0x01c18e (0x549c bytes) 000228 1778287824.170739 read 0x0220e1 thru 0x0291fe (0x711e bytes) 000229 1778287824.188838 MIRROR_RESYNC 9 mirrors 000230 1778287827.790060 read 0x038667 thru 0x03ffff (0x7999 bytes) 000232 1778287827.803151 punch from 0x030014 to 0x03a676 (0xa663 bytes) 000233 1778287827.854661 trunc from 0x03f91b to 0x03ffff (0x6e5 bytes) 000234 1778287827.860626 write 0x003438 thru 0x008e7b (0x5a44 bytes) 000235 1778287827.874995 mapwrite 0x03a416 thru 0x03ffff (0x5bea bytes) 000236 1778287827.909845 fallocate 0x013e5c thru 0x01fa87 (0xbc2c bytes) 000237 1778287827.921013 read 0x00f24b thru 0x01951c (0xa2d2 bytes) 000238 1778287827.934211 mapread 0x03ec39 thru 0x03ffff (0x13c7 bytes) 000239 1778287827.940882 MIRROR_RESYNC 9 mirrors 000240 1778287830.169385 trunc from 0x040000 to 0x00572a (0x3a8d5 bytes) 000241 1778287830.223716 trunc from 0x00572b to 0x02ddb7 (0x2868d bytes) 000242 1778287830.253096 mapread 0x0015d6 thru 0x00b669 (0xa094 bytes) 000243 1778287830.326990 trunc from 0x02cf25 to 0x02ddb7 (0xe93 bytes) 000244 1778287830.335753 mapwrite 0x0154c3 thru 0x022d88 (0xd8c6 bytes) 000245 1778287830.409567 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 2: 0x52e6b7e7 (stale) Mirror 3: 0x52e6b7e7 (stale) Mirror 4: 0x52e6b7e7 (stale) Mirror 5: 0x52e6b7e7 (stale) Mirror 6: 0x52e6b7e7 (stale) Mirror 7: 0x52e6b7e7 (stale) Mirror 8: 0x52e6b7e7 (stale) Mirror 9: 0x52e6b7e7 (stale) CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 1: 0xfc9a5fc2 Mirror 10: 0xfc9a5fc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddb8: skipped 000246 1778287831.659204 mapread 0x020bc4 thru 0x02b4e5 (0xa922 bytes) 000247 1778287831.713333 punch from 0x013849 to 0x01daa6 (0xa25e bytes) 000248 1778287831.760325 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 2: 0x52e6b7e7 (stale) Mirror 3: 0x52e6b7e7 (stale) Mirror 4: 0x52e6b7e7 (stale) Mirror 5: 0x52e6b7e7 (stale) Mirror 6: 0x52e6b7e7 (stale) Mirror 7: 0x52e6b7e7 (stale) Mirror 8: 0x52e6b7e7 (stale) Mirror 9: 0x52e6b7e7 (stale) Mirror 10: 0xfc9a5fc2 (stale) CRC-32 checksum value for chunk [0, 0x2ddb8): Mirror 1: 0x9fe356fb Mirror 11: 0x9fe356fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddb8: skipped 000249 1778287833.581098 fallocate 0x03f4e9 thru 0x03ffff (0xb17 bytes) 000250 1778287833.676155 write 0x012cda thru 0x02176d (0xea94 bytes) 000251 1778287833.726137 mapread 0x00ef2d thru 0x016ea6 (0x7f7a bytes) 000252 1778287833.778156 mapwrite 0x003e8e thru 0x004f7b (0x10ee bytes) 000253 1778287833.875940 MIRROR_RESYNC 11 mirrors 000254 1778287837.529911 write 0x022bf1 thru 0x024a4e (0x1e5e bytes) 000255 1778287837.617314 mapread 0x024a5e thru 0x02ddb7 (0x935a bytes) 000256 1778287837.633963 trunc from 0x00caad to 0x019436 (0xc98a bytes) 000257 1778287837.641067 mapwrite 0x01b1e5 thru 0x01cb4a (0x1966 bytes) 000258 1778287837.677036 mapwrite 0x02f859 thru 0x03f1e8 (0xf990 bytes) 000260 1778287837.752030 mapread 0x024f52 thru 0x02e3a2 (0x9451 bytes) 000261 1778287837.765435 MIRROR_RESYNC 11 mirrors 000262 1778287841.101321 read 0x03ef78 thru 0x03f1e8 (0x271 bytes) 000263 1778287841.107874 mapread 0x02fecf thru 0x0398ed (0x9a1f bytes) 000264 1778287841.145763 punch from 0x03b039 to 0x03f1e8 (0x41b0 bytes) 000266 1778287841.220603 mapread 0x01f7b9 thru 0x0235b1 (0x3df9 bytes) 000267 1778287841.268965 mapread 0x013e6a thru 0x01e8f9 (0xaa90 bytes) 000268 1778287841.305189 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f1e9): Mirror 2: 0x38967399 (stale) Mirror 3: 0x38967399 (stale) Mirror 4: 0x38967399 (stale) Mirror 5: 0x38967399 (stale) Mirror 6: 0x38967399 (stale) Mirror 7: 0x38967399 (stale) Mirror 8: 0x38967399 (stale) Mirror 9: 0x38967399 (stale) Mirror 11: 0x38967399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1e9): Mirror 2: 0x38967399 (stale) Mirror 3: 0x38967399 (stale) Mirror 4: 0x38967399 (stale) Mirror 5: 0x38967399 (stale) Mirror 6: 0x38967399 (stale) Mirror 7: 0x38967399 (stale) Mirror 8: 0x38967399 (stale) Mirror 9: 0x38967399 (stale) Mirror 11: 0x38967399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1e9: skipped 000269 1778287842.245034 read 0x039244 thru 0x03efea (0x5da7 bytes) 000270 1778287842.274548 trunc from 0x01eb43 to 0x026e49 (0x8307 bytes) 000271 1778287842.283515 read 0x00f2f8 thru 0x01cb1d (0xd826 bytes) 000272 1778287842.327389 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f1e9): Mirror 2: 0x38967399 (stale) Mirror 4: 0x38967399 (stale) Mirror 5: 0x38967399 (stale) Mirror 6: 0x38967399 (stale) Mirror 7: 0x38967399 (stale) Mirror 8: 0x38967399 (stale) Mirror 9: 0x38967399 (stale) Mirror 11: 0x38967399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1e9): Mirror 2: 0x38967399 (stale) Mirror 4: 0x38967399 (stale) Mirror 5: 0x38967399 (stale) Mirror 6: 0x38967399 (stale) Mirror 7: 0x38967399 (stale) Mirror 8: 0x38967399 (stale) Mirror 9: 0x38967399 (stale) Mirror 11: 0x38967399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1e9: skipped 000273 1778287843.238265 write 0x000865 thru 0x008340 (0x7adc bytes) 000275 1778287843.275176 punch from 0x020c52 to 0x02dd14 (0xd0c3 bytes) 000276 1778287843.288195 trunc from 0x003376 to 0x0040c8 (0xd53 bytes) 000278 1778287843.318783 trunc from 0x03ab5a to 0x03f1e8 (0x468f bytes) 000279 1778287843.326996 trunc from 0x03f1e9 to 0x02d394 (0x11e54 bytes) 000280 1778287843.359430 mapread 0x017144 thru 0x025539 (0xe3f6 bytes) 000281 1778287843.410837 mapwrite 0x0171d7 thru 0x025e1a (0xec44 bytes) 000282 1778287843.464814 punch from 0x021c39 to 0x02a1bd (0x8585 bytes) 000283 1778287843.475580 write 0x00ab4f thru 0x011737 (0x6be9 bytes) 000284 1778287843.515686 MIRROR_RESYNC 9 mirrors 000285 1778287846.854107 mapread 0x0157ad thru 0x024d13 (0xf567 bytes) 000286 1778287846.914766 fallocate 0x0132d4 thru 0x020b54 (0xd881 bytes) 000287 1778287846.955213 trunc from 0x00ebcd to 0x013537 (0x496b bytes) 000288 1778287846.963206 fallocate 0x03772a thru 0x03ffff (0x88d6 bytes) 000289 1778287846.973464 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2d395): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d395): Mirror 2: 0x6c968882 (stale) Mirror 4: 0x6c968882 (stale) Mirror 5: 0x6c968882 (stale) Mirror 6: 0x6c968882 (stale) Mirror 7: 0x6c968882 (stale) Mirror 8: 0x6c968882 (stale) Mirror 9: 0x6c968882 (stale) Mirror 11: 0x6c968882 (stale) CRC-32 checksum value for chunk [0, 0x2d395): Mirror 1: 0xd87a6f95 Mirror 12: 0xd87a6f95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d395: skipped 000290 1778287848.129525 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d395): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d395): Mirror 2: 0x6c968882 (stale) Mirror 4: 0x6c968882 (stale) Mirror 5: 0x6c968882 (stale) Mirror 6: 0x6c968882 (stale) Mirror 7: 0x6c968882 (stale) Mirror 8: 0x6c968882 (stale) Mirror 9: 0x6c968882 (stale) Mirror 11: 0x6c968882 (stale) CRC-32 checksum value for chunk [0, 0x2d395): Mirror 1: 0xd87a6f95 Mirror 12: 0xd87a6f95 Mirror 13: 0xd87a6f95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d395: skipped 000291 1778287849.860253 read 0x008670 thru 0x014866 (0xc1f7 bytes) 000292 1778287849.877668 punch from 0x028b08 to 0x02d394 (0x488d bytes) 000293 1778287849.969849 fallocate 0x03bbb7 thru 0x03bc1f (0x69 bytes) 000294 1778287849.977042 trunc from 0x004a86 to 0x014173 (0xf6ee bytes) 000296 1778287849.996430 trunc from 0x026e0f to 0x0349a6 (0xdb98 bytes) 000297 1778287850.010078 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 2: 0xc10a6be5 (stale) Mirror 4: 0xc10a6be5 (stale) Mirror 5: 0xc10a6be5 (stale) Mirror 6: 0xc10a6be5 (stale) Mirror 7: 0xc10a6be5 (stale) Mirror 8: 0xc10a6be5 (stale) Mirror 9: 0xc10a6be5 (stale) Mirror 11: 0xc10a6be5 (stale) Mirror 12: 0x48c2435b (stale) Mirror 13: 0x48c2435b (stale) CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 1: 0x80a69216 Mirror 14: 0x80a69216 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc20: skipped 000298 1778287851.287870 trunc from 0x017ea2 to 0x01b303 (0x3462 bytes) 000299 1778287851.340310 trunc from 0x03bc20 to 0x00e71d (0x2d502 bytes) 000300 1778287851.371818 mapread 0x00bf41 thru 0x00e71d (0x27dd bytes) 000301 1778287851.405782 punch from 0x009dbc to 0x00e71d (0x4962 bytes) 000302 1778287851.412217 read 0x00b3f6 thru 0x00c06f (0xc7a bytes) 000303 1778287851.418706 punch from 0x009531 to 0x00e71d (0x51ed bytes) 000304 1778287851.425212 write 0x0193b3 thru 0x01c712 (0x3360 bytes) 000305 1778287851.427128 trunc from 0x01c713 to 0x037c94 (0x1b582 bytes) 000306 1778287851.448421 write 0x03f11a thru 0x03ffff (0xee6 bytes) 000307 1778287851.450222 mapwrite 0x02498e thru 0x024e4e (0x4c1 bytes) 000308 1778287851.479256 write 0x0047f6 thru 0x006c83 (0x248e bytes) 000309 1778287851.492908 mapread 0x02e26b thru 0x03bbfc (0xd992 bytes) 000310 1778287851.516603 trunc from 0x03b9bd to 0x03ffff (0x4643 bytes) 000311 1778287851.523181 trunc from 0x040000 to 0x009a71 (0x3658e bytes) 000312 1778287851.545806 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x8d5c6016 (stale) Mirror 4: 0x8d5c6016 (stale) Mirror 5: 0x8d5c6016 (stale) Mirror 6: 0x8d5c6016 (stale) Mirror 7: 0x8d5c6016 (stale) Mirror 8: 0x8d5c6016 (stale) Mirror 11: 0x8d5c6016 (stale) Mirror 12: 0x8d5c6016 (stale) Mirror 13: 0x8d5c6016 (stale) Mirror 14: 0xe315ed10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x8d5c6016 (stale) Mirror 4: 0x8d5c6016 (stale) Mirror 5: 0x8d5c6016 (stale) Mirror 6: 0x8d5c6016 (stale) Mirror 7: 0x8d5c6016 (stale) Mirror 8: 0x8d5c6016 (stale) Mirror 11: 0x8d5c6016 (stale) Mirror 12: 0x8d5c6016 (stale) Mirror 13: 0x8d5c6016 (stale) Mirror 14: 0xe315ed10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a72: skipped 000313 1778287852.113162 fallocate 0x023c92 thru 0x02bf01 (0x8270 bytes) 000314 1778287852.120332 mapwrite 0x0068b9 thru 0x01476a (0xdeb2 bytes) 000315 1778287852.202528 mapwrite 0x01a8b5 thru 0x01bd8d (0x14d9 bytes) 000316 1778287852.252920 read 0x00d656 thru 0x00f53d (0x1ee8 bytes) 000317 1778287852.258900 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1bd8e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd8e): Mirror 2: 0x9b3073c5 (stale) Mirror 4: 0x9b3073c5 (stale) Mirror 5: 0x9b3073c5 (stale) Mirror 6: 0x9b3073c5 (stale) Mirror 7: 0x9b3073c5 (stale) Mirror 8: 0x9b3073c5 (stale) Mirror 11: 0x9b3073c5 (stale) Mirror 12: 0x89336ade (stale) Mirror 13: 0x89336ade (stale) Mirror 14: 0x7c776701 (stale) CRC-32 checksum value for chunk [0, 0x1bd8e): Mirror 1: 0x3e7996d Mirror 15: 0x3e7996d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd8e: skipped 000318 1778287853.264814 trunc from 0x01bd8e to 0x02717d (0xb3f0 bytes) 000319 1778287853.406519 punch from 0x015740 to 0x022080 (0xc941 bytes) 000320 1778287853.414239 trunc from 0x02717e to 0x00d69d (0x19ae0 bytes) 000321 1778287853.435527 punch from 0x000d49 to 0x003403 (0x26bb bytes) 000325 1778287853.444446 write 0x008a80 thru 0x015900 (0xce81 bytes) 000326 1778287853.454467 fallocate 0x02f9c0 thru 0x03f84f (0xfe90 bytes) 000328 1778287853.460971 trunc from 0x03f850 to 0x00bd07 (0x33b48 bytes) 000329 1778287853.485140 read 0x00b485 thru 0x00bd07 (0x883 bytes) 000330 1778287853.486112 write 0x011978 thru 0x01355e (0x1be7 bytes) 000331 1778287853.489240 read 0x00f8ed thru 0x01355e (0x3c72 bytes) 000332 1778287853.499688 trunc from 0x0027d8 to 0x00fd77 (0xd5a0 bytes) 000333 1778287853.516728 write 0x01c327 thru 0x0256d5 (0x93af bytes) 000334 1778287853.518459 trunc from 0x0256d6 to 0x029aa4 (0x43cf bytes) 000335 1778287853.538890 trunc from 0x010321 to 0x016ae4 (0x67c4 bytes) 000336 1778287853.544328 read 0x0082d1 thru 0x0104bf (0x81ef bytes) 000337 1778287853.559994 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29aa5): Mirror 2: 0x4c133468 (stale) Mirror 4: 0x4c133468 (stale) Mirror 5: 0x4c133468 (stale) Mirror 6: 0x4c133468 (stale) Mirror 7: 0x4c133468 (stale) Mirror 8: 0x4c133468 (stale) Mirror 12: 0x20819e2f (stale) Mirror 13: 0x20819e2f (stale) Mirror 14: 0x26872ae3 (stale) Mirror 15: 0xcb15374d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29aa5): Mirror 2: 0x4c133468 (stale) Mirror 4: 0x4c133468 (stale) Mirror 5: 0x4c133468 (stale) Mirror 6: 0x4c133468 (stale) Mirror 7: 0x4c133468 (stale) Mirror 8: 0x4c133468 (stale) Mirror 12: 0x20819e2f (stale) Mirror 13: 0x20819e2f (stale) Mirror 14: 0x26872ae3 (stale) Mirror 15: 0xcb15374d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29aa5: skipped 000338 1778287854.509299 read 0x020ad7 thru 0x023af0 (0x301a bytes) 000339 1778287854.522279 mapwrite 0x03c705 thru 0x03ffff (0x38fb bytes) 000340 1778287854.697033 mapwrite 0x00715c thru 0x014f6d (0xde12 bytes) 000341 1778287854.830428 fallocate 0x02c1f0 thru 0x034110 (0x7f21 bytes) 000343 1778287854.855545 punch from 0x0335e8 to 0x03f38c (0xbda5 bytes) 000344 1778287854.872219 trunc from 0x0097e2 to 0x010b94 (0x73b3 bytes) 000345 1778287854.892795 fallocate 0x03908b thru 0x03ffff (0x6f75 bytes) 000347 1778287854.904836 trunc from 0x02adb3 to 0x032214 (0x7462 bytes) 000348 1778287854.923249 punch from 0x014d1c to 0x020489 (0xb76e bytes) 000350 1778287854.947138 mapwrite 0x006fea thru 0x00e262 (0x7279 bytes) 000351 1778287855.025511 trunc from 0x040000 to 0x00d842 (0x327bd bytes) 000352 1778287855.074315 trunc from 0x00d843 to 0x013b11 (0x62cf bytes) 000354 1778287855.118189 fallocate 0x020f4a thru 0x0246ff (0x37b6 bytes) 000355 1778287855.140575 trunc from 0x024700 to 0x020137 (0x45c8 bytes) 000357 1778287855.173150 MIRROR_RESYNC 11 mirrors 000358 1778287858.162642 fallocate 0x01b03f thru 0x028a71 (0xda33 bytes) 000360 1778287858.202683 read 0x01e006 thru 0x025b57 (0x7b52 bytes) 000361 1778287858.214188 trunc from 0x0152c1 to 0x01a9e1 (0x5721 bytes) 000363 1778287858.220083 punch from 0x00f4d9 to 0x017c5f (0x8787 bytes) 000364 1778287858.225632 punch from 0x0209a2 to 0x0214c3 (0xb22 bytes) 000366 1778287858.233300 read 0x0170e9 thru 0x01dbd6 (0x6aee bytes) 000367 1778287858.249862 read 0x024035 thru 0x024f04 (0xed0 bytes) 000368 1778287858.251020 punch from 0x026159 to 0x028a71 (0x2919 bytes) 000369 1778287858.256710 mapwrite 0x01c13e thru 0x024c19 (0x8adc bytes) 000370 1778287858.281403 punch from 0x0278ca to 0x028a71 (0x11a8 bytes) 000372 1778287858.288075 fallocate 0x00fd4b thru 0x01e6b5 (0xe96b bytes) 000373 1778287858.296459 punch from 0x01fc36 to 0x023f8e (0x4359 bytes) 000374 1778287858.304489 write 0x0284c9 thru 0x036b5c (0xe694 bytes) 000375 1778287858.316543 trunc from 0x012c02 to 0x01d2dd (0xa6dc bytes) 000376 1778287858.322634 trunc from 0x036b5d to 0x0255d2 (0x1158a bytes) 000377 1778287858.352415 mapwrite 0x0289c6 thru 0x03767b (0xecb6 bytes) 000378 1778287858.426104 read 0x007a82 thru 0x00b758 (0x3cd7 bytes) 000379 1778287858.432424 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3767c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3767c): Mirror 2: 0x495212ce (stale) Mirror 4: 0x495212ce (stale) Mirror 5: 0x495212ce (stale) Mirror 6: 0x495212ce (stale) Mirror 7: 0x495212ce (stale) Mirror 8: 0x495212ce (stale) Mirror 12: 0x495212ce (stale) Mirror 13: 0x495212ce (stale) Mirror 14: 0x495212ce (stale) Mirror 15: 0x495212ce (stale) CRC-32 checksum value for chunk [0, 0x3767c): Mirror 1: 0x6f568ced Mirror 16: 0x6f568ced lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3767c: skipped 000380 1778287859.936883 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3767c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3767c): Mirror 2: 0x495212ce (stale) Mirror 4: 0x495212ce (stale) Mirror 5: 0x495212ce (stale) Mirror 6: 0x495212ce (stale) Mirror 7: 0x495212ce (stale) Mirror 8: 0x495212ce (stale) Mirror 12: 0x495212ce (stale) Mirror 13: 0x495212ce (stale) Mirror 14: 0x495212ce (stale) Mirror 15: 0x495212ce (stale) CRC-32 checksum value for chunk [0, 0x3767c): Mirror 1: 0x6f568ced Mirror 16: 0x6f568ced Mirror 17: 0x6f568ced lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3767c: skipped 000381 1778287861.698730 MIRROR_RESYNC 13 mirrors 000382 1778287866.653024 trunc from 0x03767c to 0x02213a (0x15541 bytes) 000383 1778287866.738989 write 0x0160aa thru 0x02592b (0xf882 bytes) 000384 1778287866.754475 fallocate 0x019556 thru 0x028d22 (0xf7cd bytes) 000385 1778287866.766473 read 0x00efae thru 0x0102ab (0x12fe bytes) 000386 1778287866.787330 mapwrite 0x026c55 thru 0x032e13 (0xc1bf bytes) 000387 1778287866.912095 mapwrite 0x03994c thru 0x03ffff (0x66b4 bytes) 000388 1778287867.011070 trunc from 0x031f93 to 0x032f99 (0x1007 bytes) 000389 1778287867.022895 write 0x017d89 thru 0x020c0d (0x8e85 bytes) 000390 1778287867.028230 read 0x0049c9 thru 0x008b3f (0x4177 bytes) 000391 1778287867.074475 fallocate 0x01e95e thru 0x01f8b5 (0xf58 bytes) 000392 1778287867.088079 read 0x00613d thru 0x00b84c (0x5710 bytes) 000393 1778287867.090201 fallocate 0x003303 thru 0x005ec4 (0x2bc2 bytes) 000394 1778287867.099630 mapread 0x0383e5 thru 0x03ffff (0x7c1b bytes) 000395 1778287867.115866 mapwrite 0x025206 thru 0x02df61 (0x8d5c bytes) 000396 1778287867.180460 fallocate 0x02506d thru 0x032d8c (0xdd20 bytes) 000397 1778287867.189694 mapwrite 0x03a09a thru 0x03ffff (0x5f66 bytes) 000398 1778287867.219844 punch from 0x002f6b to 0x00af8b (0x8021 bytes) 000399 1778287867.228867 mapwrite 0x001077 thru 0x00652d (0x54b7 bytes) 000401 1778287867.276695 mapread 0x031cc5 thru 0x037e68 (0x61a4 bytes) 000402 1778287867.280709 trunc from 0x0005e3 to 0x003bd6 (0x35f4 bytes) 000403 1778287867.291315 mapread 0x030593 thru 0x031eda (0x1948 bytes) 000404 1778287867.295198 trunc from 0x028e83 to 0x02af47 (0x20c5 bytes) 000405 1778287867.305392 punch from 0x021bae to 0x02c290 (0xa6e3 bytes) 000406 1778287867.319082 trunc from 0x0232a1 to 0x024ec9 (0x1c29 bytes) 000408 1778287867.327322 read 0x028da6 thru 0x02d70d (0x4968 bytes) 000409 1778287867.351184 mapread 0x03eb68 thru 0x03f18e (0x627 bytes) 000410 1778287867.352311 read 0x008eb4 thru 0x016f23 (0xe070 bytes) 000411 1778287867.360776 write 0x00e343 thru 0x01d68e (0xf34c bytes) 000412 1778287867.377278 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1460b95b (stale) Mirror 5: 0x1460b95b (stale) Mirror 6: 0x1460b95b (stale) Mirror 7: 0x1460b95b (stale) Mirror 8: 0x1460b95b (stale) Mirror 12: 0x1460b95b (stale) Mirror 13: 0x1460b95b (stale) Mirror 14: 0x1460b95b (stale) Mirror 15: 0x1460b95b (stale) Mirror 16: 0x1460b95b (stale) Mirror 17: 0x1460b95b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1460b95b (stale) Mirror 5: 0x1460b95b (stale) Mirror 6: 0x1460b95b (stale) Mirror 7: 0x1460b95b (stale) Mirror 8: 0x1460b95b (stale) Mirror 12: 0x1460b95b (stale) Mirror 13: 0x1460b95b (stale) Mirror 14: 0x1460b95b (stale) Mirror 15: 0x1460b95b (stale) Mirror 16: 0x1460b95b (stale) Mirror 17: 0x1460b95b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000413 1778287868.586663 mapread 0x00eec3 thru 0x013c7e (0x4dbc bytes) 000415 1778287868.643315 write 0x033b46 thru 0x0388bf (0x4d7a bytes) 000416 1778287868.672149 mapwrite 0x03bbf8 thru 0x03ecaa (0x30b3 bytes) 000417 1778287868.746097 read 0x019e62 thru 0x02392d (0x9acc bytes) 000418 1778287868.780900 fallocate 0x025438 thru 0x028995 (0x355e bytes) 000419 1778287868.795638 write 0x026436 thru 0x02aca9 (0x4874 bytes) 000420 1778287868.843503 mapread 0x020c87 thru 0x02a2c8 (0x9642 bytes) 000421 1778287868.848026 mapwrite 0x01944a thru 0x026aa6 (0xd65d bytes) 000422 1778287868.890327 read 0x00e035 thru 0x01ab2d (0xcaf9 bytes) 000423 1778287868.894912 write 0x00a487 thru 0x00d171 (0x2ceb bytes) 000427 1778287868.909943 write 0x02e786 thru 0x02f7ca (0x1045 bytes) 000428 1778287868.913233 mapread 0x03c160 thru 0x03ffff (0x3ea0 bytes) 000429 1778287868.916385 trunc from 0x03a387 to 0x03ffff (0x5c79 bytes) 000430 1778287868.940323 MIRROR_RESYNC 12 mirrors 000431 1778287874.269102 write 0x001f3e thru 0x00f1f4 (0xd2b7 bytes) 000433 1778287874.336936 read 0x03a3b2 thru 0x03e447 (0x4096 bytes) 000435 1778287874.349731 trunc from 0x040000 to 0x03c5aa (0x3a55 bytes) 000436 1778287874.417537 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3c5ab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5ab): Mirror 4: 0xbf81ad0 (stale) Mirror 5: 0xbf81ad0 (stale) Mirror 6: 0xbf81ad0 (stale) Mirror 7: 0xbf81ad0 (stale) Mirror 8: 0xbf81ad0 (stale) Mirror 12: 0xbf81ad0 (stale) Mirror 13: 0xbf81ad0 (stale) Mirror 14: 0xbf81ad0 (stale) Mirror 15: 0xbf81ad0 (stale) Mirror 16: 0xbf81ad0 (stale) Mirror 17: 0xbf81ad0 (stale) CRC-32 checksum value for chunk [0, 0x3c5ab): Mirror 1: 0xe81905c7 Mirror 18: 0xe81905c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5ab: skipped 000437 1778287876.904761 trunc from 0x03c5ab to 0x031ae7 (0xaac3 bytes) 000438 1778287877.012233 mapwrite 0x020e86 thru 0x026211 (0x538c bytes) 000439 1778287877.064834 trunc from 0x031ae8 to 0x02b017 (0x6ad0 bytes) 000440 1778287877.098819 mapwrite 0x01d2ca thru 0x025c5e (0x8995 bytes) 000441 1778287877.126618 trunc from 0x02b018 to 0x020fdb (0xa03c bytes) 000442 1778287877.146312 mapwrite 0x01e76e thru 0x022a65 (0x42f8 bytes) 000443 1778287877.191800 punch from 0x0225f7 to 0x022a65 (0x46f bytes) 000444 1778287877.198958 read 0x003123 thru 0x00b57f (0x845d bytes) 000445 1778287877.220794 write 0x037836 thru 0x03ffff (0x87ca bytes) 000446 1778287877.227910 fallocate 0x02695d thru 0x029f88 (0x362c bytes) 000447 1778287877.239765 mapwrite 0x032d16 thru 0x03e440 (0xb72b bytes) 000448 1778287877.287976 mapwrite 0x035f9a thru 0x03aa66 (0x4acd bytes) 000449 1778287877.338174 trunc from 0x040000 to 0x00be25 (0x341da bytes) 000450 1778287877.387666 read 0x009379 thru 0x00be25 (0x2aad bytes) 000451 1778287877.389579 read 0x008f5d thru 0x00be25 (0x2ec9 bytes) 000452 1778287877.390325 trunc from 0x00b40d to 0x00be25 (0xa19 bytes) 000453 1778287877.401240 trunc from 0x00be26 to 0x01279b (0x6976 bytes) 000454 1778287877.460657 mapread 0x0024eb thru 0x01155d (0xf073 bytes) 000455 1778287877.489231 punch from 0x003ef5 to 0x004bf0 (0xcfc bytes) 000456 1778287877.510970 MIRROR_RESYNC 13 mirrors 000457 1778287880.536468 trunc from 0x01279c to 0x02fc6b (0x1d4d0 bytes) 000459 1778287880.624825 write 0x027ead thru 0x0318d9 (0x9a2d bytes) 000460 1778287880.637013 read 0x029525 thru 0x0318d9 (0x83b5 bytes) 000461 1778287880.637737 mapwrite 0x03895b thru 0x03ffff (0x76a5 bytes) 000462 1778287880.763513 punch from 0x02871f to 0x037b2d (0xf40f bytes) 000463 1778287880.779890 mapwrite 0x0057ae thru 0x010cd8 (0xb52b bytes) 000464 1778287880.836233 trunc from 0x040000 to 0x0094f3 (0x36b0c bytes) 000465 1778287880.867681 trunc from 0x00493c to 0x0094f3 (0x4bb8 bytes) 000466 1778287880.879921 mapwrite 0x036c8b thru 0x03ffff (0x9375 bytes) 000467 1778287880.986131 trunc from 0x040000 to 0x000277 (0x3fd88 bytes) 000468 1778287881.016233 punch from 0x00015a to 0x000277 (0x11e bytes) 000469 1778287881.023092 punch from 0x000070 to 0x000277 (0x208 bytes) 000470 1778287881.033046 read 0x000063 thru 0x000277 (0x215 bytes) 000471 1778287881.041767 mapread 0x000129 thru 0x000277 (0x14f bytes) 000472 1778287881.043033 trunc from 0x000042 to 0x000277 (0x236 bytes) 000473 1778287881.060125 mapread 0x000058 thru 0x000277 (0x220 bytes) 000474 1778287881.070954 write 0x009dce thru 0x014868 (0xaa9b bytes) 000475 1778287881.078194 fallocate 0x035b48 thru 0x03c0a2 (0x655b bytes) 000476 1778287881.096558 fallocate 0x000d16 thru 0x010c67 (0xff52 bytes) 000477 1778287881.110678 mapread 0x0112f2 thru 0x014868 (0x3577 bytes) 000478 1778287881.112142 trunc from 0x00502a to 0x00852d (0x3504 bytes) 000479 1778287881.119127 trunc from 0x014869 to 0x024978 (0x10110 bytes) 000480 1778287881.141707 mapwrite 0x0202af thru 0x0272d4 (0x7026 bytes) 000482 1778287881.213129 mapread 0x002513 thru 0x011757 (0xf245 bytes) 000483 1778287881.217425 mapwrite 0x034442 thru 0x03519e (0xd5d bytes) 000484 1778287881.264663 trunc from 0x03519f to 0x00e29f (0x26eff bytes) 000485 1778287881.293683 write 0x038127 thru 0x03ffff (0x7ed9 bytes) 000486 1778287881.304463 read 0x0314e0 thru 0x03d40d (0xbf2e bytes) 000487 1778287881.331290 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 6: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b1b1727 Mirror 19: 0x4b1b1727 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1778287883.181444 fallocate 0x022000 thru 0x02e8bd (0xc8be bytes) 000489 1778287883.257526 trunc from 0x02daf5 to 0x02edc0 (0x12cc bytes) 000490 1778287883.264783 trunc from 0x01a3bf to 0x0209ec (0x662e bytes) 000492 1778287883.281499 mapwrite 0x003176 thru 0x009ff6 (0x6e81 bytes) 000493 1778287883.433286 mapwrite 0x006d9d thru 0x0075d7 (0x83b bytes) 000494 1778287883.511454 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 6: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf7b1e33 Mirror 20: 0xaf7b1e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000495 1778287885.981311 fallocate 0x03a113 thru 0x03ffff (0x5eed bytes) 000496 1778287886.093524 mapwrite 0x03d72b thru 0x03ffff (0x28d5 bytes) 000497 1778287886.216606 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 6: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 6: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1778287887.186874 trunc from 0x040000 to 0x02c3ac (0x13c53 bytes) 000499 1778287887.220566 fallocate 0x000caa thru 0x00ea29 (0xdd80 bytes) 000501 1778287887.229402 fallocate 0x00e655 thru 0x019369 (0xad15 bytes) 000502 1778287887.237215 fallocate 0x0136ae thru 0x018952 (0x52a5 bytes) 000503 1778287887.247866 mapread 0x012d7a thru 0x02037c (0xd603 bytes) 000505 1778287887.319748 mapwrite 0x01c225 thru 0x02463e (0x841a bytes) 000506 1778287887.354313 mapread 0x0105cb thru 0x01a007 (0x9a3d bytes) 000507 1778287887.357946 mapread 0x00a98f thru 0x01a5de (0xfc50 bytes) 000508 1778287887.384297 mapread 0x006602 thru 0x01463e (0xe03d bytes) 000509 1778287887.389418 mapwrite 0x01c83f thru 0x021b42 (0x5304 bytes) 000510 1778287887.431205 read 0x008561 thru 0x00fc1b (0x76bb bytes) 000511 1778287887.435097 fallocate 0x019ff4 thru 0x02452f (0xa53c bytes) 000512 1778287887.456670 mapread 0x00a51a thru 0x01521d (0xad04 bytes) 000513 1778287887.474561 mapwrite 0x020d30 thru 0x027779 (0x6a4a bytes) 000515 1778287887.597933 mapwrite 0x031c60 thru 0x03a251 (0x85f2 bytes) 000516 1778287887.729322 trunc from 0x01ea96 to 0x021813 (0x2d7e bytes) 000518 1778287887.744042 mapread 0x007c9f thru 0x009af1 (0x1e53 bytes) 000519 1778287887.745215 punch from 0x01e658 to 0x02b0f2 (0xca9b bytes) 000520 1778287887.770491 trunc from 0x02f06b to 0x030322 (0x12b8 bytes) 000521 1778287887.792685 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3a252): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a252): Mirror 4: 0x93f9b1e6 (stale) Mirror 5: 0x93f9b1e6 (stale) Mirror 6: 0x93f9b1e6 (stale) Mirror 7: 0x93f9b1e6 (stale) Mirror 8: 0x93f9b1e6 (stale) Mirror 12: 0x93f9b1e6 (stale) Mirror 13: 0x93f9b1e6 (stale) Mirror 14: 0x93f9b1e6 (stale) Mirror 15: 0x93f9b1e6 (stale) Mirror 16: 0x93f9b1e6 (stale) Mirror 17: 0x93f9b1e6 (stale) Mirror 18: 0x93f9b1e6 (stale) Mirror 19: 0xf65f56d8 (stale) CRC-32 checksum value for chunk [0, 0x3a252): Mirror 1: 0xa09025de Mirror 20: 0xa09025de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a252: skipped 000522 1778287889.331432 trunc from 0x03a252 to 0x01706f (0x231e2 bytes) 000523 1778287889.403776 trunc from 0x017070 to 0x035dba (0x1ed4b bytes) 000525 1778287889.428135 fallocate 0x003acb thru 0x00c7a2 (0x8cd8 bytes) 000526 1778287889.434770 mapread 0x0154f9 thru 0x01803b (0x2b43 bytes) 000528 1778287889.460475 write 0x03c2e3 thru 0x03ffff (0x3d1d bytes) 000529 1778287889.467746 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) Mirror 20: 0xca196b4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) Mirror 20: 0xca196b4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000530 1778287890.396464 write 0x019408 thru 0x01cde7 (0x39e0 bytes) 000531 1778287890.412639 trunc from 0x01cee6 to 0x025710 (0x882b bytes) 000532 1778287890.427725 trunc from 0x0262e9 to 0x02a751 (0x4469 bytes) 000533 1778287890.432957 fallocate 0x030028 thru 0x036531 (0x650a bytes) 000534 1778287890.440932 trunc from 0x0381d7 to 0x03ffff (0x7e29 bytes) 000536 1778287890.446776 mapread 0x010301 thru 0x01af79 (0xac79 bytes) 000537 1778287890.459841 mapwrite 0x012bde thru 0x01f87e (0xcca1 bytes) 000538 1778287890.500110 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe1dba691 (stale) Mirror 5: 0xe1dba691 (stale) Mirror 7: 0xe1dba691 (stale) Mirror 8: 0xe1dba691 (stale) Mirror 12: 0xe1dba691 (stale) Mirror 13: 0xe1dba691 (stale) Mirror 14: 0xe1dba691 (stale) Mirror 15: 0xe1dba691 (stale) Mirror 16: 0xe1dba691 (stale) Mirror 17: 0xe1dba691 (stale) Mirror 18: 0xe1dba691 (stale) Mirror 19: 0x4b1b1727 (stale) Mirror 20: 0xca196b4e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe16944fc Mirror 21: 0xe16944fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1778287892.730815 trunc from 0x040000 to 0x00b31e (0x34ce1 bytes) 000541 1778287892.816781 mapwrite 0x014896 thru 0x01732e (0x2a99 bytes) 000542 1778287892.891087 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1732f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1732f): Mirror 4: 0x1972c512 (stale) Mirror 5: 0x1972c512 (stale) Mirror 7: 0x1972c512 (stale) Mirror 8: 0x1972c512 (stale) Mirror 12: 0x1972c512 (stale) Mirror 13: 0x1972c512 (stale) Mirror 14: 0x1972c512 (stale) Mirror 15: 0x1972c512 (stale) Mirror 16: 0x1972c512 (stale) Mirror 17: 0x1972c512 (stale) Mirror 18: 0x1972c512 (stale) Mirror 19: 0xcd4d7e39 (stale) Mirror 20: 0xd4740455 (stale) Mirror 21: 0x61da40b4 (stale) CRC-32 checksum value for chunk [0, 0x1732f): Mirror 1: 0x999039b5 Mirror 22: 0x999039b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1732f: skipped 000543 1778287894.360716 read 0x002303 thru 0x00994d (0x764b bytes) 000544 1778287894.388906 punch from 0x0022b2 to 0x00ca3d (0xa78c bytes) 000545 1778287894.463895 write 0x03202c thru 0x035f2f (0x3f04 bytes) 000546 1778287894.470796 write 0x039bf7 thru 0x03ffff (0x6409 bytes) 000547 1778287894.474325 mapwrite 0x0321fa thru 0x03310c (0xf13 bytes) 000551 1778287894.496043 punch from 0x02d855 to 0x02e2b8 (0xa64 bytes) 000552 1778287894.501026 mapread 0x02957d thru 0x02bd34 (0x27b8 bytes) 000553 1778287894.528657 mapwrite 0x014138 thru 0x01a5b2 (0x647b bytes) 000554 1778287894.583618 fallocate 0x0105e9 thru 0x01a3b0 (0x9dc8 bytes) 000555 1778287894.591723 mapread 0x002493 thru 0x00759c (0x510a bytes) 000556 1778287894.614535 trunc from 0x029fb4 to 0x0396ac (0xf6f9 bytes) 000557 1778287894.623898 trunc from 0x01523e to 0x01b5cf (0x6392 bytes) 000558 1778287894.633831 mapwrite 0x00bdb9 thru 0x01a457 (0xe69f bytes) 000559 1778287894.683300 write 0x005b30 thru 0x0061db (0x6ac bytes) 000560 1778287894.687048 write 0x02160a thru 0x03119b (0xfb92 bytes) 000561 1778287894.707199 trunc from 0x040000 to 0x02e0b8 (0x11f47 bytes) 000562 1778287894.758383 fallocate 0x039b76 thru 0x03ffff (0x648a bytes) 000563 1778287894.766431 punch from 0x039a10 to 0x03ffff (0x65f0 bytes) 000564 1778287894.779944 read 0x02e0e3 thru 0x03da70 (0xf98e bytes) 000565 1778287894.800793 write 0x0272a2 thru 0x030609 (0x9368 bytes) 000566 1778287894.810431 mapread 0x03ae21 thru 0x03ffff (0x51df bytes) 000567 1778287894.811694 mapwrite 0x01a93e thru 0x0294ed (0xebb0 bytes) 000568 1778287894.842913 punch from 0x03a9dc to 0x03ffff (0x5624 bytes) 000569 1778287894.850816 write 0x004c27 thru 0x00858c (0x3966 bytes) 000570 1778287894.852830 mapread 0x01116c thru 0x01c092 (0xaf27 bytes) 000571 1778287894.866490 MIRROR_RESYNC 16 mirrors 000572 1778287899.246697 mapread 0x01042d thru 0x018bdb (0x87af bytes) 000573 1778287899.281570 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) 000574 1778287899.932990 write 0x0326c3 thru 0x03cb90 (0xa4ce bytes) 000575 1778287900.027885 mapread 0x00d90f thru 0x0144ae (0x6ba0 bytes) 000576 1778287900.054416 punch from 0x02b720 to 0x031ed2 (0x67b3 bytes) 000577 1778287900.067138 read 0x007e00 thru 0x00991b (0x1b1c bytes) 000578 1778287900.068018 punch from 0x00458f to 0x00a76d (0x61df bytes) 000579 1778287900.077230 punch from 0x030d81 to 0x03296f (0x1bef bytes) 000580 1778287900.099889 mapread 0x0106b2 thru 0x013e64 (0x37b3 bytes) 000581 1778287900.101189 trunc from 0x03e2b1 to 0x03ffff (0x1d4f bytes) 000582 1778287900.115266 write 0x022dac thru 0x031c5d (0xeeb2 bytes) 000583 1778287900.139912 trunc from 0x00cf88 to 0x0101b3 (0x322c bytes) 000584 1778287900.144512 punch from 0x001c95 to 0x00a2d9 (0x8645 bytes) 000585 1778287900.151799 write 0x02ee17 thru 0x033668 (0x4852 bytes) 000586 1778287900.155917 read 0x01b027 thru 0x01d190 (0x216a bytes) 000587 1778287900.159314 punch from 0x01f932 to 0x025a57 (0x6126 bytes) 000588 1778287900.182020 punch from 0x037646 to 0x03ffff (0x89ba bytes) 000589 1778287900.191963 trunc from 0x020d55 to 0x02ba10 (0xacbc bytes) 000590 1778287900.203912 mapwrite 0x02918e thru 0x02aefc (0x1d6f bytes) 000591 1778287900.259006 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf337d1d0 (stale) Mirror 5: 0xf337d1d0 (stale) Mirror 7: 0xf337d1d0 (stale) Mirror 8: 0xf337d1d0 (stale) Mirror 12: 0xf337d1d0 (stale) Mirror 13: 0xf337d1d0 (stale) Mirror 14: 0xf337d1d0 (stale) Mirror 16: 0xf337d1d0 (stale) Mirror 18: 0xf337d1d0 (stale) Mirror 19: 0xf337d1d0 (stale) Mirror 20: 0xf337d1d0 (stale) Mirror 21: 0xf337d1d0 (stale) Mirror 22: 0xf337d1d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf337d1d0 (stale) Mirror 5: 0xf337d1d0 (stale) Mirror 7: 0xf337d1d0 (stale) Mirror 8: 0xf337d1d0 (stale) Mirror 12: 0xf337d1d0 (stale) Mirror 13: 0xf337d1d0 (stale) Mirror 14: 0xf337d1d0 (stale) Mirror 16: 0xf337d1d0 (stale) Mirror 18: 0xf337d1d0 (stale) Mirror 19: 0xf337d1d0 (stale) Mirror 20: 0xf337d1d0 (stale) Mirror 21: 0xf337d1d0 (stale) Mirror 22: 0xf337d1d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000592 1778287901.323864 fallocate 0x0239ec thru 0x025db9 (0x23ce bytes) 000593 1778287901.329726 mapwrite 0x008561 thru 0x015593 (0xd033 bytes) 000594 1778287901.382537 mapread 0x01831d thru 0x027f4c (0xfc30 bytes) 000595 1778287901.402616 mapread 0x031aed thru 0x038b25 (0x7039 bytes) 000596 1778287901.422780 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf337d1d0 (stale) Mirror 5: 0xf337d1d0 (stale) Mirror 7: 0xf337d1d0 (stale) Mirror 12: 0xf337d1d0 (stale) Mirror 13: 0xf337d1d0 (stale) Mirror 14: 0xf337d1d0 (stale) Mirror 16: 0xf337d1d0 (stale) Mirror 18: 0xf337d1d0 (stale) Mirror 19: 0xf337d1d0 (stale) Mirror 20: 0xf337d1d0 (stale) Mirror 21: 0xf337d1d0 (stale) Mirror 22: 0xf337d1d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf337d1d0 (stale) Mirror 5: 0xf337d1d0 (stale) Mirror 7: 0xf337d1d0 (stale) Mirror 12: 0xf337d1d0 (stale) Mirror 13: 0xf337d1d0 (stale) Mirror 14: 0xf337d1d0 (stale) Mirror 16: 0xf337d1d0 (stale) Mirror 18: 0xf337d1d0 (stale) Mirror 19: 0xf337d1d0 (stale) Mirror 20: 0xf337d1d0 (stale) Mirror 21: 0xf337d1d0 (stale) Mirror 22: 0xf337d1d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1778287902.335891 mapread 0x01614a thru 0x01a6c4 (0x457b bytes) 000598 1778287902.369080 mapwrite 0x02032b thru 0x028e61 (0x8b37 bytes) 000600 1778287902.423587 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf337d1d0 (stale) Mirror 5: 0xf337d1d0 (stale) Mirror 7: 0xf337d1d0 (stale) Mirror 12: 0xf337d1d0 (stale) Mirror 13: 0xf337d1d0 (stale) Mirror 14: 0xf337d1d0 (stale) Mirror 16: 0xf337d1d0 (stale) Mirror 18: 0xf337d1d0 (stale) Mirror 19: 0xf337d1d0 (stale) Mirror 20: 0xf337d1d0 (stale) Mirror 21: 0xf337d1d0 (stale) Mirror 22: 0xf337d1d0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee6b979c Mirror 23: 0xee6b979c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000601 1778287904.027205 mapread 0x03f777 thru 0x03ffff (0x889 bytes) 000602 1778287904.057244 MIRROR_RESYNC 14 mirrors 000603 1778287907.238016 write 0x01b6ae thru 0x021206 (0x5b59 bytes) 000604 1778287907.405929 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee6b979c (stale) Mirror 5: 0xee6b979c (stale) Mirror 7: 0xee6b979c (stale) Mirror 12: 0xee6b979c (stale) Mirror 13: 0xee6b979c (stale) Mirror 14: 0xee6b979c (stale) Mirror 16: 0xee6b979c (stale) Mirror 18: 0xee6b979c (stale) Mirror 19: 0xee6b979c (stale) Mirror 20: 0xee6b979c (stale) Mirror 21: 0xee6b979c (stale) Mirror 23: 0xee6b979c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee6b979c (stale) Mirror 5: 0xee6b979c (stale) Mirror 7: 0xee6b979c (stale) Mirror 12: 0xee6b979c (stale) Mirror 13: 0xee6b979c (stale) Mirror 14: 0xee6b979c (stale) Mirror 16: 0xee6b979c (stale) Mirror 18: 0xee6b979c (stale) Mirror 19: 0xee6b979c (stale) Mirror 20: 0xee6b979c (stale) Mirror 21: 0xee6b979c (stale) Mirror 23: 0xee6b979c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1778287908.514629 mapwrite 0x01ba72 thru 0x01f859 (0x3de8 bytes) 000606 1778287908.569076 mapread 0x01a3ee thru 0x0288e2 (0xe4f5 bytes) 000607 1778287908.596617 trunc from 0x040000 to 0x016c86 (0x29379 bytes) 000608 1778287908.677573 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x16c87): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16c87): Mirror 4: 0x8d1c19fa (stale) Mirror 5: 0x8d1c19fa (stale) Mirror 7: 0x8d1c19fa (stale) Mirror 12: 0x8d1c19fa (stale) Mirror 13: 0x8d1c19fa (stale) Mirror 14: 0x8d1c19fa (stale) Mirror 16: 0x8d1c19fa (stale) Mirror 18: 0x8d1c19fa (stale) Mirror 19: 0x8d1c19fa (stale) Mirror 20: 0x8d1c19fa (stale) Mirror 21: 0x8d1c19fa (stale) Mirror 23: 0x8d1c19fa (stale) CRC-32 checksum value for chunk [0, 0x16c87): Mirror 1: 0x8d1c19fa Mirror 24: 0x8d1c19fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16c87: skipped 000610 1778287910.742513 mapread 0x00f1a8 thru 0x016c86 (0x7adf bytes) 000611 1778287910.802866 trunc from 0x016c87 to 0x00ea25 (0x8261 bytes) 000612 1778287910.900626 mapread 0x00dacc thru 0x00ea25 (0xf5a bytes) 000613 1778287910.921174 write 0x0194f6 thru 0x0263af (0xceba bytes) 000614 1778287910.943041 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x263b0): Mirror 4: 0x5be53aa8 (stale) Mirror 5: 0x5be53aa8 (stale) Mirror 7: 0x5be53aa8 (stale) Mirror 12: 0x5be53aa8 (stale) Mirror 13: 0x5be53aa8 (stale) Mirror 14: 0x5be53aa8 (stale) Mirror 16: 0x5be53aa8 (stale) Mirror 18: 0x5be53aa8 (stale) Mirror 19: 0x5be53aa8 (stale) Mirror 20: 0x5be53aa8 (stale) Mirror 21: 0x5be53aa8 (stale) Mirror 24: 0xda1180c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x263b0): Mirror 4: 0x5be53aa8 (stale) Mirror 5: 0x5be53aa8 (stale) Mirror 7: 0x5be53aa8 (stale) Mirror 12: 0x5be53aa8 (stale) Mirror 13: 0x5be53aa8 (stale) Mirror 14: 0x5be53aa8 (stale) Mirror 16: 0x5be53aa8 (stale) Mirror 18: 0x5be53aa8 (stale) Mirror 19: 0x5be53aa8 (stale) Mirror 20: 0x5be53aa8 (stale) Mirror 21: 0x5be53aa8 (stale) Mirror 24: 0xda1180c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x263b0: skipped 000615 1778287912.052491 punch from 0x00afde to 0x011b50 (0x6b73 bytes) 000616 1778287912.057998 write 0x035d7d thru 0x03ae78 (0x50fc bytes) 000617 1778287912.067594 mapwrite 0x0202cf thru 0x02c6d1 (0xc403 bytes) 000618 1778287912.123878 write 0x006446 thru 0x0109b2 (0xa56d bytes) 000620 1778287912.139713 mapwrite 0x00d387 thru 0x016cf2 (0x996c bytes) 000622 1778287912.172105 trunc from 0x03ae79 to 0x0216e5 (0x19793 bytes) 000623 1778287912.190692 write 0x022b3e thru 0x03153c (0xe9ff bytes) 000624 1778287912.192614 mapwrite 0x021b43 thru 0x022f54 (0x1412 bytes) 000625 1778287912.213467 punch from 0x01b923 to 0x029c2b (0xe309 bytes) 000627 1778287912.223601 read 0x00176d thru 0x00bb89 (0xa41d bytes) 000628 1778287912.224501 trunc from 0x03153d to 0x02305e (0xe4de bytes) 000629 1778287912.249220 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2305f): Mirror 4: 0xd98690b8 (stale) Mirror 5: 0xd98690b8 (stale) Mirror 7: 0xd98690b8 (stale) Mirror 12: 0xd98690b8 (stale) Mirror 13: 0xd98690b8 (stale) Mirror 14: 0xd98690b8 (stale) Mirror 16: 0xd98690b8 (stale) Mirror 18: 0xd98690b8 (stale) Mirror 19: 0xd98690b8 (stale) Mirror 20: 0xd98690b8 (stale) Mirror 24: 0xd88ca14d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2305f): Mirror 4: 0xd98690b8 (stale) Mirror 5: 0xd98690b8 (stale) Mirror 7: 0xd98690b8 (stale) Mirror 12: 0xd98690b8 (stale) Mirror 13: 0xd98690b8 (stale) Mirror 14: 0xd98690b8 (stale) Mirror 16: 0xd98690b8 (stale) Mirror 18: 0xd98690b8 (stale) Mirror 19: 0xd98690b8 (stale) Mirror 20: 0xd98690b8 (stale) Mirror 24: 0xd88ca14d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2305f: skipped 000630 1778287913.193147 mapwrite 0x01dba4 thru 0x026473 (0x88d0 bytes) 000631 1778287913.279717 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x26474): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26474): Mirror 4: 0xbd5f581e (stale) Mirror 5: 0xbd5f581e (stale) Mirror 7: 0xbd5f581e (stale) Mirror 12: 0xbd5f581e (stale) Mirror 13: 0xbd5f581e (stale) Mirror 14: 0xbd5f581e (stale) Mirror 16: 0xbd5f581e (stale) Mirror 18: 0xbd5f581e (stale) Mirror 19: 0xbd5f581e (stale) Mirror 20: 0xbd5f581e (stale) Mirror 24: 0x1e58bbb5 (stale) CRC-32 checksum value for chunk [0, 0x26474): Mirror 1: 0x201147dc Mirror 25: 0x201147dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26474: skipped 000633 1778287914.973864 mapwrite 0x035da2 thru 0x036f8e (0x11ed bytes) 000634 1778287915.125528 read 0x024384 thru 0x028b2e (0x47ab bytes) 000635 1778287915.137188 write 0x01776b thru 0x023d2a (0xc5c0 bytes) 000637 1778287915.156714 trunc from 0x036f8f to 0x014411 (0x22b7d bytes) 000638 1778287915.205236 mapread 0x004240 thru 0x0047bf (0x580 bytes) 000639 1778287915.217289 punch from 0x009be1 to 0x00a902 (0xd22 bytes) 000640 1778287915.224295 punch from 0x003272 to 0x00e16f (0xaefe bytes) 000641 1778287915.237083 MIRROR_RESYNC 13 mirrors 000642 1778287918.328955 write 0x02adc5 thru 0x032ce2 (0x7f1e bytes) 000643 1778287918.421533 write 0x017349 thru 0x024c22 (0xd8da bytes) 000644 1778287918.446257 fallocate 0x006db2 thru 0x009e8b (0x30da bytes) 000645 1778287918.458516 read 0x0221f7 thru 0x022982 (0x78c bytes) 000646 1778287918.462614 mapwrite 0x016997 thru 0x01ef7a (0x85e4 bytes) 000647 1778287918.546334 write 0x013a4c thru 0x017d57 (0x430c bytes) 000648 1778287918.555805 trunc from 0x02ae65 to 0x032ce2 (0x7e7e bytes) 000650 1778287918.582519 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ce3): Mirror 4: 0x808c534d (stale) Mirror 5: 0x808c534d (stale) Mirror 7: 0x808c534d (stale) Mirror 12: 0x808c534d (stale) Mirror 13: 0x808c534d (stale) Mirror 14: 0x808c534d (stale) Mirror 16: 0x808c534d (stale) Mirror 18: 0x808c534d (stale) Mirror 19: 0x808c534d (stale) Mirror 20: 0x808c534d (stale) Mirror 24: 0x808c534d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ce3): Mirror 4: 0x808c534d (stale) Mirror 5: 0x808c534d (stale) Mirror 7: 0x808c534d (stale) Mirror 12: 0x808c534d (stale) Mirror 13: 0x808c534d (stale) Mirror 14: 0x808c534d (stale) Mirror 16: 0x808c534d (stale) Mirror 18: 0x808c534d (stale) Mirror 19: 0x808c534d (stale) Mirror 20: 0x808c534d (stale) Mirror 24: 0x808c534d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ce3: skipped 000652 1778287920.008794 mapread 0x01906c thru 0x02169b (0x8630 bytes) 000653 1778287920.039632 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ce3): Mirror 4: 0x808c534d (stale) Mirror 5: 0x808c534d (stale) Mirror 7: 0x808c534d (stale) Mirror 13: 0x808c534d (stale) Mirror 14: 0x808c534d (stale) Mirror 16: 0x808c534d (stale) Mirror 18: 0x808c534d (stale) Mirror 19: 0x808c534d (stale) Mirror 20: 0x808c534d (stale) Mirror 24: 0x808c534d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ce3): Mirror 4: 0x808c534d (stale) Mirror 5: 0x808c534d (stale) Mirror 7: 0x808c534d (stale) Mirror 13: 0x808c534d (stale) Mirror 14: 0x808c534d (stale) Mirror 16: 0x808c534d (stale) Mirror 18: 0x808c534d (stale) Mirror 19: 0x808c534d (stale) Mirror 20: 0x808c534d (stale) Mirror 24: 0x808c534d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ce3: skipped 000654 1778287921.241163 trunc from 0x032ce3 to 0x00c1b7 (0x26b2b bytes) 000655 1778287921.295202 fallocate 0x02dc05 thru 0x03059f (0x299b bytes) 000656 1778287921.307752 trunc from 0x007265 to 0x009058 (0x1df4 bytes) 000657 1778287921.315709 mapwrite 0x007c51 thru 0x01581e (0xdbce bytes) 000658 1778287921.432552 trunc from 0x01581f to 0x03c449 (0x26c2b bytes) 000659 1778287921.466142 trunc from 0x023209 to 0x026765 (0x355d bytes) 000660 1778287921.480129 mapread 0x00eb12 thru 0x016810 (0x7cff bytes) 000661 1778287921.503513 punch from 0x01bf27 to 0x02328c (0x7366 bytes) 000662 1778287921.519545 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c44a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c44a): Mirror 4: 0xa9182bb (stale) Mirror 5: 0xa9182bb (stale) Mirror 7: 0xa9182bb (stale) Mirror 13: 0xa9182bb (stale) Mirror 14: 0xa9182bb (stale) Mirror 16: 0xa9182bb (stale) Mirror 18: 0xa9182bb (stale) Mirror 19: 0xa9182bb (stale) Mirror 20: 0xa9182bb (stale) Mirror 24: 0xa9182bb (stale) CRC-32 checksum value for chunk [0, 0x3c44a): Mirror 1: 0x2cd491af Mirror 25: 0x2cd491af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c44a: skipped 000663 1778287923.328763 fallocate 0x03642e thru 0x03ffff (0x9bd2 bytes) 000664 1778287923.430792 mapread 0x034794 thru 0x03c130 (0x799d bytes) 000665 1778287923.513118 trunc from 0x02e3b1 to 0x02ea99 (0x6e9 bytes) 000666 1778287923.524228 trunc from 0x01ba2c to 0x022bd5 (0x71aa bytes) 000667 1778287923.546414 trunc from 0x0169d2 to 0x020ec9 (0xa4f8 bytes) 000668 1778287923.565590 mapwrite 0x021a8c thru 0x03191a (0xfe8f bytes) 000669 1778287923.666581 mapread 0x009642 thru 0x00a5d2 (0xf91 bytes) 000670 1778287923.673492 read 0x00db82 thru 0x01a09d (0xc51c bytes) 000671 1778287923.694280 punch from 0x001840 to 0x00a2fa (0x8abb bytes) 000672 1778287923.708299 trunc from 0x03b4f7 to 0x03c449 (0xf53 bytes) 000673 1778287923.717686 fallocate 0x02443c thru 0x02b3c5 (0x6f8a bytes) 000674 1778287923.726645 punch from 0x02807e to 0x03468c (0xc60f bytes) 000675 1778287923.735676 trunc from 0x025e4c to 0x02ce7e (0x7033 bytes) 000676 1778287923.746191 fallocate 0x039f96 thru 0x03fbbe (0x5c29 bytes) 000677 1778287923.758095 write 0x01355a thru 0x0220d7 (0xeb7e bytes) 000678 1778287923.784158 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbbf): Mirror 4: 0x4e9a63f (stale) Mirror 5: 0x4e9a63f (stale) Mirror 7: 0x4e9a63f (stale) Mirror 13: 0x4e9a63f (stale) Mirror 14: 0x4e9a63f (stale) Mirror 18: 0x4e9a63f (stale) Mirror 19: 0x4e9a63f (stale) Mirror 20: 0x4e9a63f (stale) Mirror 24: 0x4e9a63f (stale) Mirror 25: 0xf90d582e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbbf): Mirror 4: 0x4e9a63f (stale) Mirror 5: 0x4e9a63f (stale) Mirror 7: 0x4e9a63f (stale) Mirror 13: 0x4e9a63f (stale) Mirror 14: 0x4e9a63f (stale) Mirror 18: 0x4e9a63f (stale) Mirror 19: 0x4e9a63f (stale) Mirror 20: 0x4e9a63f (stale) Mirror 24: 0x4e9a63f (stale) Mirror 25: 0xf90d582e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbbf: skipped 000679 1778287924.599277 mapwrite 0x028f0a thru 0x02ed0b (0x5e02 bytes) 000680 1778287924.722712 mapwrite 0x001b55 thru 0x00d894 (0xbd40 bytes) 000681 1778287924.830064 mapread 0x01db96 thru 0x0257f1 (0x7c5c bytes) 000682 1778287924.855058 fallocate 0x013249 thru 0x01f5f5 (0xc3ad bytes) 000683 1778287924.870570 read 0x03eda6 thru 0x03fbbe (0xe19 bytes) 000684 1778287924.882450 trunc from 0x03fbbf to 0x03a80a (0x53b4 bytes) 000685 1778287924.919413 punch from 0x022dc1 to 0x024761 (0x19a1 bytes) 000686 1778287924.932108 trunc from 0x02ee97 to 0x03a184 (0xb2ee bytes) 000687 1778287924.939976 mapread 0x01b7e2 thru 0x02176d (0x5f8c bytes) 000688 1778287924.941176 trunc from 0x0032fb to 0x0042ef (0xff5 bytes) 000689 1778287924.960516 read 0x0097c5 thru 0x0123b9 (0x8bf5 bytes) 000691 1778287924.979674 mapread 0x016f76 thru 0x01c63c (0x56c7 bytes) 000692 1778287924.981559 trunc from 0x03a80b to 0x0234e7 (0x17323 bytes) 000693 1778287925.002006 trunc from 0x00bd34 to 0x013c6b (0x7f38 bytes) 000694 1778287925.009626 mapread 0x00767e thru 0x0150bb (0xda3e bytes) 000695 1778287925.091783 mapwrite 0x020232 thru 0x02f448 (0xf217 bytes) 000697 1778287925.354118 mapwrite 0x03f424 thru 0x03ffff (0xbdc bytes) 000698 1778287925.505702 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x58c4d7d0 (stale) Mirror 5: 0x58c4d7d0 (stale) Mirror 7: 0x58c4d7d0 (stale) Mirror 13: 0x58c4d7d0 (stale) Mirror 14: 0x58c4d7d0 (stale) Mirror 18: 0x58c4d7d0 (stale) Mirror 19: 0x58c4d7d0 (stale) Mirror 24: 0x58c4d7d0 (stale) Mirror 25: 0x6cc12619 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x58c4d7d0 (stale) Mirror 5: 0x58c4d7d0 (stale) Mirror 7: 0x58c4d7d0 (stale) Mirror 13: 0x58c4d7d0 (stale) Mirror 14: 0x58c4d7d0 (stale) Mirror 18: 0x58c4d7d0 (stale) Mirror 19: 0x58c4d7d0 (stale) Mirror 24: 0x58c4d7d0 (stale) Mirror 25: 0x6cc12619 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000699 1778287926.774887 mapwrite 0x0164a6 thru 0x02638e (0xfee9 bytes) 000700 1778287926.844644 write 0x0343ea thru 0x039cbc (0x58d3 bytes) 000701 1778287926.867299 write 0x00f7b3 thru 0x01e240 (0xea8e bytes) 000702 1778287926.880220 trunc from 0x00b940 to 0x00cbf3 (0x12b4 bytes) 000703 1778287926.892074 MIRROR_RESYNC 10 mirrors 000704 1778287931.225716 fallocate 0x019fe2 thru 0x020d42 (0x6d61 bytes) 000705 1778287931.268199 fallocate 0x005777 thru 0x00bfd5 (0x685f bytes) 000706 1778287931.276790 mapwrite 0x030200 thru 0x03f752 (0xf553 bytes) 000707 1778287931.396268 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7be44c7f (stale) Mirror 5: 0x7be44c7f (stale) Mirror 7: 0x7be44c7f (stale) Mirror 13: 0x7be44c7f (stale) Mirror 14: 0x7be44c7f (stale) Mirror 18: 0x7be44c7f (stale) Mirror 19: 0x7be44c7f (stale) Mirror 24: 0x7be44c7f (stale) Mirror 25: 0x7be44c7f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9265dad5 Mirror 26: 0x9265dad5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1778287932.890273 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7be44c7f (stale) Mirror 5: 0x7be44c7f (stale) Mirror 7: 0x7be44c7f (stale) Mirror 13: 0x7be44c7f (stale) Mirror 14: 0x7be44c7f (stale) Mirror 18: 0x7be44c7f (stale) Mirror 19: 0x7be44c7f (stale) Mirror 24: 0x7be44c7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7be44c7f (stale) Mirror 5: 0x7be44c7f (stale) Mirror 7: 0x7be44c7f (stale) Mirror 13: 0x7be44c7f (stale) Mirror 14: 0x7be44c7f (stale) Mirror 18: 0x7be44c7f (stale) Mirror 19: 0x7be44c7f (stale) Mirror 24: 0x7be44c7f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9265dad5 Mirror 26: 0x9265dad5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000709 1778287934.119495 mapread 0x012e85 thru 0x0148fe (0x1a7a bytes) 000710 1778287934.163970 mapread 0x0385fc thru 0x03ffff (0x7a04 bytes) 000712 1778287934.225974 trunc from 0x006726 to 0x00b998 (0x5273 bytes) 000713 1778287934.280962 trunc from 0x040000 to 0x03a1fe (0x5e01 bytes) 000714 1778287934.345189 read 0x034a49 thru 0x03a1fe (0x57b6 bytes) 000715 1778287934.361167 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a1ff): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1ff): Mirror 4: 0x6a1c7322 (stale) Mirror 5: 0x6a1c7322 (stale) Mirror 7: 0x6a1c7322 (stale) Mirror 13: 0x6a1c7322 (stale) Mirror 14: 0x6a1c7322 (stale) Mirror 18: 0x6a1c7322 (stale) Mirror 19: 0x6a1c7322 (stale) Mirror 24: 0x6a1c7322 (stale) Mirror 26: 0xef0f30c2 (stale) CRC-32 checksum value for chunk [0, 0x3a1ff): Mirror 1: 0x54b9a558 Mirror 27: 0x54b9a558 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1ff: skipped 000716 1778287935.808843 trunc from 0x03a1ff to 0x031ea2 (0x835c bytes) 000717 1778287935.925974 fallocate 0x01f178 thru 0x021de0 (0x2c69 bytes) 000718 1778287935.946858 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ea3): Mirror 4: 0x406cdcca (stale) Mirror 5: 0x406cdcca (stale) Mirror 7: 0x406cdcca (stale) Mirror 13: 0x406cdcca (stale) Mirror 14: 0x406cdcca (stale) Mirror 18: 0x406cdcca (stale) Mirror 19: 0x406cdcca (stale) Mirror 24: 0x406cdcca (stale) Mirror 26: 0x618c5582 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ea3): Mirror 4: 0x406cdcca (stale) Mirror 5: 0x406cdcca (stale) Mirror 7: 0x406cdcca (stale) Mirror 13: 0x406cdcca (stale) Mirror 14: 0x406cdcca (stale) Mirror 18: 0x406cdcca (stale) Mirror 19: 0x406cdcca (stale) Mirror 24: 0x406cdcca (stale) Mirror 26: 0x618c5582 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ea3: skipped 000719 1778287936.821902 mapread 0x031872 thru 0x031ea2 (0x631 bytes) 000720 1778287936.836244 write 0x0291a1 thru 0x034aec (0xb94c bytes) 000721 1778287936.874658 punch from 0x006eb7 to 0x016d1f (0xfe69 bytes) 000722 1778287936.890959 trunc from 0x00185e to 0x00213f (0x8e2 bytes) 000724 1778287936.918183 fallocate 0x03dcee thru 0x03ffff (0x2312 bytes) 000725 1778287936.934633 mapwrite 0x00087f thru 0x00c131 (0xb8b3 bytes) 000726 1778287936.988984 trunc from 0x040000 to 0x0198ca (0x26735 bytes) 000727 1778287937.018972 mapread 0x008d15 thru 0x00b7b3 (0x2a9f bytes) 000729 1778287937.023868 trunc from 0x00b51a to 0x018ab5 (0xd59c bytes) 000730 1778287937.038070 fallocate 0x016043 thru 0x01c15f (0x611d bytes) 000731 1778287937.046315 read 0x0025c0 thru 0x00b10b (0x8b4c bytes) 000732 1778287937.055825 write 0x03297a thru 0x039a09 (0x7090 bytes) 000733 1778287937.065672 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39a0a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a0a): Mirror 4: 0x240a1963 (stale) Mirror 5: 0x240a1963 (stale) Mirror 7: 0x240a1963 (stale) Mirror 13: 0x240a1963 (stale) Mirror 14: 0x240a1963 (stale) Mirror 18: 0x240a1963 (stale) Mirror 19: 0x240a1963 (stale) Mirror 24: 0x240a1963 (stale) Mirror 26: 0xb5c56cc1 (stale) CRC-32 checksum value for chunk [0, 0x39a0a): Mirror 1: 0xedb034dd Mirror 27: 0xedb034dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a0a: skipped 000734 1778287938.548827 mapread 0x019a8a thru 0x02325a (0x97d1 bytes) 000735 1778287938.589891 fallocate 0x0022c3 thru 0x002707 (0x445 bytes) 000736 1778287938.630084 mapwrite 0x020701 thru 0x02da50 (0xd350 bytes) 000737 1778287938.669541 trunc from 0x033f13 to 0x039a09 (0x5af7 bytes) 000738 1778287938.676870 trunc from 0x01dbf6 to 0x01fbfb (0x2006 bytes) 000739 1778287938.682040 trunc from 0x029a7e to 0x030f2c (0x74af bytes) 000740 1778287938.688251 trunc from 0x01ec8d to 0x02ea93 (0xfe07 bytes) 000741 1778287938.693192 trunc from 0x02ebac to 0x035df5 (0x724a bytes) 000743 1778287938.698752 write 0x0216e2 thru 0x0256d2 (0x3ff1 bytes) 000744 1778287938.718676 MIRROR_RESYNC 11 mirrors 000745 1778287941.884644 mapread 0x022e12 thru 0x02c1d5 (0x93c4 bytes) 000746 1778287941.912014 trunc from 0x0296bf to 0x03957f (0xfec1 bytes) 000747 1778287941.958149 write 0x02577f thru 0x027566 (0x1de8 bytes) 000748 1778287941.969324 mapwrite 0x007a7a thru 0x012dcd (0xb354 bytes) 000749 1778287942.027753 trunc from 0x039a0a to 0x026bab (0x12e5e bytes) 000750 1778287942.067180 trunc from 0x026bac to 0x005bca (0x20fe1 bytes) 000751 1778287942.096080 mapwrite 0x023d9a thru 0x02f61a (0xb881 bytes) 000752 1778287942.148334 punch from 0x002c93 to 0x012b6e (0xfedc bytes) 000753 1778287942.152657 read 0x02e571 thru 0x02f61a (0x10aa bytes) 000754 1778287942.153371 mapwrite 0x00d045 thru 0x01bd4c (0xed08 bytes) 000755 1778287942.196795 mapwrite 0x02d3aa thru 0x02fb22 (0x2779 bytes) 000756 1778287942.223204 mapread 0x013985 thru 0x0157bd (0x1e39 bytes) 000757 1778287942.224689 mapwrite 0x002714 thru 0x00a3e4 (0x7cd1 bytes) 000758 1778287942.241772 trunc from 0x02fb23 to 0x03b981 (0xbe5f bytes) 000759 1778287942.250838 write 0x027477 thru 0x02b57f (0x4109 bytes) 000760 1778287942.252085 punch from 0x038fbb to 0x0391d7 (0x21d bytes) 000761 1778287942.255659 trunc from 0x03b982 to 0x03226b (0x9716 bytes) 000762 1778287942.268909 mapwrite 0x016f7b thru 0x02602a (0xf0b0 bytes) 000763 1778287942.290658 mapwrite 0x002123 thru 0x008080 (0x5f5e bytes) 000764 1778287942.344092 trunc from 0x03226c to 0x03df4e (0xbce3 bytes) 000765 1778287942.371079 mapread 0x02be6e thru 0x03668c (0xa81f bytes) 000766 1778287942.380564 write 0x009040 thru 0x014010 (0xafd1 bytes) 000767 1778287942.382211 mapwrite 0x00c3c6 thru 0x00deb4 (0x1aef bytes) 000768 1778287942.396795 trunc from 0x03df4f to 0x02e025 (0xff29 bytes) 000769 1778287942.411590 read 0x0285fd thru 0x02e025 (0x5a29 bytes) 000770 1778287942.415202 MIRROR_RESYNC 11 mirrors 000771 1778287943.423744 mapread 0x008b15 thru 0x00bbbf (0x30ab bytes) 000772 1778287943.457820 mapread 0x024116 thru 0x02a24c (0x6137 bytes) 000773 1778287943.485991 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) 000775 1778287944.157627 mapread 0x021fac thru 0x024cd1 (0x2d26 bytes) 000776 1778287944.222405 punch from 0x02911c to 0x02e025 (0x4f0a bytes) 000778 1778287944.366197 read 0x00087a thru 0x0075cd (0x6d54 bytes) 000779 1778287944.386045 punch from 0x00d9cb to 0x0124f7 (0x4b2d bytes) 000780 1778287944.407950 write 0x00e698 thru 0x01b80d (0xd176 bytes) 000781 1778287944.464280 mapwrite 0x02565b thru 0x02a989 (0x532f bytes) 000782 1778287944.516888 punch from 0x00710d to 0x00ad1e (0x3c12 bytes) 000783 1778287944.526260 fallocate 0x024ffd thru 0x02ec38 (0x9c3c bytes) 000784 1778287944.532914 write 0x03513f thru 0x03ffff (0xaec1 bytes) 000785 1778287944.538997 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc670be7f (stale) Mirror 5: 0xc670be7f (stale) Mirror 7: 0xc670be7f (stale) Mirror 14: 0xc670be7f (stale) Mirror 18: 0xc670be7f (stale) Mirror 19: 0xc670be7f (stale) Mirror 24: 0xc670be7f (stale) Mirror 27: 0xc670be7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc670be7f (stale) Mirror 5: 0xc670be7f (stale) Mirror 7: 0xc670be7f (stale) Mirror 14: 0xc670be7f (stale) Mirror 18: 0xc670be7f (stale) Mirror 19: 0xc670be7f (stale) Mirror 24: 0xc670be7f (stale) Mirror 27: 0xc670be7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1778287945.413075 mapread 0x015e90 thru 0x01af59 (0x50ca bytes) 000787 1778287945.447607 mapread 0x03a3e0 thru 0x03bb97 (0x17b8 bytes) 000788 1778287945.462148 fallocate 0x02b0d2 thru 0x030a73 (0x59a2 bytes) 000789 1778287945.469262 mapwrite 0x00c811 thru 0x012e78 (0x6668 bytes) 000790 1778287945.499845 mapwrite 0x007478 thru 0x00b6be (0x4247 bytes) 000791 1778287945.527413 trunc from 0x002270 to 0x007bae (0x593f bytes) 000792 1778287945.538635 fallocate 0x0152e3 thru 0x01928b (0x3fa9 bytes) 000793 1778287945.561971 punch from 0x035895 to 0x03b6c0 (0x5e2c bytes) 000794 1778287945.585110 punch from 0x03ab8d to 0x03ffff (0x5473 bytes) 000795 1778287945.600755 write 0x00b4a8 thru 0x013a69 (0x85c2 bytes) 000796 1778287945.614211 mapwrite 0x01e6f0 thru 0x023744 (0x5055 bytes) 000797 1778287945.664501 write 0x00f7a5 thru 0x018409 (0x8c65 bytes) 000798 1778287945.666109 write 0x023ffa thru 0x0319d6 (0xd9dd bytes) 000799 1778287945.678627 fallocate 0x034221 thru 0x03ffff (0xbddf bytes) 000800 1778287945.687914 fallocate 0x003769 thru 0x012190 (0xea28 bytes) 000801 1778287945.693290 mapread 0x002b51 thru 0x009c9c (0x714c bytes) 000802 1778287945.703858 punch from 0x0324ed to 0x03a984 (0x8498 bytes) 000803 1778287945.709276 mapread 0x010d33 thru 0x01ed89 (0xe057 bytes) 000805 1778287945.716132 MIRROR_RESYNC 9 mirrors 000806 1778287949.489215 mapread 0x031e2a thru 0x03b255 (0x942c bytes) 000807 1778287949.536063 read 0x0022dc thru 0x00e7b9 (0xc4de bytes) 000809 1778287949.565981 mapwrite 0x01b470 thru 0x01d15a (0x1ceb bytes) 000811 1778287949.874253 write 0x032cbb thru 0x03624a (0x3590 bytes) 000812 1778287949.899919 trunc from 0x040000 to 0x02971e (0x168e1 bytes) 000813 1778287949.963912 read 0x014d75 thru 0x01d81c (0x8aa8 bytes) 000814 1778287949.964769 fallocate 0x02510e thru 0x02d3fb (0x82ee bytes) 000815 1778287949.977980 write 0x01ed2e thru 0x02a2dc (0xb5af bytes) 000816 1778287950.036749 MIRROR_RESYNC 9 mirrors 000817 1778287953.397121 trunc from 0x02d3fc to 0x00266f (0x2ad8c bytes) 000818 1778287953.460936 mapwrite 0x028608 thru 0x02ec47 (0x6640 bytes) 000819 1778287953.554767 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ec48): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec48): Mirror 4: 0xb629c1ed (stale) Mirror 5: 0xb629c1ed (stale) Mirror 7: 0xb629c1ed (stale) Mirror 14: 0xb629c1ed (stale) Mirror 18: 0xb629c1ed (stale) Mirror 19: 0xb629c1ed (stale) Mirror 24: 0xb629c1ed (stale) Mirror 27: 0xb629c1ed (stale) CRC-32 checksum value for chunk [0, 0x2ec48): Mirror 1: 0xc7ae59e1 Mirror 28: 0xc7ae59e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec48: skipped 000820 1778287955.867940 mapread 0x02892d thru 0x02b4f6 (0x2bca bytes) 000821 1778287955.938296 punch from 0x01b21b to 0x02280d (0x75f3 bytes) 000822 1778287956.041910 write 0x01ecda thru 0x02d694 (0xe9bb bytes) 000823 1778287956.074754 read 0x01bb1a thru 0x01c4e3 (0x9ca bytes) 000825 1778287956.076079 write 0x0332bd thru 0x034f0e (0x1c52 bytes) 000826 1778287956.107388 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x34f0f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f0f): Mirror 4: 0x1c62605d (stale) Mirror 5: 0x1c62605d (stale) Mirror 7: 0x1c62605d (stale) Mirror 14: 0x1c62605d (stale) Mirror 18: 0x1c62605d (stale) Mirror 19: 0x1c62605d (stale) Mirror 24: 0x1c62605d (stale) Mirror 27: 0x1c62605d (stale) Mirror 28: 0x46954b01 (stale) CRC-32 checksum value for chunk [0, 0x34f0f): Mirror 1: 0x38d8ad19 Mirror 29: 0x38d8ad19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f0f: skipped 000827 1778287958.399515 read 0x00c2a3 thru 0x00e26c (0x1fca bytes) 000828 1778287958.472817 read 0x033a90 thru 0x034f0e (0x147f bytes) 000829 1778287958.515545 read 0x02185c thru 0x02bf4c (0xa6f1 bytes) 000830 1778287958.544596 MIRROR_RESYNC 11 mirrors 000831 1778287963.252641 mapwrite 0x009ba1 thru 0x015c32 (0xc092 bytes) 000832 1778287963.512880 read 0x0127fa thru 0x01cca3 (0xa4aa bytes) 000833 1778287963.524845 punch from 0x016784 to 0x01aad3 (0x4350 bytes) 000834 1778287963.541940 fallocate 0x006052 thru 0x014d7c (0xed2b bytes) 000835 1778287963.556663 write 0x030351 thru 0x033ccb (0x397b bytes) 000836 1778287963.585752 read 0x0225e3 thru 0x026605 (0x4023 bytes) 000837 1778287963.627801 mapread 0x0132d6 thru 0x020e51 (0xdb7c bytes) 000838 1778287963.629735 trunc from 0x034f0f to 0x026aea (0xe424 bytes) 000840 1778287963.705060 write 0x0162d2 thru 0x018947 (0x2676 bytes) 000841 1778287963.717578 trunc from 0x01e308 to 0x026aea (0x87e3 bytes) 000842 1778287963.737251 write 0x029858 thru 0x02ab30 (0x12d9 bytes) 000844 1778287963.758509 write 0x01e6fa thru 0x029f58 (0xb85f bytes) 000845 1778287963.772205 fallocate 0x035db8 thru 0x03ffff (0xa248 bytes) 000846 1778287963.812550 trunc from 0x022ddf to 0x02d7a2 (0xa9c4 bytes) 000847 1778287963.856713 fallocate 0x02a072 thru 0x02ae0c (0xd9b bytes) 000848 1778287963.868414 fallocate 0x00b953 thru 0x0121ea (0x6898 bytes) 000849 1778287963.882425 punch from 0x000e7e to 0x00cc38 (0xbdbb bytes) 000850 1778287963.894261 fallocate 0x02996b thru 0x0324d3 (0x8b69 bytes) 000851 1778287963.920430 fallocate 0x038fa8 thru 0x03ffff (0x7058 bytes) 000852 1778287963.929489 mapwrite 0x014c95 thru 0x01dd26 (0x9092 bytes) 000853 1778287963.989898 trunc from 0x007afc to 0x00f8e1 (0x7de6 bytes) 000854 1778287964.016946 write 0x0294d2 thru 0x02b1a3 (0x1cd2 bytes) 000856 1778287964.075477 trunc from 0x01581d to 0x019322 (0x3b06 bytes) 000857 1778287964.096321 trunc from 0x03a99d to 0x03ffff (0x5663 bytes) 000858 1778287964.135076 write 0x001ef9 thru 0x007af3 (0x5bfb bytes) 000859 1778287964.168486 trunc from 0x040000 to 0x02fb94 (0x1046b bytes) 000860 1778287964.246645 MIRROR_RESYNC 11 mirrors 000861 1778287970.685681 punch from 0x00bc85 to 0x00fed0 (0x424c bytes) 000862 1778287970.820997 trunc from 0x02fb95 to 0x025d77 (0x9e1d bytes) 000864 1778287970.886946 mapread 0x002ca2 thru 0x0032b3 (0x612 bytes) 000865 1778287970.919766 trunc from 0x01e2c1 to 0x01f0d4 (0xe14 bytes) 000866 1778287970.928765 mapread 0x021c39 thru 0x025d77 (0x413f bytes) 000867 1778287970.960640 mapwrite 0x013234 thru 0x01b065 (0x7e32 bytes) 000868 1778287971.019168 trunc from 0x025d78 to 0x00ee2f (0x16f48 bytes) 000869 1778287971.046555 trunc from 0x00ee30 to 0x01a980 (0xbb51 bytes) 000870 1778287971.154824 mapread 0x01004b thru 0x0142e9 (0x429f bytes) 000871 1778287971.192209 fallocate 0x01b31d thru 0x02834e (0xd032 bytes) 000872 1778287971.214272 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1a981): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a981): Mirror 4: 0x1eb5f602 (stale) Mirror 5: 0x1eb5f602 (stale) Mirror 7: 0x1eb5f602 (stale) Mirror 14: 0x1eb5f602 (stale) Mirror 18: 0x1eb5f602 (stale) Mirror 19: 0x1eb5f602 (stale) Mirror 24: 0x1eb5f602 (stale) Mirror 27: 0x1eb5f602 (stale) Mirror 28: 0x1eb5f602 (stale) Mirror 29: 0x1eb5f602 (stale) CRC-32 checksum value for chunk [0, 0x1a981): Mirror 1: 0x6e0bcb70 Mirror 30: 0x6e0bcb70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a981: skipped 000873 1778287973.156637 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1a981): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a981): Mirror 4: 0x1eb5f602 (stale) Mirror 5: 0x1eb5f602 (stale) Mirror 7: 0x1eb5f602 (stale) Mirror 14: 0x1eb5f602 (stale) Mirror 18: 0x1eb5f602 (stale) Mirror 19: 0x1eb5f602 (stale) Mirror 24: 0x1eb5f602 (stale) Mirror 27: 0x1eb5f602 (stale) Mirror 28: 0x1eb5f602 (stale) Mirror 29: 0x1eb5f602 (stale) CRC-32 checksum value for chunk [0, 0x1a981): Mirror 1: 0x6e0bcb70 Mirror 30: 0x6e0bcb70 Mirror 31: 0x6e0bcb70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a981: skipped 000874 1778287975.406271 punch from 0x00d4f8 to 0x010c29 (0x3732 bytes) 000875 1778287975.550199 mapread 0x0113a2 thru 0x01a980 (0x95df bytes) 000876 1778287975.589108 trunc from 0x00af00 to 0x01a980 (0xfa81 bytes) 000877 1778287975.605686 read 0x00a897 thru 0x00c638 (0x1da2 bytes) 000878 1778287975.617649 trunc from 0x01a981 to 0x011be4 (0x8d9c bytes) 000879 1778287975.726069 write 0x029084 thru 0x038981 (0xf8fe bytes) 000880 1778287975.730089 write 0x015839 thru 0x01efc0 (0x9788 bytes) 000881 1778287975.756297 punch from 0x020e6e to 0x020eb5 (0x48 bytes) 000882 1778287975.769212 mapwrite 0x01696d thru 0x01d1df (0x6873 bytes) 000883 1778287975.858164 trunc from 0x0188ae to 0x0250b0 (0xc803 bytes) 000884 1778287975.888522 trunc from 0x031343 to 0x038981 (0x763f bytes) 000885 1778287975.907197 mapread 0x036d68 thru 0x03730b (0x5a4 bytes) 000886 1778287975.958859 mapread 0x023dd8 thru 0x0289cd (0x4bf6 bytes) 000887 1778287975.983096 trunc from 0x038982 to 0x01675a (0x22227 bytes) 000888 1778287976.062186 punch from 0x00c646 to 0x0136d1 (0x708c bytes) 000889 1778287976.076726 write 0x023d88 thru 0x02b905 (0x7b7e bytes) 000891 1778287976.078999 mapwrite 0x03c24f thru 0x03ffff (0x3db1 bytes) 000892 1778287976.196582 trunc from 0x040000 to 0x0210f2 (0x1ef0d bytes) 000893 1778287976.258290 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x210f3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210f3): Mirror 4: 0x159bbd08 (stale) Mirror 5: 0x159bbd08 (stale) Mirror 7: 0x159bbd08 (stale) Mirror 14: 0x159bbd08 (stale) Mirror 18: 0x159bbd08 (stale) Mirror 19: 0x159bbd08 (stale) Mirror 24: 0x159bbd08 (stale) Mirror 27: 0x159bbd08 (stale) Mirror 28: 0x159bbd08 (stale) Mirror 29: 0x159bbd08 (stale) Mirror 30: 0xb04594a4 (stale) Mirror 31: 0xb04594a4 (stale) CRC-32 checksum value for chunk [0, 0x210f3): Mirror 1: 0x2339441a Mirror 32: 0x2339441a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210f3: skipped 000895 1778287978.580426 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x210f3): Mirror 4: 0x159bbd08 (stale) Mirror 5: 0x159bbd08 (stale) Mirror 14: 0x159bbd08 (stale) Mirror 18: 0x159bbd08 (stale) Mirror 19: 0x159bbd08 (stale) Mirror 24: 0x159bbd08 (stale) Mirror 27: 0x159bbd08 (stale) Mirror 28: 0x159bbd08 (stale) Mirror 29: 0x159bbd08 (stale) Mirror 30: 0xb04594a4 (stale) Mirror 31: 0xb04594a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210f3): Mirror 4: 0x159bbd08 (stale) Mirror 5: 0x159bbd08 (stale) Mirror 14: 0x159bbd08 (stale) Mirror 18: 0x159bbd08 (stale) Mirror 19: 0x159bbd08 (stale) Mirror 24: 0x159bbd08 (stale) Mirror 27: 0x159bbd08 (stale) Mirror 28: 0x159bbd08 (stale) Mirror 29: 0x159bbd08 (stale) Mirror 30: 0xb04594a4 (stale) Mirror 31: 0xb04594a4 (stale) CRC-32 checksum value for chunk [0, 0x210f3): Mirror 1: 0x2339441a Mirror 32: 0x2339441a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210f3: skipped 000896 1778287980.526223 trunc from 0x0210f3 to 0x032a2e (0x1193c bytes) 000897 1778287980.605043 punch from 0x01dd50 to 0x021c77 (0x3f28 bytes) 000898 1778287980.612994 punch from 0x006770 to 0x01563b (0xeecc bytes) 000899 1778287980.621024 mapwrite 0x02954d thru 0x02f559 (0x600d bytes) 000900 1778287980.690984 mapwrite 0x020f9c thru 0x026559 (0x55be bytes) 000901 1778287980.733670 write 0x00c2da thru 0x00c404 (0x12b bytes) 000902 1778287980.757014 read 0x027b9d thru 0x02f77a (0x7bde bytes) 000903 1778287980.757830 fallocate 0x033e6c thru 0x036053 (0x21e8 bytes) 000904 1778287980.766028 read 0x000e2c thru 0x00949f (0x8674 bytes) 000906 1778287980.786961 write 0x02d653 thru 0x02f309 (0x1cb7 bytes) 000907 1778287980.796904 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x36054): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36054): Mirror 4: 0x828dd67a (stale) Mirror 5: 0x828dd67a (stale) Mirror 14: 0x828dd67a (stale) Mirror 18: 0x828dd67a (stale) Mirror 19: 0x828dd67a (stale) Mirror 24: 0x828dd67a (stale) Mirror 27: 0x828dd67a (stale) Mirror 28: 0x828dd67a (stale) Mirror 29: 0x828dd67a (stale) Mirror 30: 0xd9c7b321 (stale) Mirror 31: 0xd9c7b321 (stale) Mirror 32: 0xffb793de (stale) CRC-32 checksum value for chunk [0, 0x36054): Mirror 1: 0x6df07452 Mirror 33: 0x6df07452 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36054: skipped 000908 1778287982.676602 mapwrite 0x01b1bb thru 0x0250df (0x9f25 bytes) 000909 1778287982.830284 mapread 0x031253 thru 0x036053 (0x4e01 bytes) 000910 1778287982.838793 mapwrite 0x028222 thru 0x03434b (0xc12a bytes) 000912 1778287982.879496 fallocate 0x012ece thru 0x018bd1 (0x5d04 bytes) 000913 1778287982.890595 trunc from 0x0007e8 to 0x0056f2 (0x4f0b bytes) 000914 1778287982.897086 mapread 0x0147d3 thru 0x01f007 (0xa835 bytes) 000915 1778287982.910073 read 0x0185e8 thru 0x02056c (0x7f85 bytes) 000916 1778287982.910858 fallocate 0x0183a4 thru 0x0242f6 (0xbf53 bytes) 000917 1778287982.920232 trunc from 0x036054 to 0x038ef7 (0x2ea4 bytes) 000918 1778287982.970329 fallocate 0x036944 thru 0x03ffff (0x96bc bytes) 000920 1778287982.977255 trunc from 0x038ef8 to 0x022efd (0x15ffa bytes) 000921 1778287983.002263 fallocate 0x01a35c thru 0x023617 (0x92bc bytes) 000922 1778287983.012233 write 0x024594 thru 0x02c96c (0x83d9 bytes) 000923 1778287983.016233 punch from 0x02b5e9 to 0x02c96c (0x1384 bytes) 000924 1778287983.046704 read 0x007b7c thru 0x01402f (0xc4b4 bytes) 000925 1778287983.068335 trunc from 0x02c96d to 0x0318fa (0x4f8e bytes) 000926 1778287983.119281 trunc from 0x0318fb to 0x00e322 (0x235d8 bytes) 000927 1778287983.154812 read 0x007967 thru 0x00c7b6 (0x4e50 bytes) 000928 1778287983.155663 write 0x03d103 thru 0x03ffff (0x2efd bytes) 000929 1778287983.159671 trunc from 0x040000 to 0x01d685 (0x2297a bytes) 000930 1778287983.185584 punch from 0x00181d to 0x00e435 (0xcc19 bytes) 000932 1778287983.200028 MIRROR_RESYNC 14 mirrors 000933 1778287986.173829 write 0x00e85f thru 0x017bb1 (0x9353 bytes) 000934 1778287986.317051 trunc from 0x0124ab to 0x0195c0 (0x7116 bytes) 000935 1778287986.371413 mapread 0x001704 thru 0x0100c4 (0xe9c1 bytes) 000936 1778287986.425585 read 0x004bce thru 0x00e95a (0x9d8d bytes) 000937 1778287986.431654 mapread 0x002e38 thru 0x00bf5e (0x9127 bytes) 000938 1778287986.442372 fallocate 0x014662 thru 0x021e63 (0xd802 bytes) 000939 1778287986.456279 trunc from 0x00cf05 to 0x01bf91 (0xf08d bytes) 000940 1778287986.481166 trunc from 0x016f56 to 0x01d685 (0x6730 bytes) 000942 1778287986.507477 read 0x00b90a thru 0x011a95 (0x618c bytes) 000943 1778287986.528222 trunc from 0x000173 to 0x003714 (0x35a2 bytes) 000944 1778287986.559218 mapwrite 0x00f175 thru 0x01338d (0x4219 bytes) 000945 1778287986.596289 trunc from 0x001b02 to 0x00ba4c (0x9f4b bytes) 000946 1778287986.621773 write 0x006d3a thru 0x00a066 (0x332d bytes) 000947 1778287986.654092 write 0x01fe5c thru 0x02b26e (0xb413 bytes) 000948 1778287986.656119 punch from 0x024b6a to 0x02b26e (0x6705 bytes) 000949 1778287986.689194 read 0x00170e thru 0x00dd1b (0xc60e bytes) 000950 1778287986.703238 mapread 0x02699b thru 0x02b26e (0x48d4 bytes) 000951 1778287986.704540 trunc from 0x00c719 to 0x0173a9 (0xac91 bytes) 000952 1778287986.721015 fallocate 0x033b08 thru 0x0347bd (0xcb6 bytes) 000953 1778287986.734118 punch from 0x004fb3 to 0x00f7ee (0xa83c bytes) 000955 1778287986.752274 mapread 0x021a1a thru 0x022711 (0xcf8 bytes) 000956 1778287986.758219 mapwrite 0x0016d7 thru 0x002cff (0x1629 bytes) 000957 1778287986.774098 mapwrite 0x00acd7 thru 0x00cc2f (0x1f59 bytes) 000958 1778287986.840141 trunc from 0x02b26f to 0x00c8fd (0x1e971 bytes) 000959 1778287986.883201 read 0x005e38 thru 0x00c8fd (0x6ac6 bytes) 000960 1778287986.901185 trunc from 0x00c8fe to 0x005c6f (0x6c8e bytes) 000961 1778287986.921866 read 0x003479 thru 0x005c6f (0x27f7 bytes) 000963 1778287986.923178 trunc from 0x002cc2 to 0x00336e (0x6ad bytes) 000964 1778287986.954078 fallocate 0x034343 thru 0x037b2d (0x37eb bytes) 000965 1778287986.972195 read 0x0016fc thru 0x005c6f (0x4574 bytes) 000966 1778287986.995830 write 0x03eaed thru 0x03ffff (0x1513 bytes) 000968 1778287987.001832 write 0x01b372 thru 0x028627 (0xd2b6 bytes) 000969 1778287987.018423 mapwrite 0x006010 thru 0x00ae8f (0x4e80 bytes) 000970 1778287987.056633 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xe20eea22 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xe20eea22 (stale) Mirror 32: 0xe20eea22 (stale) Mirror 33: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a3c1e85 Mirror 34: 0x9a3c1e85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000971 1778287988.895997 trunc from 0x040000 to 0x00fa13 (0x305ec bytes) 000972 1778287988.981415 mapwrite 0x02a474 thru 0x02a9ec (0x579 bytes) 000973 1778287989.016021 MIRROR_RESYNC 15 mirrors 000974 1778287993.054453 mapwrite 0x00467d thru 0x00b152 (0x6ad6 bytes) 000976 1778287993.245160 MIRROR_RESYNC 15 mirrors 000977 1778287996.176936 fallocate 0x00ef81 thru 0x014d39 (0x5db9 bytes) 000978 1778287996.227482 punch from 0x013b84 to 0x01aa65 (0x6ee2 bytes) 000979 1778287996.234149 punch from 0x009fe2 to 0x00a489 (0x4a8 bytes) 000980 1778287996.239714 trunc from 0x02a9ed to 0x0112cb (0x19721 bytes) 000981 1778287996.260502 trunc from 0x006935 to 0x0112cb (0xa997 bytes) 000982 1778287996.273552 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x184:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x112cc): Mirror 4: 0x41e77b26 (stale) Mirror 5: 0x41e77b26 (stale) Mirror 14: 0x41e77b26 (stale) Mirror 18: 0x41e77b26 (stale) Mirror 19: 0x41e77b26 (stale) Mirror 24: 0x41e77b26 (stale) Mirror 27: 0x41e77b26 (stale) Mirror 28: 0x41e77b26 (stale) Mirror 30: 0x41e77b26 (stale) Mirror 31: 0x41e77b26 (stale) Mirror 32: 0x41e77b26 (stale) Mirror 33: 0x41e77b26 (stale) Mirror 34: 0x41e77b26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x112cc): Mirror 4: 0x41e77b26 (stale) Mirror 5: 0x41e77b26 (stale) Mirror 14: 0x41e77b26 (stale) Mirror 18: 0x41e77b26 (stale) Mirror 19: 0x41e77b26 (stale) Mirror 24: 0x41e77b26 (stale) Mirror 27: 0x41e77b26 (stale) Mirror 28: 0x41e77b26 (stale) Mirror 30: 0x41e77b26 (stale) Mirror 31: 0x41e77b26 (stale) Mirror 32: 0x41e77b26 (stale) Mirror 33: 0x41e77b26 (stale) Mirror 34: 0x41e77b26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x112cc: skipped 000983 1778287996.919013 MIRROR_RESYNC 14 mirrors 000984 1778287999.353164 MIRROR_RESYNC 14 mirrors 000985 1778288000.158739 read 0x00933d thru 0x0112cb (0x7f8f bytes) 000986 1778288000.175796 punch from 0x000a66 to 0x001870 (0xe0b bytes) 000987 1778288000.228734 mapread 0x00f323 thru 0x0112cb (0x1fa9 bytes) 000988 1778288000.259003 write 0x001220 thru 0x00fed3 (0xecb4 bytes) 000990 1778288000.283643 trunc from 0x002973 to 0x00b3c0 (0x8a4e bytes) 000991 1778288000.311056 trunc from 0x00d789 to 0x00e6fc (0xf74 bytes) 000993 1778288000.320938 mapread 0x00cafc thru 0x0112cb (0x47d0 bytes) 000994 1778288000.340770 punch from 0x0059b4 to 0x00b528 (0x5b75 bytes) 000995 1778288000.356276 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x112cc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x112cc): Mirror 4: 0x7f434a42 (stale) Mirror 5: 0x7f434a42 (stale) Mirror 14: 0x7f434a42 (stale) Mirror 18: 0x7f434a42 (stale) Mirror 19: 0x7f434a42 (stale) Mirror 24: 0x7f434a42 (stale) Mirror 27: 0x7f434a42 (stale) Mirror 28: 0x7f434a42 (stale) Mirror 30: 0x7f434a42 (stale) Mirror 31: 0x7f434a42 (stale) Mirror 32: 0x7f434a42 (stale) Mirror 33: 0x7f434a42 (stale) Mirror 34: 0x7f434a42 (stale) CRC-32 checksum value for chunk [0, 0x112cc): Mirror 1: 0xc5a586f2 Mirror 35: 0xc5a586f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x112cc: skipped 000996 1778288002.035200 mapread 0x009f0f thru 0x010064 (0x6156 bytes) 000997 1778288002.107770 MIRROR_RESYNC 15 mirrors 000998 1778288004.690568 MIRROR_RESYNC 15 mirrors 000999 1778288005.622917 fallocate 0x002be4 thru 0x00bd2f (0x914c bytes) All operations completed A-OK! PASS 70a (231s) == sanity-flr test 71: check mirror extend parent fid ==== 20:53:32 (1778288012) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 20:53:43 (1778288023) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21391 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892109 s, 11.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 20:53:52 (1778288032) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3769920 .. Extending file size to 7445600 .. failed Extending file size to 7531008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8609440 .. Extending file size to 6453728 .. Extending file size to 7324000 .. Extending file size to 7854720 .. Extending file size to 8533952 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8845120 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8564224 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9349472 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60300 60302 60303 60304 60306 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) 192.168.201.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (96s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:55:28 (1778288128) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3717696 .. Extending file size to 8576160 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8760960 .. Extending file size to 3776640 .. Extending file size to 4760256 .. Extending file size to 4820416 .. Extending file size to 7944736 .. Extending file size to 8659584 .. Extending file size to 9186048 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8852064 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61941 61942 61943 61944 61945 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) 192.168.201.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:56:49 (1778288209) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1269088 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2376 1268928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18704 3588316 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24524 3582496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43228 7170812 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1269088 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2376 1268928 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18704 3588316 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24524 3582496 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43228 7170812 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13882 s, 15.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207537 s, 10.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa8aeaa94 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:57:10 (1778288230) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144196 s, 14.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 20:57:18 (1778288238) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:57:26 (1778288246) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136187 s, 15.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:57:34 (1778288254) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124719 s, 16.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:57:43 (1778288263) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0938974 s, 22.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:58:16 (1778288296) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147247 s, 7.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:59:12 (1778288352) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186318 s, 11.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:00:08 (1778288408) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg104-server: Pool lustre.test_204f created Adding targets to pool oleg104-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg104-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161266 s, 13.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg104-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg104-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg104-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:01:29 (1778288489) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:01:39 (1778288499) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:01:48 (1778288508) oleg104-server: Pool lustre.test_206 created Adding targets to pool oleg104-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg104-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg104-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg104-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg104-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:02:18 (1778288538) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548517 s, 19.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700714 s, 15.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdee73e07 Mirror 3: 0xdee73e07 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:02:29 (1778288549) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:02:33 (1778288553) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:02:37 (1778288557) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0745632 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2040000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2040000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2040000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d80000 Read this (4 bytes): BBBB 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:03:19 (1778288599) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0113239 s, 723 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:03:27 (1778288607) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901074 s, 11.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611226 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 21:03:28.000000000 -0400 Modify: 2026-05-08 21:03:28.000000000 -0400 Change: 2026-05-08 21:03:30.000000000 -0400 Birth: 2026-05-08 21:03:29.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:03:36 (1778288616) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968041 s, 10.8 MB/s oleg104-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg104-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg104-server: O/280000401/d17/b1: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:04:01 (1778288641) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.325633 s, 126 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187184 s, 219 kB/s size after second write -rw-r--r-- 1 root root 4096 May 8 21:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 8 21:04 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:04:09 (1778288649) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:04:17 (1778288657) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2058 sec =========== 21:04:26 (1778288666) === sanity-flr: start cleanup 21:04:28 (1778288668) === === sanity-flr: finish cleanup 21:04:36 (1778288676) ===