-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 01:19:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg632-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:20:12 (1787548812) === oleg632-client.virtnet: executing check_config_client /mnt/lustre oleg632-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg632-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff976a106ac800.idle_timeout=debug osc.lustre-OST0001-osc-ffff976a106ac800.idle_timeout=debug disable quota as required oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:20:55 (1787548855) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8233] == sanity-flr test 0a: lfs mirror create with -N option == 01:20:58 (1787548858) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:21:19 (1787548879) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:21:23 (1787548883) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:21:27 (1787548887) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:21:57 (1787548917) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:22:00 (1787548920) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:22:04 (1787548924) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff976a106ac800.stats=clear osc.lustre-OST0001-osc-ffff976a106ac800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:22:13 (1787548933) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:22:24 (1787548944) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:22:33 (1787548953) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:22:37 (1787548957) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:22:45 (1787548965) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:22:56 (1787548976) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:23:04 (1787548984) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:23:11 (1787548991) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:23:20 (1787549000) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:23:31 (1787549011) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:23:38 (1787549018) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.963511 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06195 s, 16.9 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:23:49 (1787549029) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.809943 s, 25.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:23:59 (1787549039) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:24:11 (1787549051) Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:25:29 (1787549129) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.320836 s, 16.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:26:14 (1787549174) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:27:39 (1787549259) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.70868 s, 14.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 01:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.061s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.784s user 0m0.000s sys 0m0.160s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.224s user 0m0.000s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg632-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976a106ac800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:29:14 (1787549354) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:29:17 (1787549357) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:29:20 (1787549360) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:29:23 (1787549363) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 01:29:30 (1787549370) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.546011 s, 23.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:29:42 (1787549382) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050897 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027819 s, 37.7 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:29:51 (1787549391) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:29:59 (1787549399) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00209962 s, 488 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136974 s, 748 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00173583 s, 590 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155385 s, 659 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 01:30:09 (1787549409) create test files with size 15747143 .. 858793cc74e707d2ad1f1e5eb103df20 - 0b6988a9162ed55471388ebfc68852f7 - 93f1c5980deaa1e2708e3e97aeb19d20 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 01:30:31 (1787549431) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:30:45 (1787549445) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:30:52 (1787549452) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105223 s, 10.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505421 s, 20.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 01:31:01 (1787549461) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141607 s, 14.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.484335 s, 8.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206009 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 01:31:20 (1787549480) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:31:24 (1787549484) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:31:28 (1787549488) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022108 s, 47.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:31:36 (1787549496) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.255157 s, 12.3 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:31:45 (1787549505) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:31:54 (1787549514) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.227082 s, 46.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:32:03 (1787549523) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff976a20a0a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.274439 s, 38.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff976a044f0000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:32:17 (1787549537) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:32:52 (1787549572) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:33:30 (1787549610) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:33:33 (1787549613) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280674 s, 11.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.311471 s, 10.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xce3aac31 (nosync,stale) Mirror 2: 0xce3aac31 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 09db93cae287d098fcbfe59203c0aaad - mirror 1 checksum: 41278ef0ec38539f294eb7ac19612aec - mirror 2 checksum: 09db93cae287d098fcbfe59203c0aaad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T01:33:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T01:33:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 09db93cae287d098fcbfe59203c0aaad - mirror 1 checksum: 41278ef0ec38539f294eb7ac19612aec - mirror 2 checksum: 41278ef0ec38539f294eb7ac19612aec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T01:33:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T01:33:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:33:45 (1787549625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289752 s, 36.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:33:53 (1787549633) Failing mds1 on oleg632-server Stopping /mnt/lustre-mds1 (opts:) on oleg632-server 01:34:05 (1787549645) shut down facet: mds1 facet_host: oleg632-server facet_failover_host: oleg632-server Failover mds1 to oleg632-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-MDT0000 01:34:36 (1787549676) targets are mounted 01:34:36 (1787549676) facet_failover done oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422452 s, 24.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:34:53 (1787549693) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0663111 s, 309 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0473803 s, 432 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:35:04 (1787549704) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0494989 s, 414 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0122079 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:35:13 (1787549713) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0111089 s, 1.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:35:21 (1787549721) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0833112 s, 246 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:35:30 (1787549730) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00430964 s, 950 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0534106 s, 76.7 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:35:39 (1787549739) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:36:04 (1787549764) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:36:30 (1787549790) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:36:53 (1787549813) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:37:16 (1787549836) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 01:37:23 (1787549843) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57578 mirror split pid 57580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57578 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 01:38:31 (1787549911) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9914 skipping zero size read fallocating to largest ever: 0x12731 000005 1787549914.307426 fallocate 0x00c98e thru 0x012730 (0x5da3 bytes) 000006 1787549914.316007 punch from 0x00970b to 0x009d89 (0x67f bytes) 000009 1787549914.322245 write 0x037a4f thru 0x03ded1 (0x6483 bytes) 000010 1787549914.332116 trunc from 0x03ded2 to 0x0006b7 (0x3d81a bytes) 000011 1787549914.352549 trunc from 0x000139 to 0x0006b7 (0x57f bytes) 000013 1787549914.362500 read 0x0004a0 thru 0x0006b7 (0x218 bytes) 000014 1787549914.370763 mapread 0x0004d2 thru 0x0006b7 (0x1e6 bytes) fallocating to largest ever: 0x3a398 000018 1787549914.379769 fallocate 0x02d8ac thru 0x03a397 (0xcaec bytes) 000019 1787549914.393281 mapread 0x013ed6 thru 0x021b3e (0xdc69 bytes) 000020 1787549914.466950 read 0x020f76 thru 0x02df7c (0xd007 bytes) 000021 1787549914.487024 read 0x002299 thru 0x00ab88 (0x88f0 bytes) 000022 1787549914.489662 fallocate 0x002257 thru 0x007ddb (0x5b85 bytes) 000023 1787549914.495314 fallocate 0x029c60 thru 0x035e03 (0xc1a4 bytes) 000024 1787549914.499541 trunc from 0x03a398 to 0x01b442 (0x1ef55 bytes) 000025 1787549914.519814 trunc from 0x01b443 to 0x0399a9 (0x1e567 bytes) 000026 1787549914.540890 read 0x000f1e thru 0x001678 (0x75b bytes) 000027 1787549914.541556 mapread 0x02f0a6 thru 0x03222b (0x3186 bytes) 000029 1787549914.554853 fallocate 0x01c5b8 thru 0x01dbf9 (0x1642 bytes) 000030 1787549914.561034 trunc from 0x002eed to 0x00c0bf (0x91d3 bytes) 000031 1787549914.571231 MIRROR_EXTEND to 2 mirrors 000032 1787549915.348365 trunc from 0x025c01 to 0x0288a5 (0x2ca5 bytes) 000033 1787549915.398832 mapwrite 0x02d8d9 thru 0x037bd7 (0xa2ff bytes) 000034 1787549915.467643 punch from 0x020bf5 to 0x0226ca (0x1ad6 bytes) 000035 1787549915.473453 punch from 0x01fbe0 to 0x02af19 (0xb33a bytes) 000036 1787549915.486922 trunc from 0x0399aa to 0x000ce4 (0x38cc5 bytes) 000037 1787549915.559175 mapwrite 0x01e01b thru 0x02a8a6 (0xc88c bytes) 000038 1787549915.647710 trunc from 0x02a8a7 to 0x016d71 (0x13b35 bytes) 000039 1787549915.682806 mapread 0x01582e thru 0x016d71 (0x1544 bytes) 000040 1787549915.692772 mapwrite 0x02a448 thru 0x0351c6 (0xad7f bytes) 000041 1787549915.776925 write 0x026cc1 thru 0x0354e0 (0xe820 bytes) 000042 1787549915.778657 write 0x00e1dd thru 0x01cb1e (0xe942 bytes) 000043 1787549915.788270 read 0x028540 thru 0x033df0 (0xb8b1 bytes) 000044 1787549915.789375 mapread 0x012688 thru 0x01fef9 (0xd872 bytes) 000045 1787549915.811241 mapwrite 0x0047c0 thru 0x012ca2 (0xe4e3 bytes) 000046 1787549915.878229 punch from 0x021257 to 0x02a65c (0x9406 bytes) 000047 1787549915.896597 trunc from 0x00bdd9 to 0x015010 (0x9238 bytes) 000049 1787549915.904374 fallocate 0x000564 thru 0x0099e2 (0x947f bytes) 000050 1787549915.913857 read 0x0332b6 thru 0x0354e0 (0x222b bytes) 000051 1787549915.914880 write 0x010553 thru 0x01baa9 (0xb557 bytes) 000053 1787549915.924812 trunc from 0x0180e2 to 0x0216b7 (0x95d6 bytes) 000055 1787549915.953687 mapread 0x014de6 thru 0x017835 (0x2a50 bytes) 000056 1787549915.955944 trunc from 0x00c907 to 0x01ae32 (0xe52c bytes) 000057 1787549915.960921 write 0x035b03 thru 0x037fad (0x24ab bytes) 000058 1787549915.963931 mapread 0x00b6fd thru 0x00c1ff (0xb03 bytes) 000059 1787549915.969222 write 0x032ea3 thru 0x036e66 (0x3fc4 bytes) 000061 1787549915.973027 read 0x001643 thru 0x00d770 (0xc12e bytes) 000062 1787549915.987332 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000064 1787549916.147158 MIRROR_EXTEND to 2 mirrors 000065 1787549917.043638 trunc from 0x037fae to 0x01aab7 (0x1d4f6 bytes) 000066 1787549917.101320 write 0x00dfb7 thru 0x01d0f1 (0xf13b bytes) 000067 1787549917.113325 read 0x016aee thru 0x01d0f1 (0x6604 bytes) 000068 1787549917.113906 write 0x019807 thru 0x022112 (0x890c bytes) 000069 1787549917.121215 mapwrite 0x0182d9 thru 0x018d30 (0xa58 bytes) 000070 1787549917.147800 fallocate 0x035636 thru 0x039596 (0x3f61 bytes) 000071 1787549917.160147 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: 0xb8cebcb4 (stale) CRC-32 checksum value for chunk [0, 0x22113): Mirror 1: 0xa7f4629b Mirror 3: 0xa7f4629b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22113: skipped 000072 1787549918.201205 mapwrite 0x00e88c thru 0x01d35b (0xead0 bytes) 000074 1787549918.361647 write 0x01622e thru 0x01ff30 (0x9d03 bytes) 000075 1787549918.373004 mapread 0x0120f4 thru 0x0149e0 (0x28ed bytes) 000076 1787549918.374564 read 0x0084fc thru 0x00cd82 (0x4887 bytes) 000077 1787549918.375346 fallocate 0x01780e thru 0x01a475 (0x2c68 bytes) 000079 1787549918.388296 mapread 0x00392d thru 0x00cf42 (0x9616 bytes) 000080 1787549918.405689 read 0x02021e thru 0x022112 (0x1ef5 bytes) 000081 1787549918.427105 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: 0xb8cebcb4 (stale) Mirror 3: 0xa7f4629b (stale) CRC-32 checksum value for chunk [0, 0x22113): Mirror 1: 0x7cb6161c Mirror 4: 0x7cb6161c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22113: skipped 000082 1787549919.635205 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22113): Mirror 2: 0xb8cebcb4 (stale) Mirror 3: 0xa7f4629b (stale) CRC-32 checksum value for chunk [0, 0x22113): Mirror 1: 0x7cb6161c Mirror 4: 0x7cb6161c Mirror 5: 0x7cb6161c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22113: skipped 000083 1787549920.647902 write 0x00d153 thru 0x00fbad (0x2a5b bytes) 000084 1787549920.698819 trunc from 0x017e02 to 0x01aadc (0x2cdb bytes) 000086 1787549920.717134 read 0x00ca9a thru 0x00efb5 (0x251c bytes) 000088 1787549920.725705 punch from 0x021a46 to 0x022112 (0x6cd bytes) 000089 1787549920.732550 mapwrite 0x00a4d7 thru 0x00a722 (0x24c bytes) 000090 1787549920.759469 write 0x02bebc thru 0x031e31 (0x5f76 bytes) 000091 1787549920.760705 write 0x004130 thru 0x0113cd (0xd29e bytes) 000092 1787549920.773727 mapwrite 0x002933 thru 0x00a653 (0x7d21 bytes) 000094 1787549920.816255 write 0x02d7bd thru 0x02eb1a (0x135e bytes) 000095 1787549920.817633 fallocate 0x01e630 thru 0x023cde (0x56af bytes) 000097 1787549920.828212 MIRROR_RESYNC 5 mirrors 000098 1787549922.664547 MIRROR_RESYNC 5 mirrors 000099 1787549923.099922 trunc from 0x031e32 to 0x020e6a (0x10fc7 bytes) 000100 1787549923.137778 read 0x00064e thru 0x00fd04 (0xf6b7 bytes) 000101 1787549923.157454 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e6b: skipped 000103 1787549923.632976 trunc from 0x0188e6 to 0x019ba8 (0x12c3 bytes) 000104 1787549923.643661 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 1: 0x46a5af29 Mirror 6: 0x46a5af29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e6b: skipped 000105 1787549924.990930 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 1: 0x46a5af29 Mirror 6: 0x46a5af29 Mirror 7: 0x46a5af29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e6b: skipped 000106 1787549925.907787 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 2: 0x46a5af29 (stale) Mirror 3: 0x46a5af29 (stale) Mirror 5: 0x46a5af29 (stale) CRC-32 checksum value for chunk [0, 0x20e6b): Mirror 1: 0x46a5af29 Mirror 7: 0x46a5af29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e6b: skipped 000107 1787549926.650723 read 0x002de7 thru 0x01284e (0xfa68 bytes) 000108 1787549926.678799 write 0x027ac6 thru 0x02f511 (0x7a4c bytes) 000110 1787549926.732967 read 0x020f12 thru 0x02f511 (0xe600 bytes) 000111 1787549926.749970 mapread 0x0024b3 thru 0x007910 (0x545e bytes) 000112 1787549926.751258 fallocate 0x0145c7 thru 0x0149f2 (0x42c bytes) 000113 1787549926.761318 read 0x00d93a thru 0x010d2e (0x33f5 bytes) 000114 1787549926.777149 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f512): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f512): Mirror 2: 0x3deb0501 (stale) Mirror 3: 0x3deb0501 (stale) Mirror 5: 0x3deb0501 (stale) Mirror 7: 0xb38d3840 (stale) CRC-32 checksum value for chunk [0, 0x2f512): Mirror 1: 0x38363ecf Mirror 8: 0x38363ecf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f512: skipped 000115 1787549928.010067 write 0x0242e6 thru 0x02a01a (0x5d35 bytes) 000116 1787549928.115168 trunc from 0x016fc7 to 0x01c83b (0x5875 bytes) 000117 1787549928.126065 fallocate 0x01197e thru 0x018c04 (0x7287 bytes) 000119 1787549928.156915 mapread 0x02377a thru 0x02e487 (0xad0e bytes) 000120 1787549928.207845 trunc from 0x014490 to 0x01bdb7 (0x7928 bytes) 000121 1787549928.224033 MIRROR_RESYNC 6 mirrors 000122 1787549929.809049 mapread 0x018514 thru 0x0186a2 (0x18f bytes) 000124 1787549929.815305 read 0x021d4a thru 0x028cc4 (0x6f7b bytes) 000125 1787549929.842564 fallocate 0x023bfa thru 0x02a6f7 (0x6afe bytes) 000126 1787549929.880524 write 0x018a90 thru 0x01c38b (0x38fc bytes) 000127 1787549929.936786 trunc from 0x02f512 to 0x00f0bb (0x20456 bytes) 000128 1787549929.989808 write 0x00fb47 thru 0x01c93d (0xcdf7 bytes) 000129 1787549930.020607 write 0x002b9d thru 0x00f7ed (0xcc51 bytes) 000130 1787549930.038822 read 0x00debe thru 0x0148a5 (0x69e8 bytes) 000132 1787549930.044593 trunc from 0x006c6b to 0x014803 (0xdb99 bytes) 000133 1787549930.151776 write 0x038a96 thru 0x03ece4 (0x624f bytes) 000134 1787549930.159860 write 0x0062c0 thru 0x00c100 (0x5e41 bytes) 000135 1787549930.224726 write 0x00c38c thru 0x019c4e (0xd8c3 bytes) 000136 1787549930.229271 trunc from 0x017287 to 0x01feb8 (0x8c32 bytes) 000137 1787549930.276747 trunc from 0x03ece5 to 0x00ae99 (0x33e4b bytes) 000138 1787549930.332021 mapread 0x0077d7 thru 0x00938d (0x1bb7 bytes) 000139 1787549930.333007 write 0x03f54e thru 0x03ffff (0xab2 bytes) 000140 1787549930.342032 trunc from 0x040000 to 0x00cc76 (0x33389 bytes) 000141 1787549930.370415 mapwrite 0x01014e thru 0x01df4a (0xddfd bytes) 000142 1787549930.520748 fallocate 0x0282cd thru 0x032594 (0xa2c8 bytes) 000143 1787549930.525585 mapwrite 0x00148b thru 0x00fb20 (0xe696 bytes) 000144 1787549930.564218 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1df4b): Mirror 2: 0x409eecb3 (stale) Mirror 3: 0x409eecb3 (stale) Mirror 5: 0x409eecb3 (stale) Mirror 8: 0x409eecb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df4b): Mirror 2: 0x409eecb3 (stale) Mirror 3: 0x409eecb3 (stale) Mirror 5: 0x409eecb3 (stale) Mirror 8: 0x409eecb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df4b: skipped 000146 1787549930.943539 write 0x006013 thru 0x010966 (0xa954 bytes) 000147 1787549930.965090 fallocate 0x0034c3 thru 0x00b928 (0x8466 bytes) 000148 1787549930.972971 write 0x007c59 thru 0x01003f (0x83e7 bytes) 000149 1787549930.974570 read 0x006ec2 thru 0x009391 (0x24d0 bytes) 000150 1787549930.976145 read 0x003dbd thru 0x012275 (0xe4b9 bytes) 000152 1787549930.991669 fallocate 0x002f1c thru 0x004cf5 (0x1dda bytes) 000153 1787549930.997006 trunc from 0x001b7a to 0x00334a (0x17d1 bytes) 000154 1787549931.005940 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1df4b): Mirror 2: 0x409eecb3 (stale) Mirror 3: 0x409eecb3 (stale) Mirror 8: 0x409eecb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df4b): Mirror 2: 0x409eecb3 (stale) Mirror 3: 0x409eecb3 (stale) Mirror 8: 0x409eecb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df4b: skipped 000155 1787549931.505690 write 0x03e5ab thru 0x03ffff (0x1a55 bytes) 000156 1787549931.519905 punch from 0x007da6 to 0x00ed46 (0x6fa1 bytes) 000157 1787549931.528684 write 0x021ff4 thru 0x0278d7 (0x58e4 bytes) 000158 1787549931.554659 mapwrite 0x0191c9 thru 0x01c4c4 (0x32fc bytes) 000159 1787549931.631741 trunc from 0x040000 to 0x002600 (0x3d9ff bytes) 000161 1787549931.669163 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2601): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2601): Mirror 2: 0xaf0b5969 (stale) Mirror 3: 0xaf0b5969 (stale) Mirror 8: 0xaf0b5969 (stale) CRC-32 checksum value for chunk [0, 0x2601): Mirror 1: 0x9e139cf Mirror 9: 0x9e139cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2601: skipped 000162 1787549932.780266 read 0x0003fa thru 0x002600 (0x2207 bytes) 000163 1787549932.790120 mapread 0x001f42 thru 0x002600 (0x6bf bytes) 000164 1787549932.809820 read 0x00033d thru 0x0025d3 (0x2297 bytes) 000165 1787549932.818914 mapwrite 0x02901f thru 0x02b6f0 (0x26d2 bytes) 000166 1787549932.921918 punch from 0x026760 to 0x026874 (0x115 bytes) 000167 1787549932.927735 trunc from 0x02b6f1 to 0x0011f9 (0x2a4f7 bytes) 000168 1787549932.956186 mapwrite 0x012abc thru 0x0137a8 (0xced bytes) 000169 1787549933.010415 fallocate 0x0265fb thru 0x0311e4 (0xabea bytes) 000170 1787549933.019913 mapread 0x011e01 thru 0x0137a8 (0x19a8 bytes) 000171 1787549933.039987 fallocate 0x019df4 thru 0x020f39 (0x7146 bytes) 000172 1787549933.053034 mapread 0x005110 thru 0x0137a8 (0xe699 bytes) 000173 1787549933.129268 read 0x003de6 thru 0x0070a3 (0x32be bytes) 000174 1787549933.136853 read 0x00f652 thru 0x0137a8 (0x4157 bytes) 000175 1787549933.137987 fallocate 0x0107ab thru 0x011dfa (0x1650 bytes) 000176 1787549933.147292 trunc from 0x00d16d to 0x00e35d (0x11f1 bytes) 000177 1787549933.161507 fallocate 0x023820 thru 0x028fa3 (0x5784 bytes) 000178 1787549933.170745 read 0x01a531 thru 0x027b2a (0xd5fa bytes) 000179 1787549933.185568 trunc from 0x028fa4 to 0x001677 (0x2792c bytes) 000180 1787549933.205659 read 0x0013cb thru 0x001677 (0x2ad bytes) 000181 1787549933.206562 mapread 0x000eea thru 0x001677 (0x78e bytes) 000182 1787549933.211270 write 0x037ff9 thru 0x03ffff (0x8007 bytes) 000183 1787549933.213977 write 0x0239e7 thru 0x02aaa9 (0x70c3 bytes) 000184 1787549933.228881 mapwrite 0x02e58b thru 0x0318b8 (0x332e bytes) 000185 1787549933.269664 punch from 0x020234 to 0x0284f2 (0x82bf bytes) 000186 1787549933.276005 punch from 0x0374d4 to 0x03f352 (0x7e7f bytes) 000187 1787549933.286793 fallocate 0x027a57 thru 0x02f481 (0x7a2b bytes) 000188 1787549933.291920 trunc from 0x037001 to 0x03ffff (0x8fff bytes) 000189 1787549933.297885 mapread 0x01a098 thru 0x01fe8b (0x5df4 bytes) 000190 1787549933.299048 trunc from 0x033b67 to 0x035375 (0x180f bytes) 000191 1787549933.304223 write 0x02cd87 thru 0x03ab90 (0xde0a bytes) 000192 1787549933.312687 punch from 0x00a1c6 to 0x019d41 (0xfb7c bytes) 000193 1787549933.317794 mapread 0x001a83 thru 0x007921 (0x5e9f bytes) 000194 1787549933.319310 read 0x001bfd thru 0x005942 (0x3d46 bytes) 000195 1787549933.320045 trunc from 0x0261f2 to 0x02a8af (0x46be bytes) 000196 1787549933.325993 mapread 0x033d54 thru 0x03ffff (0xc2ac bytes) 000197 1787549933.338964 write 0x01b79b thru 0x0269a7 (0xb20d bytes) 000198 1787549933.341728 trunc from 0x03a183 to 0x03ffff (0x5e7d bytes) 000199 1787549933.361667 trunc from 0x040000 to 0x012a54 (0x2d5ab bytes) 000200 1787549933.384959 punch from 0x00702a to 0x00a78b (0x3762 bytes) 000201 1787549933.390778 punch from 0x0103c7 to 0x012a54 (0x268e bytes) 000202 1787549933.395548 read 0x006092 thru 0x006ca4 (0xc13 bytes) 000203 1787549933.396631 trunc from 0x012a55 to 0x03170b (0x1ecb7 bytes) 000205 1787549933.416035 mapwrite 0x02123b thru 0x02357d (0x2343 bytes) 000206 1787549933.460376 mapwrite 0x01d457 thru 0x02600c (0x8bb6 bytes) 000207 1787549933.507680 read 0x005ccf thru 0x00a831 (0x4b63 bytes) 000208 1787549933.512251 read 0x023a31 thru 0x03018b (0xc75b bytes) fallocating to largest ever: 0x40000 000210 1787549933.517469 fallocate 0x03e281 thru 0x03ffff (0x1d7f bytes) 000211 1787549933.526423 fallocate 0x00e80f thru 0x016706 (0x7ef8 bytes) 000212 1787549933.533926 fallocate 0x031e66 thru 0x035754 (0x38ef bytes) 000213 1787549933.540800 trunc from 0x03170c to 0x03f558 (0xde4d bytes) 000214 1787549933.559913 trunc from 0x03f559 to 0x0031d6 (0x3c382 bytes) 000215 1787549933.581983 trunc from 0x0031d7 to 0x036d4c (0x33b76 bytes) 000216 1787549933.602700 trunc from 0x036d4d to 0x012067 (0x24ce5 bytes) 000217 1787549933.629625 mapread 0x000deb thru 0x005901 (0x4b17 bytes) 000218 1787549933.643998 mapread 0x00a792 thru 0x012067 (0x78d6 bytes) 000219 1787549933.645855 mapread 0x0084da thru 0x012067 (0x9b8e bytes) 000220 1787549933.650378 fallocate 0x005b2d thru 0x00b703 (0x5bd7 bytes) 000221 1787549933.675822 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12068): Mirror 2: 0xe8180fbf (stale) Mirror 3: 0xe8180fbf (stale) Mirror 9: 0xb2f39494 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12068): Mirror 2: 0xe8180fbf (stale) Mirror 3: 0xe8180fbf (stale) Mirror 9: 0xb2f39494 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12068: skipped 000222 1787549934.221436 fallocate 0x02847c thru 0x031e74 (0x99f9 bytes) 000224 1787549934.231734 trunc from 0x031e75 to 0x011f6d (0x1ff07 bytes) 000225 1787549934.263919 fallocate 0x0130be thru 0x022f85 (0xfec8 bytes) 000226 1787549934.271264 read 0x01194f thru 0x011f6d (0x61f bytes) 000227 1787549934.283952 mapwrite 0x029722 thru 0x036e12 (0xd6f1 bytes) 000228 1787549934.403869 mapread 0x028d1f thru 0x02b4bb (0x279d bytes) 000229 1787549934.406006 trunc from 0x000158 to 0x00ebab (0xea54 bytes) 000230 1787549934.418032 mapwrite 0x018797 thru 0x0264a4 (0xdd0e bytes) 000232 1787549934.499024 trunc from 0x036e13 to 0x007da0 (0x2f072 bytes) 000233 1787549934.523780 read 0x005f63 thru 0x007da0 (0x1e3e bytes) 000234 1787549934.550036 trunc from 0x007da1 to 0x0035ee (0x47b2 bytes) 000235 1787549934.600949 read 0x00056b thru 0x0035ee (0x3084 bytes) 000236 1787549934.601988 trunc from 0x0035ef to 0x03d8d3 (0x3a2e5 bytes) 000237 1787549934.625573 punch from 0x032991 to 0x03b96b (0x8fdb bytes) 000238 1787549934.630443 mapread 0x03c3f5 thru 0x03d813 (0x141f bytes) 000239 1787549934.663417 punch from 0x014dc9 to 0x02193c (0xcb74 bytes) 000240 1787549934.669665 mapread 0x012453 thru 0x01837e (0x5f2c bytes) 000242 1787549934.698975 trunc from 0x00e828 to 0x00f678 (0xe51 bytes) 000243 1787549934.705952 mapwrite 0x019a76 thru 0x020e9c (0x7427 bytes) 000244 1787549934.767243 read 0x023ee4 thru 0x02cd4b (0x8e68 bytes) 000245 1787549934.769266 mapread 0x03b65c thru 0x03d8d3 (0x2278 bytes) 000246 1787549934.771564 mapwrite 0x00009d thru 0x0056d5 (0x5639 bytes) 000248 1787549934.793123 fallocate 0x009d7d thru 0x00eba4 (0x4e28 bytes) 000249 1787549934.801279 punch from 0x009377 to 0x00fae5 (0x676f bytes) 000250 1787549934.806906 punch from 0x025341 to 0x02658b (0x124b bytes) 000251 1787549934.812501 mapwrite 0x00f3d9 thru 0x01aef3 (0xbb1b bytes) 000253 1787549934.848118 trunc from 0x001f71 to 0x003bc1 (0x1c51 bytes) 000254 1787549934.854625 fallocate 0x029e1c thru 0x031ff2 (0x81d7 bytes) 000255 1787549934.860156 fallocate 0x025b78 thru 0x02b7cc (0x5c55 bytes) 000256 1787549934.873620 MIRROR_RESYNC 4 mirrors 000257 1787549936.120759 MIRROR_EXTEND to 5 mirrors 000258 1787549937.077885 MIRROR_EXTEND to 6 mirrors 000259 1787549938.130851 write 0x0357af thru 0x037c63 (0x24b5 bytes) 000260 1787549938.193375 write 0x02a1c6 thru 0x0310c5 (0x6f00 bytes) 000261 1787549938.203648 trunc from 0x03d8d4 to 0x028028 (0x158ab bytes) 000262 1787549938.230601 read 0x0105de thru 0x015b72 (0x5595 bytes) 000263 1787549938.238584 write 0x003c67 thru 0x012b2d (0xeec7 bytes) 000264 1787549938.244443 fallocate 0x012c2f thru 0x01fbf2 (0xcfc4 bytes) 000265 1787549938.249467 mapwrite 0x00eaac thru 0x00fe1b (0x1370 bytes) 000266 1787549938.265110 trunc from 0x028029 to 0x03f1c2 (0x1719a bytes) 000267 1787549938.281357 trunc from 0x03f1c3 to 0x02fb50 (0xf672 bytes) 000268 1787549938.298960 read 0x02e647 thru 0x02fb50 (0x150a bytes) 000269 1787549938.316059 punch from 0x0087ca to 0x00de27 (0x565e bytes) 000271 1787549938.322615 read 0x01466d thru 0x0188cc (0x4260 bytes) 000272 1787549938.324155 trunc from 0x00898a to 0x00c3d0 (0x3a47 bytes) 000273 1787549938.329214 trunc from 0x00c2e8 to 0x00c9eb (0x704 bytes) 000274 1787549938.335988 mapread 0x0132ac thru 0x01436c (0x10c1 bytes) 000275 1787549938.337258 mapread 0x022514 thru 0x02dac6 (0xb5b3 bytes) 000276 1787549938.339485 mapwrite 0x023ae7 thru 0x02eeea (0xb404 bytes) 000278 1787549938.363299 mapread 0x01175b thru 0x01758a (0x5e30 bytes) 000279 1787549938.365107 trunc from 0x02fb51 to 0x035027 (0x54d7 bytes) 000280 1787549938.383667 trunc from 0x035028 to 0x01b434 (0x19bf3 bytes) 000281 1787549938.405363 mapwrite 0x00e672 thru 0x01ce7f (0xe80e bytes) 000282 1787549938.475202 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ce80): Mirror 3: 0x4ce3954d (stale) Mirror 9: 0x4ce3954d (stale) Mirror 10: 0x4ce3954d (stale) Mirror 11: 0x4ce3954d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ce80): Mirror 3: 0x4ce3954d (stale) Mirror 9: 0x4ce3954d (stale) Mirror 10: 0x4ce3954d (stale) Mirror 11: 0x4ce3954d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ce80: skipped 000283 1787549938.984214 mapread 0x004142 thru 0x01398f (0xf84e bytes) 000284 1787549939.026022 write 0x03fb9a thru 0x03ffff (0x466 bytes) 000286 1787549939.043169 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x66c1155 (stale) Mirror 10: 0x66c1155 (stale) Mirror 11: 0x66c1155 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x66c1155 (stale) Mirror 10: 0x66c1155 (stale) Mirror 11: 0x66c1155 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000288 1787549939.492287 punch from 0x02d684 to 0x03ad7a (0xd6f7 bytes) 000289 1787549939.497106 punch from 0x03a95a to 0x03ffff (0x56a6 bytes) 000290 1787549939.502169 punch from 0x0286c3 to 0x037833 (0xf171 bytes) 000291 1787549939.506883 read 0x03aae2 thru 0x03ffff (0x551e bytes) 000292 1787549939.516406 read 0x01b428 thru 0x0266b1 (0xb28a bytes) 000293 1787549939.527338 fallocate 0x03f4cc thru 0x03ffff (0xb34 bytes) 000294 1787549939.538181 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x66c1155 (stale) Mirror 10: 0x66c1155 (stale) Mirror 11: 0x66c1155 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf3e8c17 Mirror 12: 0xf3e8c17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1787549940.927784 write 0x039995 thru 0x03ffff (0x666b bytes) 000296 1787549941.006884 mapread 0x025f88 thru 0x02ca7c (0x6af5 bytes) 000297 1787549941.027733 trunc from 0x040000 to 0x027b28 (0x184d7 bytes) 000298 1787549941.050214 fallocate 0x02df9b thru 0x03acda (0xcd40 bytes) 000299 1787549941.055992 punch from 0x0065e9 to 0x014903 (0xe31b bytes) 000300 1787549941.062880 mapwrite 0x022b4a thru 0x02e240 (0xb6f7 bytes) 000301 1787549941.142913 MIRROR_RESYNC 5 mirrors 000302 1787549942.599359 MIRROR_RESYNC 5 mirrors 000303 1787549943.198345 read 0x01d917 thru 0x022fe4 (0x56ce bytes) 000304 1787549943.213635 mapwrite 0x01423c thru 0x01f68c (0xb451 bytes) 000305 1787549943.351464 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e241): Mirror 10: 0xdc379d96 (stale) Mirror 11: 0xdc379d96 (stale) Mirror 12: 0xdc379d96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e241): Mirror 10: 0xdc379d96 (stale) Mirror 11: 0xdc379d96 (stale) Mirror 12: 0xdc379d96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e241: skipped 000306 1787549943.771203 trunc from 0x012a09 to 0x012df7 (0x3ef bytes) 000307 1787549943.777213 write 0x00248d thru 0x00ff3d (0xdab1 bytes) 000308 1787549943.795216 mapwrite 0x03209a thru 0x033ead (0x1e14 bytes) 000309 1787549943.872053 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33eae): Mirror 10: 0x5620fac9 (stale) Mirror 12: 0x5620fac9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33eae): Mirror 10: 0x5620fac9 (stale) Mirror 12: 0x5620fac9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33eae: skipped 000310 1787549944.330550 MIRROR_RESYNC 3 mirrors 000311 1787549945.287778 read 0x00fa3b thru 0x019b63 (0xa129 bytes) 000312 1787549945.296721 punch from 0x03073b to 0x033ead (0x3773 bytes) 000313 1787549945.345536 mapread 0x00b106 thru 0x0133ff (0x82fa bytes) 000314 1787549945.371183 fallocate 0x03de44 thru 0x03ffff (0x21bc bytes) 000315 1787549945.375851 read 0x015177 thru 0x024eb5 (0xfd3f bytes) 000316 1787549945.388970 mapread 0x00bf2c thru 0x019dc4 (0xde99 bytes) 000317 1787549945.394648 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33eae): Mirror 10: 0xea65012f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33eae): Mirror 10: 0xea65012f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33eae: skipped 000318 1787549945.657911 trunc from 0x033eae to 0x038ca6 (0x4df9 bytes) 000319 1787549945.679316 read 0x017a71 thru 0x026bee (0xf17e bytes) 000320 1787549945.688628 mapread 0x01e73a thru 0x0216ba (0x2f81 bytes) 000322 1787549945.690750 fallocate 0x02f903 thru 0x03eec7 (0xf5c5 bytes) 000324 1787549945.700180 MIRROR_RESYNC 2 mirrors 000326 1787549946.515221 read 0x026175 thru 0x034565 (0xe3f1 bytes) 000327 1787549946.537984 MIRROR_EXTEND to 3 mirrors 000329 1787549947.655368 trunc from 0x0142bd to 0x01b008 (0x6d4c bytes) 000330 1787549947.695300 trunc from 0x0026b6 to 0x00450a (0x1e55 bytes) 000331 1787549947.703084 punch from 0x014d6d to 0x022858 (0xdaec bytes) 000333 1787549947.720061 mapwrite 0x011dc7 thru 0x0148ab (0x2ae5 bytes) 000335 1787549947.783108 mapread 0x00ba44 thru 0x011e38 (0x63f5 bytes) 000336 1787549947.828514 MIRROR_RESYNC 3 mirrors 000337 1787549949.020093 trunc from 0x02aa1c to 0x02ab7c (0x161 bytes) 000338 1787549949.072442 fallocate 0x029e8a thru 0x0397ab (0xf922 bytes) 000339 1787549949.086008 trunc from 0x024df3 to 0x02be68 (0x7076 bytes) 000340 1787549949.095126 trunc from 0x038ca7 to 0x00983d (0x2f469 bytes) 000341 1787549949.143028 trunc from 0x00983e to 0x01dfc4 (0x14787 bytes) 000342 1787549949.174216 mapread 0x0139d2 thru 0x01d919 (0x9f48 bytes) 000344 1787549949.211760 write 0x01ae36 thru 0x0247c6 (0x9991 bytes) 000345 1787549949.215831 mapwrite 0x01e680 thru 0x02ce4b (0xe7cc bytes) 000347 1787549949.306695 write 0x0039c6 thru 0x0116a7 (0xdce2 bytes) 000348 1787549949.314618 mapread 0x008542 thru 0x0099aa (0x1469 bytes) 000349 1787549949.315741 write 0x02a67c thru 0x039f98 (0xf91d bytes) 000350 1787549949.318731 punch from 0x01c37b to 0x024f51 (0x8bd7 bytes) 000351 1787549949.323517 mapread 0x02fc0b thru 0x035d5a (0x6150 bytes) 000352 1787549949.325458 mapwrite 0x02e242 thru 0x031c7d (0x3a3c bytes) 000354 1787549949.341162 punch from 0x02e8dd to 0x038514 (0x9c38 bytes) 000355 1787549949.349773 trunc from 0x01069f to 0x016187 (0x5ae9 bytes) 000357 1787549949.361932 punch from 0x02e195 to 0x037396 (0x9202 bytes) 000358 1787549949.368081 punch from 0x0071e1 to 0x00cbb6 (0x59d6 bytes) 000359 1787549949.374166 mapread 0x014268 thru 0x01ed7a (0xab13 bytes) 000360 1787549949.393353 trunc from 0x01bd12 to 0x0250e1 (0x93d0 bytes) 000361 1787549949.401877 trunc from 0x0388cf to 0x039f98 (0x16ca bytes) 000362 1787549949.414514 trunc from 0x0351f4 to 0x039f98 (0x4da5 bytes) 000363 1787549949.429446 read 0x02443a thru 0x027752 (0x3319 bytes) 000364 1787549949.436751 mapwrite 0x03d4dd thru 0x03ffff (0x2b23 bytes) 000365 1787549949.488106 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x2a44843d (stale) Mirror 11: 0x2a44843d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5fe84ff5 Mirror 12: 0x5fe84ff5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000366 1787549950.717294 punch from 0x0059a9 to 0x00debb (0x8513 bytes) 000367 1787549950.776712 read 0x02394e thru 0x030425 (0xcad8 bytes) 000369 1787549950.792812 trunc from 0x01bf3e to 0x02a6fe (0xe7c1 bytes) 000370 1787549950.806122 mapwrite 0x01826b thru 0x023c80 (0xba16 bytes) 000372 1787549950.882632 fallocate 0x01c3f2 thru 0x02b273 (0xee82 bytes) 000373 1787549950.891178 write 0x02dfec thru 0x0336d9 (0x56ee bytes) 000374 1787549950.903367 mapwrite 0x007824 thru 0x007cdd (0x4ba bytes) 000375 1787549950.947171 read 0x0257fe thru 0x0267a6 (0xfa9 bytes) 000377 1787549950.948294 trunc from 0x040000 to 0x001f5b (0x3e0a4 bytes) 000378 1787549950.985561 fallocate 0x028f0c thru 0x036ad3 (0xdbc8 bytes) 000379 1787549950.992088 trunc from 0x001f5c to 0x0106f4 (0xe799 bytes) 000380 1787549951.015913 punch from 0x010679 to 0x0106f4 (0x7c bytes) 000381 1787549951.026991 punch from 0x009c0c to 0x00cdda (0x31cf bytes) 000382 1787549951.037465 trunc from 0x00fa95 to 0x0106f4 (0xc60 bytes) 000383 1787549951.052852 read 0x0079eb thru 0x0106f4 (0x8d0a bytes) 000384 1787549951.072368 read 0x004edd thru 0x0060b1 (0x11d5 bytes) 000385 1787549951.073070 trunc from 0x0106f5 to 0x01b719 (0xb025 bytes) 000387 1787549951.097223 mapwrite 0x015c6f thru 0x020d8e (0xb120 bytes) 000388 1787549951.143619 punch from 0x00598a to 0x009cd2 (0x4349 bytes) 000389 1787549951.148560 trunc from 0x01d933 to 0x020d8e (0x345c bytes) 000390 1787549951.157014 fallocate 0x033950 thru 0x036e94 (0x3545 bytes) 000391 1787549951.169030 MIRROR_RESYNC 4 mirrors 000393 1787549952.231991 fallocate 0x02c979 thru 0x039bfc (0xd284 bytes) 000394 1787549952.296812 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 10: 0x9af2c9ec (stale) Mirror 12: 0x9af2c9ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 10: 0x9af2c9ec (stale) Mirror 12: 0x9af2c9ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bfd: skipped 000395 1787549952.711626 fallocate 0x02b945 thru 0x0321fb (0x68b7 bytes) 000396 1787549952.720099 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 10: 0x9af2c9ec (stale) Mirror 12: 0x9af2c9ec (stale) CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 1: 0x9af2c9ec Mirror 13: 0x9af2c9ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bfd: skipped 000397 1787549953.687657 punch from 0x000de2 to 0x00e26b (0xd48a bytes) 000398 1787549953.735206 mapwrite 0x01ffed thru 0x02f7b5 (0xf7c9 bytes) 000399 1787549953.797626 trunc from 0x01ac78 to 0x01eaa2 (0x3e2b bytes) 000400 1787549953.808770 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 12: 0x9af2c9ec (stale) Mirror 13: 0x9af2c9ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bfd): Mirror 12: 0x9af2c9ec (stale) Mirror 13: 0x9af2c9ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bfd: skipped 000401 1787549954.235161 punch from 0x0174aa to 0x019e37 (0x298e bytes) 000402 1787549954.241958 trunc from 0x039bfd to 0x006b0f (0x330ed bytes) 000403 1787549954.261270 read 0x00150e thru 0x003bd8 (0x26cb bytes) 000404 1787549954.272802 write 0x007440 thru 0x008df7 (0x19b8 bytes) 000405 1787549954.273966 mapwrite 0x004bd7 thru 0x00e07e (0x94a8 bytes) 000406 1787549954.338263 write 0x035f3c thru 0x03ffff (0xa0c4 bytes) 000407 1787549954.345730 MIRROR_RESYNC 3 mirrors 000408 1787549955.237985 fallocate 0x034ff0 thru 0x0369b8 (0x19c9 bytes) 000409 1787549955.287366 write 0x005935 thru 0x00edfe (0x94ca bytes) 000410 1787549955.316336 MIRROR_RESYNC 3 mirrors 000411 1787549956.224732 write 0x03f17a thru 0x03ffff (0xe86 bytes) 000412 1787549956.253952 mapread 0x038263 thru 0x03ffff (0x7d9d bytes) 000413 1787549956.280692 MIRROR_RESYNC 3 mirrors 000414 1787549957.170357 mapread 0x03dc97 thru 0x03ffff (0x2369 bytes) 000416 1787549957.198366 fallocate 0x0058e3 thru 0x00971c (0x3e3a bytes) 000417 1787549957.236967 write 0x0068c2 thru 0x00ea42 (0x8181 bytes) 000418 1787549957.254311 trunc from 0x040000 to 0x024fb2 (0x1b04d bytes) 000419 1787549957.310792 write 0x00e943 thru 0x017bf9 (0x92b7 bytes) 000420 1787549957.322203 trunc from 0x001ccc to 0x004655 (0x298a bytes) 000421 1787549957.328296 punch from 0x002112 to 0x0041d5 (0x20c4 bytes) 000422 1787549957.332692 read 0x003357 thru 0x009b4f (0x67f9 bytes) 000423 1787549957.342677 trunc from 0x024fb3 to 0x03f129 (0x1a177 bytes) 000424 1787549957.362719 trunc from 0x00fe08 to 0x014851 (0x4a4a bytes) 000426 1787549957.382768 read 0x00c3b3 thru 0x00dc53 (0x18a1 bytes) 000427 1787549957.393130 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f12a): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f12a): Mirror 12: 0x8611cde9 (stale) Mirror 13: 0x8611cde9 (stale) CRC-32 checksum value for chunk [0, 0x3f12a): Mirror 1: 0xb2d63c8e Mirror 14: 0xb2d63c8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f12a: skipped 000428 1787549958.444963 fallocate 0x00cb02 thru 0x00cd7f (0x27e bytes) 000430 1787549958.509946 trunc from 0x03f12a to 0x00a8a2 (0x34887 bytes) 000432 1787549958.575129 mapwrite 0x03cd22 thru 0x03ffff (0x32de bytes) 000433 1787549958.649314 read 0x01cc3a thru 0x02ca7d (0xfe44 bytes) 000434 1787549958.660127 read 0x02edea thru 0x02efdb (0x1f2 bytes) 000435 1787549958.685939 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa4a94f8 (stale) Mirror 13: 0xa4a94f8 (stale) Mirror 14: 0x84d9c8b8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x938291ca Mirror 15: 0x938291ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000436 1787549959.683581 trunc from 0x025333 to 0x02aa4d (0x571b bytes) 000438 1787549959.735245 trunc from 0x03f5f2 to 0x03ffff (0xa0e bytes) 000439 1787549959.751726 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa4a94f8 (stale) Mirror 13: 0xa4a94f8 (stale) Mirror 14: 0x84d9c8b8 (stale) Mirror 15: 0x938291ca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf783e23f Mirror 16: 0xf783e23f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000440 1787549960.925863 trunc from 0x040000 to 0x00d1ac (0x32e53 bytes) 000441 1787549960.992564 read 0x00b668 thru 0x00d1ac (0x1b45 bytes) 000442 1787549960.998814 write 0x02de97 thru 0x033ad5 (0x5c3f bytes) 000443 1787549961.001610 trunc from 0x02609d to 0x02e439 (0x839d bytes) 000444 1787549961.025631 trunc from 0x033ad6 to 0x02d79b (0x633a bytes) 000445 1787549961.049963 write 0x01c031 thru 0x01db0d (0x1add bytes) 000446 1787549961.064597 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d79c): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d79c): Mirror 12: 0xd6abf02b (stale) Mirror 13: 0xd6abf02b (stale) Mirror 14: 0xf960c4d8 (stale) Mirror 15: 0x13219338 (stale) Mirror 16: 0x13219338 (stale) CRC-32 checksum value for chunk [0, 0x2d79c): Mirror 1: 0x15ef003d Mirror 17: 0x15ef003d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d79c: skipped 000447 1787549962.386901 mapread 0x0210d5 thru 0x026dca (0x5cf6 bytes) 000448 1787549962.418129 write 0x02bc28 thru 0x02cd5a (0x1133 bytes) 000449 1787549962.474518 mapwrite 0x034ffc thru 0x0369a2 (0x19a7 bytes) 000450 1787549962.518857 mapwrite 0x03e337 thru 0x03ffff (0x1cc9 bytes) 000451 1787549962.576956 mapread 0x00ad06 thru 0x00c028 (0x1323 bytes) 000452 1787549962.586008 read 0x010819 thru 0x01d74c (0xcf34 bytes) 000453 1787549962.594373 punch from 0x00dc6d to 0x0163eb (0x877f bytes) 000454 1787549962.603348 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xa4a94f8 (stale) Mirror 14: 0x84d9c8b8 (stale) Mirror 15: 0x938291ca (stale) Mirror 16: 0xf783e23f (stale) Mirror 17: 0xf069f487 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xa4a94f8 (stale) Mirror 14: 0x84d9c8b8 (stale) Mirror 15: 0x938291ca (stale) Mirror 16: 0xf783e23f (stale) Mirror 17: 0xf069f487 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1787549963.091151 trunc from 0x03dddd to 0x03ffff (0x2223 bytes) 000456 1787549963.103660 MIRROR_RESYNC 6 mirrors 000457 1787549965.404074 MIRROR_EXTEND to 7 mirrors 000458 1787549966.267106 read 0x00e30d thru 0x017438 (0x912c bytes) 000459 1787549966.277025 mapread 0x02cb93 thru 0x03786a (0xacd8 bytes) 000460 1787549966.307012 trunc from 0x040000 to 0x028c6d (0x17392 bytes) 000461 1787549966.367028 fallocate 0x01a517 thru 0x028d79 (0xe863 bytes) 000462 1787549966.373661 mapwrite 0x035339 thru 0x03ffff (0xacc7 bytes) 000463 1787549966.439261 mapread 0x008331 thru 0x008e68 (0xb38 bytes) 000464 1787549966.455859 mapread 0x027824 thru 0x03048c (0x8c69 bytes) 000465 1787549966.474317 fallocate 0x00e5a7 thru 0x00fc6d (0x16c7 bytes) 000466 1787549966.483814 write 0x01606f thru 0x01798d (0x191f bytes) 000467 1787549966.506931 mapwrite 0x031ba6 thru 0x036789 (0x4be4 bytes) 000468 1787549966.530320 mapwrite 0x0243df thru 0x0268f2 (0x2514 bytes) 000469 1787549966.557018 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7e89c67d (stale) Mirror 14: 0x7e89c67d (stale) Mirror 15: 0x7e89c67d (stale) Mirror 16: 0x7e89c67d (stale) Mirror 17: 0x7e89c67d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7e89c67d (stale) Mirror 14: 0x7e89c67d (stale) Mirror 15: 0x7e89c67d (stale) Mirror 16: 0x7e89c67d (stale) Mirror 17: 0x7e89c67d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000470 1787549967.081031 trunc from 0x040000 to 0x004eca (0x3b135 bytes) 000471 1787549967.112575 trunc from 0x004ecb to 0x0335cd (0x2e703 bytes) 000472 1787549967.130621 trunc from 0x0089fa to 0x00bdaa (0x33b1 bytes) 000474 1787549967.146332 punch from 0x01d907 to 0x02611c (0x8816 bytes) 000475 1787549967.152710 punch from 0x01b828 to 0x01e779 (0x2f52 bytes) 000476 1787549967.160303 mapwrite 0x01fe28 thru 0x02989e (0x9a77 bytes) 000477 1787549967.236674 trunc from 0x0335ce to 0x009e21 (0x297ac bytes) 000478 1787549967.272277 read 0x006019 thru 0x008126 (0x210e bytes) 000479 1787549967.279033 punch from 0x00580c to 0x00720a (0x19ff bytes) 000480 1787549967.286587 mapwrite 0x001f0a thru 0x010a00 (0xeaf7 bytes) 000481 1787549967.392930 write 0x009249 thru 0x010aeb (0x78a3 bytes) 000483 1787549967.396994 trunc from 0x010aec to 0x039392 (0x288a7 bytes) 000484 1787549967.425457 mapread 0x0368cb thru 0x036ceb (0x421 bytes) 000485 1787549967.439226 fallocate 0x008cde thru 0x011cad (0x8fd0 bytes) 000486 1787549967.450923 trunc from 0x039393 to 0x008b6d (0x30825 bytes) 000487 1787549967.512816 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x8b6e): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b6e): Mirror 13: 0xadb51e2d (stale) Mirror 14: 0xadb51e2d (stale) Mirror 15: 0xadb51e2d (stale) Mirror 16: 0xadb51e2d (stale) Mirror 17: 0xadb51e2d (stale) CRC-32 checksum value for chunk [0, 0x8b6e): Mirror 1: 0x6d033e05 Mirror 18: 0x6d033e05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b6e: skipped 000488 1787549968.451262 fallocate 0x0141ac thru 0x01816b (0x3fc0 bytes) 000489 1787549968.494022 mapread 0x0057e4 thru 0x008b6d (0x338a bytes) 000490 1787549968.515236 mapread 0x001f58 thru 0x008b6d (0x6c16 bytes) 000491 1787549968.516497 fallocate 0x000b9b thru 0x005c78 (0x50de bytes) 000492 1787549968.521481 trunc from 0x00470a to 0x008b6d (0x4464 bytes) 000493 1787549968.527272 trunc from 0x008b6e to 0x03a4d2 (0x31965 bytes) 000494 1787549968.580021 read 0x0106af thru 0x019e63 (0x97b5 bytes) 000496 1787549968.593242 fallocate 0x004e2c thru 0x008bcf (0x3da4 bytes) 000497 1787549968.599063 read 0x02b60d thru 0x030094 (0x4a88 bytes) 000498 1787549968.636534 punch from 0x0044db to 0x012153 (0xdc79 bytes) 000499 1787549968.643556 fallocate 0x01e446 thru 0x022488 (0x4043 bytes) 000500 1787549968.662090 trunc from 0x02a197 to 0x0340ef (0x9f59 bytes) 000501 1787549968.672379 mapwrite 0x01cf7c thru 0x0248b8 (0x793d bytes) 000502 1787549968.710299 trunc from 0x03a4d3 to 0x011af8 (0x289da bytes) 000503 1787549968.729681 read 0x0053a8 thru 0x011af8 (0xc751 bytes) 000504 1787549968.748838 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11af9): Mirror 13: 0x55f97672 (stale) Mirror 14: 0x55f97672 (stale) Mirror 15: 0x55f97672 (stale) Mirror 16: 0x55f97672 (stale) Mirror 17: 0x55f97672 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11af9): Mirror 13: 0x55f97672 (stale) Mirror 14: 0x55f97672 (stale) Mirror 15: 0x55f97672 (stale) Mirror 16: 0x55f97672 (stale) Mirror 17: 0x55f97672 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11af9: skipped 000505 1787549969.248739 mapread 0x005db2 thru 0x007e0e (0x205d bytes) 000506 1787549969.293643 mapwrite 0x013c05 thru 0x01ecc7 (0xb0c3 bytes) 000507 1787549969.467243 punch from 0x00ef96 to 0x019962 (0xa9cd bytes) 000508 1787549969.480034 trunc from 0x01ecc8 to 0x039aa2 (0x1addb bytes) 000510 1787549969.508678 read 0x01e091 thru 0x01f90a (0x187a bytes) 000511 1787549969.514448 mapwrite 0x002d2b thru 0x007010 (0x42e6 bytes) 000512 1787549969.546614 trunc from 0x01d0fa to 0x01e3b8 (0x12bf bytes) 000513 1787549969.552369 punch from 0x01e4d7 to 0x0243bb (0x5ee5 bytes) 000514 1787549969.561684 trunc from 0x039aa3 to 0x03e396 (0x48f4 bytes) 000515 1787549969.584726 write 0x03f495 thru 0x03ffff (0xb6b bytes) 000516 1787549969.588167 read 0x003a9a thru 0x007555 (0x3abc bytes) 000517 1787549969.589962 write 0x038de0 thru 0x03ffff (0x7220 bytes) 000518 1787549969.607821 trunc from 0x040000 to 0x013852 (0x2c7ad bytes) 000519 1787549969.636474 mapread 0x00ef1d thru 0x013852 (0x4936 bytes) 000520 1787549969.638362 trunc from 0x013853 to 0x017662 (0x3e10 bytes) 000521 1787549969.660532 punch from 0x0098a7 to 0x016133 (0xc88d bytes) 000522 1787549969.672007 punch from 0x007234 to 0x009ed5 (0x2ca2 bytes) 000523 1787549969.682772 punch from 0x009bc2 to 0x00d61e (0x3a5d bytes) 000524 1787549969.691448 fallocate 0x033d03 thru 0x039947 (0x5c45 bytes) 000525 1787549969.699537 punch from 0x00984f to 0x00fea1 (0x6653 bytes) 000526 1787549969.711731 MIRROR_RESYNC 6 mirrors 000527 1787549970.993756 mapwrite 0x030d33 thru 0x036ab7 (0x5d85 bytes) 000528 1787549971.085812 mapwrite 0x026afa thru 0x035063 (0xe56a bytes) 000529 1787549971.139161 trunc from 0x036ab8 to 0x0308ce (0x61e9 bytes) 000530 1787549971.183432 punch from 0x0201b5 to 0x028e9c (0x8ce8 bytes) 000531 1787549971.198361 read 0x01b433 thru 0x02a5ca (0xf198 bytes) 000533 1787549971.216209 mapwrite 0x028ebd thru 0x036f6f (0xe0b3 bytes) 000534 1787549971.310209 read 0x02f493 thru 0x0303b6 (0xf24 bytes) 000535 1787549971.311686 mapwrite 0x03aba3 thru 0x03ffff (0x545d bytes) 000537 1787549971.409366 mapread 0x035ea9 thru 0x03ffff (0xa157 bytes) 000538 1787549971.411352 read 0x013cae thru 0x018de1 (0x5134 bytes) 000539 1787549971.413818 mapwrite 0x03fadf thru 0x03ffff (0x521 bytes) 000540 1787549971.434153 mapwrite 0x00597b thru 0x00b703 (0x5d89 bytes) 000541 1787549971.466440 trunc from 0x02f3d5 to 0x03864b (0x9277 bytes) 000542 1787549971.471484 read 0x02a460 thru 0x0351df (0xad80 bytes) 000543 1787549971.487628 write 0x00c411 thru 0x01bd40 (0xf930 bytes) 000544 1787549971.497868 read 0x0045cc thru 0x009bb1 (0x55e6 bytes) 000545 1787549971.499451 mapwrite 0x02cecb thru 0x02f9cd (0x2b03 bytes) 000547 1787549971.535742 write 0x01a935 thru 0x01dfe0 (0x36ac bytes) 000548 1787549971.538228 write 0x01ac55 thru 0x0271ec (0xc598 bytes) 000549 1787549971.540797 write 0x01d6f8 thru 0x02c9ad (0xf2b6 bytes) 000550 1787549971.561751 MIRROR_RESYNC 6 mirrors 000551 1787549972.939661 write 0x01268d thru 0x01b67a (0x8fee bytes) 000552 1787549973.010321 mapread 0x02fe89 thru 0x031da8 (0x1f20 bytes) 000553 1787549973.057449 punch from 0x023feb to 0x0302ba (0xc2d0 bytes) 000554 1787549973.065771 write 0x0283ca thru 0x033d20 (0xb957 bytes) 000555 1787549973.075958 read 0x008a49 thru 0x0150f2 (0xc6aa bytes) 000556 1787549973.094591 mapread 0x023e46 thru 0x02a508 (0x66c3 bytes) 000557 1787549973.142202 mapwrite 0x03e62b thru 0x03ffff (0x19d5 bytes) 000558 1787549973.200498 write 0x00645c thru 0x00e3f4 (0x7f99 bytes) 000559 1787549973.221711 mapread 0x010eb1 thru 0x019e50 (0x8fa0 bytes) 000560 1787549973.226951 write 0x02c0d8 thru 0x0362d8 (0xa201 bytes) 000562 1787549973.235640 mapwrite 0x01a759 thru 0x026695 (0xbf3d bytes) 000564 1787549973.279536 write 0x01e4d2 thru 0x01eedd (0xa0c bytes) 000565 1787549973.280581 trunc from 0x013956 to 0x0181a9 (0x4854 bytes) 000566 1787549973.288624 fallocate 0x02c737 thru 0x02fcb6 (0x3580 bytes) 000567 1787549973.293732 write 0x03afee thru 0x03ffff (0x5012 bytes) 000568 1787549973.295005 read 0x0170eb thru 0x01e0db (0x6ff1 bytes) 000569 1787549973.303379 trunc from 0x02d0e5 to 0x03adc5 (0xdce1 bytes) 000570 1787549973.349985 trunc from 0x01200a to 0x017fc1 (0x5fb8 bytes) 000571 1787549973.354525 mapwrite 0x025ff6 thru 0x035eee (0xfef9 bytes) 000572 1787549973.450655 MIRROR_RESYNC 6 mirrors 000573 1787549975.306180 trunc from 0x040000 to 0x031a04 (0xe5fb bytes) 000574 1787549975.358133 trunc from 0x005e12 to 0x0148b0 (0xea9f bytes) 000575 1787549975.377291 read 0x02cf1d thru 0x02ea87 (0x1b6b bytes) 000576 1787549975.391947 trunc from 0x031a05 to 0x027f79 (0x9a8b bytes) 000577 1787549975.442159 mapread 0x02294c thru 0x027f79 (0x562e bytes) 000578 1787549975.480861 mapwrite 0x03f882 thru 0x03ffff (0x77e bytes) 000579 1787549975.542334 mapread 0x02bde0 thru 0x02f52b (0x374c bytes) 000580 1787549975.559999 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x2c1badcf (stale) Mirror 14: 0x2c1badcf (stale) Mirror 15: 0x2c1badcf (stale) Mirror 16: 0x2c1badcf (stale) Mirror 17: 0x2c1badcf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x682d66a2 Mirror 18: 0x682d66a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000581 1787549976.812958 mapread 0x030dcd thru 0x039450 (0x8684 bytes) 000582 1787549976.886669 mapwrite 0x0362a5 thru 0x03ffff (0x9d5b bytes) 000583 1787549977.010752 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x2c1badcf (stale) Mirror 14: 0x2c1badcf (stale) Mirror 15: 0x2c1badcf (stale) Mirror 16: 0x2c1badcf (stale) Mirror 17: 0x2c1badcf (stale) Mirror 18: 0x682d66a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd58de2d Mirror 19: 0xdd58de2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1787549978.268705 read 0x00a8b5 thru 0x016e15 (0xc561 bytes) 000585 1787549978.308599 read 0x0357c5 thru 0x03ffff (0xa83b bytes) 000586 1787549978.361257 write 0x03329f thru 0x03bd10 (0x8a72 bytes) 000587 1787549978.443754 punch from 0x0325b2 to 0x03a52f (0x7f7e bytes) 000588 1787549978.473520 mapwrite 0x01ac34 thru 0x01f0d2 (0x449f bytes) 000590 1787549978.543506 mapwrite 0x00947b thru 0x00ef19 (0x5a9f bytes) 000591 1787549978.596151 MIRROR_RESYNC 8 mirrors 000593 1787549981.393141 read 0x00423f thru 0x01121b (0xcfdd bytes) 000594 1787549981.413170 fallocate 0x015c41 thru 0x021495 (0xb855 bytes) 000596 1787549981.463357 trunc from 0x040000 to 0x006ce1 (0x3931e bytes) 000597 1787549981.497872 trunc from 0x002543 to 0x006ce1 (0x479f bytes) 000598 1787549981.508197 trunc from 0x006ce2 to 0x01c32d (0x1564c bytes) 000599 1787549981.527835 mapread 0x00dace thru 0x0121d1 (0x4704 bytes) 000600 1787549981.561309 mapwrite 0x014918 thru 0x024703 (0xfdec bytes) 000601 1787549981.666466 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24704): Mirror 14: 0xb44975aa (stale) Mirror 15: 0xb44975aa (stale) Mirror 16: 0xb44975aa (stale) Mirror 17: 0xb44975aa (stale) Mirror 18: 0xb44975aa (stale) Mirror 19: 0xb44975aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24704): Mirror 14: 0xb44975aa (stale) Mirror 15: 0xb44975aa (stale) Mirror 16: 0xb44975aa (stale) Mirror 17: 0xb44975aa (stale) Mirror 18: 0xb44975aa (stale) Mirror 19: 0xb44975aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24704: skipped 000602 1787549982.190967 trunc from 0x011a28 to 0x018326 (0x68ff bytes) 000603 1787549982.205613 trunc from 0x00a6a3 to 0x0149bf (0xa31d bytes) 000604 1787549982.211197 punch from 0x010c38 to 0x01e9d0 (0xdd99 bytes) 000605 1787549982.226854 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24704): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24704): Mirror 14: 0xb44975aa (stale) Mirror 15: 0xb44975aa (stale) Mirror 16: 0xb44975aa (stale) Mirror 17: 0xb44975aa (stale) Mirror 18: 0xb44975aa (stale) Mirror 19: 0xb44975aa (stale) CRC-32 checksum value for chunk [0, 0x24704): Mirror 1: 0x350f299 Mirror 20: 0x350f299 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24704: skipped 000607 1787549983.493266 read 0x0082ba thru 0x014a87 (0xc7ce bytes) 000608 1787549983.510078 trunc from 0x024704 to 0x01970c (0xaff7 bytes) 000609 1787549983.585731 trunc from 0x00768f to 0x00a79e (0x3110 bytes) 000610 1787549983.599136 mapread 0x0183d7 thru 0x01970c (0x1336 bytes) 000611 1787549983.622653 trunc from 0x01970d to 0x0055a9 (0x14163 bytes) 000612 1787549983.662747 mapwrite 0x00859e thru 0x017dd9 (0xf83c bytes) 000613 1787549983.773443 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17dda): Mirror 14: 0x7f4e76c3 (stale) Mirror 15: 0x7f4e76c3 (stale) Mirror 16: 0x7f4e76c3 (stale) Mirror 17: 0x7f4e76c3 (stale) Mirror 18: 0x7f4e76c3 (stale) Mirror 20: 0x74d165fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17dda): Mirror 14: 0x7f4e76c3 (stale) Mirror 15: 0x7f4e76c3 (stale) Mirror 16: 0x7f4e76c3 (stale) Mirror 17: 0x7f4e76c3 (stale) Mirror 18: 0x7f4e76c3 (stale) Mirror 20: 0x74d165fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17dda: skipped 000614 1787549984.381732 mapwrite 0x01efc8 thru 0x029fa2 (0xafdb bytes) 000615 1787549984.457678 trunc from 0x029fa3 to 0x02497c (0x5626 bytes) 000616 1787549984.490858 write 0x03b401 thru 0x03ffff (0x4bff bytes) 000617 1787549984.492564 trunc from 0x040000 to 0x001ad1 (0x3e52e bytes) 000618 1787549984.520230 mapwrite 0x00d2dd thru 0x014da5 (0x7ac9 bytes) 000619 1787549984.564129 fallocate 0x022707 thru 0x02e614 (0xbf0e bytes) 000620 1787549984.570978 trunc from 0x025d1d to 0x02c9e9 (0x6ccd bytes) 000622 1787549984.576311 mapwrite 0x03fba8 thru 0x03ffff (0x458 bytes) 000623 1787549984.613475 read 0x015740 thru 0x020307 (0xabc8 bytes) 000624 1787549984.643458 mapwrite 0x032227 thru 0x033c1c (0x19f6 bytes) 000625 1787549984.685608 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x47db32a2 (stale) Mirror 15: 0x47db32a2 (stale) Mirror 17: 0x47db32a2 (stale) Mirror 18: 0x47db32a2 (stale) Mirror 20: 0xc9dff77b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x47db32a2 (stale) Mirror 15: 0x47db32a2 (stale) Mirror 17: 0x47db32a2 (stale) Mirror 18: 0x47db32a2 (stale) Mirror 20: 0xc9dff77b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000626 1787549985.227919 read 0x010411 thru 0x010468 (0x58 bytes) 000627 1787549985.234339 trunc from 0x040000 to 0x0072b1 (0x38d4e bytes) 000629 1787549985.264353 write 0x01bc5a thru 0x02acbc (0xf063 bytes) 000630 1787549985.268136 fallocate 0x01bd37 thru 0x0282eb (0xc5b5 bytes) 000631 1787549985.283726 trunc from 0x0284e3 to 0x02acbc (0x27da bytes) 000632 1787549985.322445 write 0x020347 thru 0x025f1a (0x5bd4 bytes) 000633 1787549985.323669 mapwrite 0x00a073 thru 0x00bb8b (0x1b19 bytes) 000634 1787549985.387846 punch from 0x002819 to 0x002ae5 (0x2cd bytes) 000635 1787549985.416987 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2acbd): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acbd): Mirror 14: 0x7ac937ff (stale) Mirror 15: 0x7ac937ff (stale) Mirror 17: 0x7ac937ff (stale) Mirror 18: 0x7ac937ff (stale) Mirror 20: 0x1510a809 (stale) CRC-32 checksum value for chunk [0, 0x2acbd): Mirror 1: 0xbbb96c98 Mirror 21: 0xbbb96c98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acbd: skipped 000636 1787549986.574000 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2acbd): Mirror 14: 0x7ac937ff (stale) Mirror 15: 0x7ac937ff (stale) Mirror 17: 0x7ac937ff (stale) Mirror 18: 0x7ac937ff (stale) Mirror 20: 0x1510a809 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acbd): Mirror 14: 0x7ac937ff (stale) Mirror 15: 0x7ac937ff (stale) Mirror 17: 0x7ac937ff (stale) Mirror 18: 0x7ac937ff (stale) Mirror 20: 0x1510a809 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acbd: skipped 000637 1787549987.149277 read 0x004759 thru 0x010fa7 (0xc84f bytes) 000638 1787549987.163342 punch from 0x0211d6 to 0x02ab2b (0x9956 bytes) 000639 1787549987.184362 trunc from 0x00f412 to 0x0167a8 (0x7397 bytes) 000640 1787549987.194051 mapwrite 0x01a94d thru 0x02a8b7 (0xff6b bytes) 000642 1787549987.350293 punch from 0x00f49b to 0x01c191 (0xccf7 bytes) 000643 1787549987.358694 mapwrite 0x00dd10 thru 0x017db2 (0xa0a3 bytes) 000644 1787549987.468820 punch from 0x00a84c to 0x00eec5 (0x467a bytes) 000645 1787549987.475620 trunc from 0x02acbd to 0x039e1b (0xf15f bytes) 000646 1787549987.543670 write 0x03d1e7 thru 0x03ffff (0x2e19 bytes) 000647 1787549987.544967 write 0x00df23 thru 0x017aa9 (0x9b87 bytes) 000648 1787549987.554517 write 0x00dc25 thru 0x01ae01 (0xd1dd bytes) 000649 1787549987.556571 mapread 0x01ea1c thru 0x02c2aa (0xd88f bytes) 000650 1787549987.571759 mapwrite 0x00f76a thru 0x011779 (0x2010 bytes) 000651 1787549987.620036 MIRROR_RESYNC 6 mirrors 000652 1787549990.492879 trunc from 0x00e306 to 0x01129c (0x2f97 bytes) 000653 1787549990.552215 trunc from 0x040000 to 0x02d9ea (0x12615 bytes) 000654 1787549990.595874 punch from 0x02be51 to 0x02d9ea (0x1b9a bytes) 000655 1787549990.602700 punch from 0x02894f to 0x02d9ea (0x509c bytes) 000656 1787549990.611349 write 0x031c7b thru 0x03b56a (0x98f0 bytes) 000657 1787549990.614242 write 0x03d8bd thru 0x03e47e (0xbc2 bytes) 000658 1787549990.616133 trunc from 0x03e47f to 0x014431 (0x2a04d bytes) 000659 1787549990.647375 punch from 0x00b3dd to 0x00d288 (0x1eac bytes) 000660 1787549990.655566 read 0x00d084 thru 0x013bb4 (0x6b31 bytes) 000661 1787549990.682234 mapwrite 0x02832f thru 0x02857f (0x251 bytes) 000662 1787549990.732989 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28580): Mirror 14: 0x453c7f67 (stale) Mirror 15: 0x453c7f67 (stale) Mirror 17: 0x453c7f67 (stale) Mirror 20: 0x453c7f67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28580): Mirror 14: 0x453c7f67 (stale) Mirror 15: 0x453c7f67 (stale) Mirror 17: 0x453c7f67 (stale) Mirror 20: 0x453c7f67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28580: skipped 000663 1787549991.268295 write 0x025c13 thru 0x0303bd (0xa7ab bytes) 000665 1787549991.307280 mapwrite 0x0098f4 thru 0x00a924 (0x1031 bytes) 000666 1787549991.378555 trunc from 0x0303be to 0x01195c (0x1ea61 bytes) 000667 1787549991.415284 trunc from 0x01195d to 0x01c452 (0xaaf6 bytes) 000668 1787549991.455099 MIRROR_RESYNC 5 mirrors 000670 1787549993.284291 punch from 0x004920 to 0x00aba6 (0x6287 bytes) 000671 1787549993.353283 punch from 0x0158a9 to 0x01c452 (0x6baa bytes) 000672 1787549993.362073 mapwrite 0x0216fb thru 0x02d575 (0xbe7b bytes) 000673 1787549993.433881 trunc from 0x01a18f to 0x028fee (0xee60 bytes) 000674 1787549993.446624 punch from 0x006cd6 to 0x016c5f (0xff8a bytes) 000676 1787549993.455677 punch from 0x024445 to 0x026b5e (0x271a bytes) 000677 1787549993.481217 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d576): Mirror 14: 0xbe22465d (stale) Mirror 15: 0xbe22465d (stale) Mirror 17: 0xbe22465d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d576): Mirror 14: 0xbe22465d (stale) Mirror 15: 0xbe22465d (stale) Mirror 17: 0xbe22465d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d576: skipped 000678 1787549994.102996 mapwrite 0x00545b thru 0x00d0e5 (0x7c8b bytes) 000679 1787549994.195313 punch from 0x0042b5 to 0x00e897 (0xa5e3 bytes) 000680 1787549994.208855 read 0x00d479 thru 0x00fb12 (0x269a bytes) 000681 1787549994.219454 mapwrite 0x0167e7 thru 0x01fa8a (0x92a4 bytes) 000682 1787549994.289759 punch from 0x024b6b to 0x02d575 (0x8a0b bytes) 000683 1787549994.304369 write 0x01d6c9 thru 0x029eb7 (0xc7ef bytes) 000684 1787549994.330371 read 0x017e16 thru 0x018c78 (0xe63 bytes) 000685 1787549994.331772 write 0x01c2c6 thru 0x028d7d (0xcab8 bytes) 000686 1787549994.339512 mapread 0x01b2ff thru 0x01b6e3 (0x3e5 bytes) 000687 1787549994.341570 mapwrite 0x00c8ae thru 0x011d5b (0x54ae bytes) 000689 1787549994.379478 fallocate 0x01570c thru 0x01951f (0x3e14 bytes) 000690 1787549994.383513 fallocate 0x03c2d5 thru 0x03ffff (0x3d2b bytes) 000691 1787549994.387232 punch from 0x0059e3 to 0x0155a0 (0xfbbe bytes) 000692 1787549994.392793 punch from 0x00c9fb to 0x0130f1 (0x66f7 bytes) 000695 1787549994.401704 punch from 0x010318 to 0x019084 (0x8d6d bytes) 000697 1787549994.412865 trunc from 0x00e472 to 0x0132cd (0x4e5c bytes) 000698 1787549994.419621 write 0x03d23f thru 0x03fdf4 (0x2bb6 bytes) 000699 1787549994.421047 read 0x031292 thru 0x03a277 (0x8fe6 bytes) 000700 1787549994.471653 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fdf5): Mirror 15: 0xd7ae5fdc (stale) Mirror 17: 0xd7ae5fdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdf5): Mirror 15: 0xd7ae5fdc (stale) Mirror 17: 0xd7ae5fdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdf5: skipped 000701 1787549994.973992 mapread 0x00e140 thru 0x01ae09 (0xccca bytes) 000702 1787549995.017810 trunc from 0x0093a6 to 0x011d60 (0x89bb bytes) 000703 1787549995.022588 trunc from 0x03fdf5 to 0x014bac (0x2b248 bytes) 000704 1787549995.042731 mapwrite 0x032076 thru 0x03f3c5 (0xd350 bytes) 000705 1787549995.128783 read 0x021fa2 thru 0x029df5 (0x7e54 bytes) 000706 1787549995.162580 fallocate 0x02e189 thru 0x033eae (0x5d26 bytes) 000707 1787549995.168227 punch from 0x024253 to 0x027fbb (0x3d69 bytes) 000708 1787549995.177378 read 0x01a623 thru 0x02661b (0xbff9 bytes) 000709 1787549995.187082 write 0x0213a6 thru 0x02d90d (0xc568 bytes) 000710 1787549995.188518 punch from 0x0211b9 to 0x02d454 (0xc29c bytes) 000711 1787549995.208819 read 0x03d104 thru 0x03f3c5 (0x22c2 bytes) 000712 1787549995.209343 read 0x01ce32 thru 0x023b5f (0x6d2e bytes) 000713 1787549995.221206 punch from 0x00ffbf to 0x0112cf (0x1311 bytes) 000715 1787549995.238087 MIRROR_RESYNC 3 mirrors 000718 1787549996.593648 mapread 0x01b704 thru 0x028d8b (0xd688 bytes) 000719 1787549996.637303 punch from 0x03567e to 0x03eada (0x945d bytes) 000720 1787549996.686829 fallocate 0x002692 thru 0x00a298 (0x7c07 bytes) 000721 1787549996.694358 write 0x014aaa thru 0x021442 (0xc999 bytes) 000723 1787549996.711737 punch from 0x01b900 to 0x024221 (0x8922 bytes) 000724 1787549996.747573 punch from 0x00c09b to 0x00e2f2 (0x2258 bytes) 000725 1787549996.757083 read 0x00c2f2 thru 0x018b27 (0xc836 bytes) 000726 1787549996.817654 read 0x009d64 thru 0x0178ce (0xdb6b bytes) 000727 1787549996.886754 trunc from 0x0083d3 to 0x0107cc (0x83fa bytes) 000728 1787549996.896826 trunc from 0x02833d to 0x02b6c9 (0x338d bytes) 000729 1787549996.914797 mapwrite 0x01708f thru 0x01b953 (0x48c5 bytes) 000730 1787549996.965835 trunc from 0x0077f7 to 0x00c708 (0x4f12 bytes) 000731 1787549996.985529 punch from 0x016675 to 0x025fd6 (0xf962 bytes) 000734 1787549997.006062 write 0x026237 thru 0x027a30 (0x17fa bytes) 000735 1787549997.012772 trunc from 0x0290ec to 0x030c95 (0x7baa bytes) 000736 1787549997.024071 trunc from 0x03f3c6 to 0x03e567 (0xe5e bytes) 000737 1787549997.049926 trunc from 0x03303b to 0x03507b (0x2041 bytes) 000738 1787549997.055648 read 0x001d57 thru 0x003b5e (0x1e08 bytes) 000739 1787549997.056910 mapwrite 0x0102e6 thru 0x014a50 (0x476b bytes) 000740 1787549997.096109 mapread 0x012116 thru 0x01334b (0x1236 bytes) 000741 1787549997.097911 trunc from 0x03e568 to 0x015909 (0x28c5e bytes) 000742 1787549997.125911 trunc from 0x01590a to 0x00b24d (0xa6bc bytes) 000743 1787549997.149802 punch from 0x003b98 to 0x004f6c (0x13d5 bytes) 000744 1787549997.163955 MIRROR_RESYNC 3 mirrors 000745 1787549998.158320 write 0x00dff5 thru 0x013c22 (0x5c2e bytes) 000746 1787549998.220780 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13c23): Mirror 15: 0xa4d4f01f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13c23): Mirror 15: 0xa4d4f01f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13c23: skipped 000747 1787549998.689080 mapwrite 0x02195d thru 0x029a54 (0x80f8 bytes) 000748 1787549998.780511 trunc from 0x028487 to 0x029a54 (0x15ce bytes) 000749 1787549998.794208 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000751 1787549998.924789 punch from 0x01411c to 0x01bf1e (0x7e03 bytes) 000752 1787549998.929165 trunc from 0x007242 to 0x00abc3 (0x3982 bytes) 000753 1787549998.939949 mapread 0x012369 thru 0x019320 (0x6fb8 bytes) 000755 1787549998.964674 MIRROR_EXTEND to 2 mirrors 000756 1787549999.883346 MIRROR_RESYNC 2 mirrors 000757 1787550000.288546 MIRROR_RESYNC 2 mirrors 000758 1787550000.577464 punch from 0x000c5c to 0x005f1c (0x52c1 bytes) 000759 1787550000.629860 trunc from 0x023c42 to 0x029a54 (0x5e13 bytes) 000760 1787550000.646669 read 0x000efc thru 0x00c74e (0xb853 bytes) 000761 1787550000.666551 write 0x012ea3 thru 0x01abc5 (0x7d23 bytes) 000762 1787550000.692300 write 0x03c0bf thru 0x03ffff (0x3f41 bytes) 000763 1787550000.693542 trunc from 0x017a7b to 0x01e994 (0x6f1a bytes) 000764 1787550000.706891 mapread 0x005e05 thru 0x00d7fb (0x79f7 bytes) 000765 1787550000.713251 punch from 0x01647b to 0x020171 (0x9cf7 bytes) 000766 1787550000.720462 write 0x00aa58 thru 0x00ecaf (0x4258 bytes) 000767 1787550000.734561 trunc from 0x040000 to 0x01b885 (0x2477a bytes) 000768 1787550000.756898 read 0x018815 thru 0x01b45c (0x2c48 bytes) 000769 1787550000.769913 read 0x0179ec thru 0x01b885 (0x3e9a bytes) 000770 1787550000.771139 mapread 0x01b4ae thru 0x01b885 (0x3d8 bytes) 000771 1787550000.772637 read 0x01633e thru 0x019ec1 (0x3b84 bytes) 000772 1787550000.774121 trunc from 0x01b886 to 0x012734 (0x9151 bytes) 000773 1787550000.789237 trunc from 0x012735 to 0x035985 (0x23251 bytes) 000774 1787550000.806004 trunc from 0x035986 to 0x030d89 (0x4bfc bytes) 000775 1787550000.821211 mapread 0x02feb6 thru 0x030d89 (0xed4 bytes) 000776 1787550000.841867 trunc from 0x030d8a to 0x03fbee (0xee65 bytes) 000777 1787550000.867368 mapread 0x0388c9 thru 0x03fbee (0x7326 bytes) 000778 1787550000.887671 trunc from 0x01937d to 0x01fc12 (0x6896 bytes) 000779 1787550000.905186 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 2: 0xabf35336 (stale) CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 1: 0x21421d87 Mirror 3: 0x21421d87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbef: skipped 000780 1787550001.962773 trunc from 0x03590c to 0x03fbee (0xa2e3 bytes) 000781 1787550002.025551 trunc from 0x00e421 to 0x01dfed (0xfbcd bytes) 000782 1787550002.035060 mapwrite 0x003fe6 thru 0x00afd2 (0x6fed bytes) 000783 1787550002.085636 trunc from 0x025f50 to 0x02ebfb (0x8cac bytes) 000784 1787550002.097699 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 2: 0xabf35336 (stale) Mirror 3: 0x21421d87 (stale) CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 1: 0x2bcb10e8 Mirror 4: 0x2bcb10e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbef: skipped 000785 1787550002.994033 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 3: 0x21421d87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 3: 0x21421d87 (stale) CRC-32 checksum value for chunk [0, 0x3fbef): Mirror 1: 0x2bcb10e8 Mirror 4: 0x2bcb10e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbef: skipped 000786 1787550004.150945 mapwrite 0x010e72 thru 0x017f51 (0x70e0 bytes) 000787 1787550004.372818 read 0x02b937 thru 0x038f98 (0xd662 bytes) 000788 1787550004.455830 read 0x017998 thru 0x025eac (0xe515 bytes) 000789 1787550004.474987 trunc from 0x03fbef to 0x036cdb (0x8f13 bytes) 000790 1787550004.579333 trunc from 0x036cdc to 0x000e9e (0x35e3d bytes) 000791 1787550004.605042 read 0x0006ea thru 0x000e9e (0x7b5 bytes) 000792 1787550004.609801 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe9f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe9f): Mirror 3: 0x74343274 (stale) Mirror 4: 0x74343274 (stale) CRC-32 checksum value for chunk [0, 0xe9f): Mirror 1: 0x74343274 Mirror 5: 0x74343274 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe9f: skipped 000793 1787550005.390008 punch from 0x000b47 to 0x000e9e (0x358 bytes) 000794 1787550005.482089 write 0x001189 thru 0x00a7eb (0x9663 bytes) 000795 1787550005.491759 read 0x000ce5 thru 0x004920 (0x3c3c bytes) 000796 1787550005.502500 fallocate 0x029c1f thru 0x02b1a4 (0x1586 bytes) 000797 1787550005.509635 trunc from 0x02b1a5 to 0x030223 (0x507f bytes) 000798 1787550005.586142 punch from 0x000b6c to 0x0045a5 (0x3a3a bytes) 000800 1787550005.610466 mapread 0x02a968 thru 0x02f433 (0x4acc bytes) 000801 1787550005.652165 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30224): Mirror 3: 0x3910c28a (stale) Mirror 5: 0x9c01c162 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30224): Mirror 3: 0x3910c28a (stale) Mirror 5: 0x9c01c162 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30224: skipped 000802 1787550006.066935 write 0x018398 thru 0x019d89 (0x19f2 bytes) 000803 1787550006.088294 trunc from 0x030224 to 0x020751 (0xfad2 bytes) 000804 1787550006.118030 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20752): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20752): Mirror 3: 0xfb4a795e (stale) Mirror 5: 0xb1d8a6a6 (stale) CRC-32 checksum value for chunk [0, 0x20752): Mirror 1: 0x477aac1 Mirror 6: 0x477aac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20752: skipped 000806 1787550007.251832 punch from 0x0155ff to 0x016e67 (0x1869 bytes) 000807 1787550007.294167 read 0x007c47 thru 0x00de8a (0x6244 bytes) 000808 1787550007.316785 trunc from 0x020752 to 0x03fc6d (0x1f51c bytes) 000809 1787550007.340688 trunc from 0x03fc6e to 0x0033da (0x3c893 bytes) 000810 1787550007.360234 trunc from 0x0033db to 0x02b8d4 (0x284fa bytes) 000811 1787550007.382698 trunc from 0x00bddb to 0x014c15 (0x8e3b bytes) 000812 1787550007.387975 trunc from 0x02b8d5 to 0x0095e3 (0x222f1 bytes) 000813 1787550007.405324 trunc from 0x004e50 to 0x0074c2 (0x2673 bytes) 000814 1787550007.411371 read 0x0077be thru 0x0095e3 (0x1e26 bytes) 000815 1787550007.438593 MIRROR_RESYNC 4 mirrors 000816 1787550008.375439 write 0x033d75 thru 0x03a98b (0x6c17 bytes) 000817 1787550008.415583 write 0x03f1d9 thru 0x03ffff (0xe27 bytes) 000818 1787550008.416810 trunc from 0x0292c2 to 0x033e36 (0xab75 bytes) 000819 1787550008.440229 write 0x00e6eb thru 0x01cef7 (0xe80d bytes) 000820 1787550008.450015 trunc from 0x040000 to 0x0109c3 (0x2f63c bytes) 000821 1787550008.475144 write 0x00a8d4 thru 0x0102ec (0x5a19 bytes) 000822 1787550008.481184 read 0x0075af thru 0x00cbed (0x563f bytes) 000823 1787550008.487135 fallocate 0x013ccc thru 0x017363 (0x3698 bytes) 000824 1787550008.493773 read 0x00873c thru 0x00aab4 (0x2379 bytes) 000825 1787550008.500483 MIRROR_RESYNC 4 mirrors 000826 1787550009.373038 MIRROR_EXTEND to 5 mirrors 000827 1787550010.245850 read 0x000e31 thru 0x00ebed (0xddbd bytes) 000828 1787550010.257287 trunc from 0x0099d4 to 0x0109c3 (0x6ff0 bytes) 000829 1787550010.291256 trunc from 0x002e58 to 0x0109c3 (0xdb6c bytes) 000830 1787550010.296246 write 0x00fcea thru 0x0145de (0x48f5 bytes) 000831 1787550010.305786 write 0x009fa7 thru 0x016b29 (0xcb83 bytes) 000832 1787550010.313099 fallocate 0x02ab0f thru 0x02d6a1 (0x2b93 bytes) 000833 1787550010.317851 trunc from 0x016b2a to 0x03b27f (0x24756 bytes) 000834 1787550010.336949 mapwrite 0x0132a0 thru 0x016ef2 (0x3c53 bytes) 000835 1787550010.364778 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b280): Mirror 3: 0x7bafe2d1 (stale) Mirror 5: 0x7bafe2d1 (stale) Mirror 7: 0x7bafe2d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b280): Mirror 3: 0x7bafe2d1 (stale) Mirror 5: 0x7bafe2d1 (stale) Mirror 7: 0x7bafe2d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b280: skipped 000836 1787550010.667031 mapwrite 0x0049ea thru 0x013095 (0xe6ac bytes) 000837 1787550010.737416 trunc from 0x006ae7 to 0x009268 (0x2782 bytes) 000838 1787550010.743510 trunc from 0x028cf0 to 0x02db54 (0x4e65 bytes) 000839 1787550010.750551 mapwrite 0x00de95 thru 0x01c655 (0xe7c1 bytes) 000840 1787550010.813821 mapread 0x02867f thru 0x035467 (0xcde9 bytes) 000841 1787550010.837553 punch from 0x00f436 to 0x01acc5 (0xb890 bytes) 000842 1787550010.843049 read 0x03a10a thru 0x03b27f (0x1176 bytes) 000843 1787550010.856083 fallocate 0x017779 thru 0x01d80c (0x6094 bytes) 000844 1787550010.861926 mapread 0x02aec4 thru 0x039ec5 (0xf002 bytes) 000845 1787550010.863827 write 0x0283f2 thru 0x02c91e (0x452d bytes) 000846 1787550010.872276 mapwrite 0x0229c5 thru 0x030a46 (0xe082 bytes) 000847 1787550010.898947 fallocate 0x010982 thru 0x011c33 (0x12b2 bytes) 000848 1787550010.909663 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b280): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b280): Mirror 3: 0x7bafe2d1 (stale) Mirror 5: 0x7bafe2d1 (stale) Mirror 7: 0x7bafe2d1 (stale) CRC-32 checksum value for chunk [0, 0x3b280): Mirror 1: 0x6e0b4fd5 Mirror 8: 0x6e0b4fd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b280: skipped 000850 1787550012.069917 mapread 0x00fdf9 thru 0x0159b9 (0x5bc1 bytes) 000851 1787550012.108671 punch from 0x024c72 to 0x02c25a (0x75e9 bytes) 000852 1787550012.154203 write 0x0068af thru 0x016618 (0xfd6a bytes) 000853 1787550012.178516 mapwrite 0x034c89 thru 0x0386f4 (0x3a6c bytes) 000854 1787550012.222813 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b280): Mirror 5: 0x7bafe2d1 (stale) Mirror 7: 0x7bafe2d1 (stale) Mirror 8: 0x6e0b4fd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b280): Mirror 5: 0x7bafe2d1 (stale) Mirror 7: 0x7bafe2d1 (stale) Mirror 8: 0x6e0b4fd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b280: skipped 000855 1787550012.733141 MIRROR_RESYNC 4 mirrors 000856 1787550014.620679 trunc from 0x01dc81 to 0x0248aa (0x6c2a bytes) 000857 1787550014.698980 mapread 0x027fd3 thru 0x03795d (0xf98b bytes) 000858 1787550014.748910 punch from 0x01bbd7 to 0x01c98c (0xdb6 bytes) 000859 1787550014.753289 mapread 0x004cf1 thru 0x012aeb (0xddfb bytes) 000860 1787550014.774940 fallocate 0x028f3e thru 0x032f50 (0xa013 bytes) 000861 1787550014.779332 mapwrite 0x01a678 thru 0x028e1e (0xe7a7 bytes) 000862 1787550014.830043 read 0x01c0e1 thru 0x022c4a (0x6b6a bytes) 000863 1787550014.830791 punch from 0x03381a to 0x037774 (0x3f5b bytes) 000864 1787550014.835945 punch from 0x01aa3a to 0x01cbed (0x21b4 bytes) 000865 1787550014.840822 trunc from 0x03b280 to 0x0381d2 (0x30ad bytes) 000866 1787550014.863857 mapread 0x00307f thru 0x007903 (0x4885 bytes) 000867 1787550014.864870 mapwrite 0x014735 thru 0x014c61 (0x52d bytes) 000868 1787550014.876296 trunc from 0x019736 to 0x0217ed (0x80b8 bytes) 000869 1787550014.881221 mapwrite 0x0152a1 thru 0x024e4a (0xfbaa bytes) 000870 1787550014.909790 trunc from 0x0381d3 to 0x01dec7 (0x1a30b bytes) 000871 1787550014.929082 write 0x03cd1f thru 0x03ffff (0x32e1 bytes) 000872 1787550014.930633 mapwrite 0x02bed8 thru 0x0370a9 (0xb1d2 bytes) 000873 1787550014.984893 trunc from 0x0204c9 to 0x02a6dd (0xa215 bytes) 000875 1787550014.993086 write 0x00481e thru 0x01153e (0xcd21 bytes) 000876 1787550014.999168 mapwrite 0x010ef5 thru 0x0148c6 (0x39d2 bytes) 000877 1787550015.091240 fallocate 0x007d8f thru 0x00e898 (0x6b0a bytes) 000878 1787550015.098507 mapwrite 0x0015a4 thru 0x00d16b (0xbbc8 bytes) 000879 1787550015.146675 mapwrite 0x01647f thru 0x023e98 (0xda1a bytes) 000880 1787550015.203803 write 0x030a3e thru 0x03a5db (0x9b9e bytes) 000882 1787550015.214699 write 0x0169a7 thru 0x023268 (0xc8c2 bytes) 000883 1787550015.225857 trunc from 0x037ebb to 0x039999 (0x1adf bytes) 000885 1787550015.246779 punch from 0x01149e to 0x016f86 (0x5ae9 bytes) 000886 1787550015.253082 trunc from 0x040000 to 0x039e26 (0x61d9 bytes) 000887 1787550015.282979 read 0x000f78 thru 0x00694e (0x59d7 bytes) 000888 1787550015.303315 write 0x00fe6e thru 0x01e493 (0xe626 bytes) 000889 1787550015.307330 fallocate 0x012929 thru 0x01fde2 (0xd4ba bytes) 000890 1787550015.324157 punch from 0x007bc8 to 0x00b6b5 (0x3aee bytes) 000891 1787550015.334113 fallocate 0x01d052 thru 0x025a47 (0x89f6 bytes) 000892 1787550015.347864 mapwrite 0x03e57e thru 0x03ffff (0x1a82 bytes) 000893 1787550015.408802 MIRROR_RESYNC 4 mirrors 000894 1787550016.943220 trunc from 0x040000 to 0x030bb6 (0xf449 bytes) 000896 1787550017.022246 mapread 0x015f82 thru 0x0245bc (0xe63b bytes) 000897 1787550017.081766 mapread 0x00330b thru 0x00b72a (0x8420 bytes) 000898 1787550017.109444 trunc from 0x030bb7 to 0x033707 (0x2b51 bytes) 000900 1787550017.132848 trunc from 0x002ccb to 0x0058bb (0x2bf1 bytes) 000901 1787550017.138549 read 0x02b707 thru 0x033707 (0x8001 bytes) 000902 1787550017.153773 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33708): Mirror 7: 0xe405dcf8 (stale) Mirror 8: 0xe405dcf8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33708): Mirror 7: 0xe405dcf8 (stale) Mirror 8: 0xe405dcf8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33708: skipped 000903 1787550017.680884 fallocate 0x011e0c thru 0x01abbf (0x8db4 bytes) 000904 1787550017.705928 read 0x01eafd thru 0x02eaae (0xffb2 bytes) 000905 1787550017.730919 trunc from 0x033708 to 0x02f24e (0x44b9 bytes) 000906 1787550017.782766 trunc from 0x000b5e to 0x007a4b (0x6eee bytes) 000907 1787550017.797778 mapread 0x02e93f thru 0x02f24e (0x910 bytes) 000909 1787550017.804028 mapwrite 0x03c8b8 thru 0x03da43 (0x118c bytes) 000910 1787550017.848043 trunc from 0x00be3f to 0x00ffde (0x41a0 bytes) 000911 1787550017.852167 read 0x00b15b thru 0x0135d8 (0x847e bytes) 000912 1787550017.862708 mapwrite 0x014e12 thru 0x01b0aa (0x6299 bytes) 000914 1787550017.886707 mapwrite 0x0148f8 thru 0x0188d1 (0x3fda bytes) 000915 1787550017.902460 fallocate 0x00f8ad thru 0x0151a6 (0x58fa bytes) 000916 1787550017.907460 mapread 0x0158d6 thru 0x022c4f (0xd37a bytes) 000917 1787550017.908634 punch from 0x013262 to 0x01d96e (0xa70d bytes) 000919 1787550017.915988 trunc from 0x0032f2 to 0x00934d (0x605c bytes) 000920 1787550017.938052 punch from 0x029486 to 0x031639 (0x81b4 bytes) 000921 1787550017.943201 mapwrite 0x00fe65 thru 0x0174c7 (0x7663 bytes) 000922 1787550018.008531 read 0x0044c0 thru 0x007ec2 (0x3a03 bytes) 000923 1787550018.019782 fallocate 0x015039 thru 0x01f48a (0xa452 bytes) 000924 1787550018.024229 read 0x025c7d thru 0x034c7b (0xefff bytes) 000925 1787550018.025723 mapread 0x0177b4 thru 0x02589f (0xe0ec bytes) 000926 1787550018.027032 mapwrite 0x0094f2 thru 0x0157fe (0xc30d bytes) 000927 1787550018.046693 read 0x0127aa thru 0x016bcf (0x4426 bytes) 000929 1787550018.048274 fallocate 0x02f623 thru 0x03a1a6 (0xab84 bytes) 000932 1787550018.053633 mapwrite 0x02671c thru 0x02d84b (0x7130 bytes) 000934 1787550018.068089 trunc from 0x03da44 to 0x0146cf (0x29374 bytes) 000935 1787550018.080292 trunc from 0x0146d0 to 0x02c252 (0x17b83 bytes) 000937 1787550018.098119 trunc from 0x02c253 to 0x00f36b (0x1cee7 bytes) 000938 1787550018.117100 mapwrite 0x01d797 thru 0x01de4f (0x6b9 bytes) 000939 1787550018.156733 fallocate 0x00782e thru 0x013b0a (0xc2dd bytes) 000940 1787550018.162520 fallocate 0x029794 thru 0x02e57c (0x4de9 bytes) 000941 1787550018.169123 trunc from 0x00392c to 0x00520e (0x18e3 bytes) 000942 1787550018.176533 write 0x024232 thru 0x026fcf (0x2d9e bytes) 000943 1787550018.178260 trunc from 0x026fd0 to 0x020ab9 (0x6516 bytes) 000944 1787550018.198096 trunc from 0x019618 to 0x02013c (0x6b25 bytes) 000945 1787550018.204373 read 0x005232 thru 0x010097 (0xae66 bytes) 000946 1787550018.217338 read 0x01a07d thru 0x020ab9 (0x6a3d bytes) 000947 1787550018.222515 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20aba): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20aba): Mirror 7: 0x3a3984c7 (stale) Mirror 8: 0x3a3984c7 (stale) CRC-32 checksum value for chunk [0, 0x20aba): Mirror 1: 0xa8897dd9 Mirror 9: 0xa8897dd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20aba: skipped 000948 1787550019.131688 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20aba): Mirror 8: 0x3a3984c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20aba): Mirror 8: 0x3a3984c7 (stale) CRC-32 checksum value for chunk [0, 0x20aba): Mirror 1: 0xa8897dd9 Mirror 9: 0xa8897dd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20aba: skipped 000949 1787550019.677220 trunc from 0x009ad2 to 0x00fa06 (0x5f35 bytes) 000950 1787550019.732323 mapread 0x016c40 thru 0x01d0e5 (0x64a6 bytes) 000951 1787550019.759154 trunc from 0x020aba to 0x0106fd (0x103bc bytes) 000952 1787550019.810860 punch from 0x00cc2a to 0x0106fd (0x3ad4 bytes) 000953 1787550019.818294 trunc from 0x0106fe to 0x038b79 (0x2847c bytes) 000954 1787550019.843987 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 8: 0x3b9ee4eb (stale) Mirror 9: 0x987feac1 (stale) CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 1: 0xb76bcd1 Mirror 10: 0xb76bcd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b7a: skipped 000955 1787550020.931306 mapread 0x006d31 thru 0x014e2e (0xe0fe bytes) 000957 1787550020.996247 mapwrite 0x00a61a thru 0x00babd (0x14a4 bytes) 000958 1787550021.166003 mapread 0x000d29 thru 0x002add (0x1db5 bytes) 000959 1787550021.176642 trunc from 0x038b7a to 0x00e473 (0x2a706 bytes) 000960 1787550021.236862 mapread 0x006c81 thru 0x0099b9 (0x2d39 bytes) 000961 1787550021.239169 punch from 0x00aa7e to 0x00e473 (0x39f6 bytes) 000962 1787550021.261919 MIRROR_RESYNC 4 mirrors 000963 1787550022.913234 trunc from 0x00b214 to 0x00e473 (0x3260 bytes) 000964 1787550022.967084 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xe474): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe474): Mirror 8: 0x7dd0f07c (stale) Mirror 9: 0x7dd0f07c (stale) Mirror 10: 0x7dd0f07c (stale) CRC-32 checksum value for chunk [0, 0xe474): Mirror 1: 0x7dd0f07c Mirror 11: 0x7dd0f07c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe474: skipped 000965 1787550023.964552 punch from 0x009605 to 0x00e473 (0x4e6f bytes) 000966 1787550024.009763 MIRROR_RESYNC 5 mirrors 000967 1787550025.205345 write 0x015c8f thru 0x02276d (0xcadf bytes) 000968 1787550025.229320 read 0x0026d4 thru 0x003e62 (0x178f bytes) 000969 1787550025.234344 write 0x039f2e thru 0x03ffff (0x60d2 bytes) 000970 1787550025.240487 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x48a4f75a (stale) Mirror 9: 0x48a4f75a (stale) Mirror 10: 0x48a4f75a (stale) Mirror 11: 0x48a4f75a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf529d29e Mirror 12: 0xf529d29e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000971 1787550026.148913 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x48a4f75a (stale) Mirror 9: 0x48a4f75a (stale) Mirror 11: 0x48a4f75a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x48a4f75a (stale) Mirror 9: 0x48a4f75a (stale) Mirror 11: 0x48a4f75a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf529d29e Mirror 12: 0xf529d29e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000972 1787550026.753864 mapread 0x0302ec thru 0x03d21c (0xcf31 bytes) 000973 1787550026.818817 punch from 0x026cb0 to 0x03577a (0xeacb bytes) 000974 1787550026.916232 write 0x031dc5 thru 0x037aef (0x5d2b bytes) 000975 1787550026.944868 fallocate 0x00f159 thru 0x011762 (0x260a bytes) 000976 1787550026.962354 punch from 0x0273e5 to 0x02d8cc (0x64e8 bytes) 000977 1787550026.972140 read 0x03e9be thru 0x03ffff (0x1642 bytes) 000978 1787550026.991158 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x48a4f75a (stale) Mirror 9: 0x48a4f75a (stale) Mirror 11: 0x48a4f75a (stale) Mirror 12: 0xf529d29e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf0418e34 Mirror 13: 0xf0418e34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1787550028.334446 write 0x03d91a thru 0x03ffff (0x26e6 bytes) 000981 1787550028.417060 trunc from 0x005dcd to 0x01521c (0xf450 bytes) 000983 1787550028.438905 MIRROR_RESYNC 6 mirrors 000984 1787550030.496271 mapwrite 0x01fd4e thru 0x025b66 (0x5e19 bytes) 000985 1787550030.602326 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1690df6f (stale) Mirror 9: 0x1690df6f (stale) Mirror 11: 0x1690df6f (stale) Mirror 13: 0x1690df6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1690df6f (stale) Mirror 9: 0x1690df6f (stale) Mirror 11: 0x1690df6f (stale) Mirror 13: 0x1690df6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1787550031.033850 trunc from 0x040000 to 0x00c474 (0x33b8b bytes) 000987 1787550031.060691 trunc from 0x004217 to 0x007af8 (0x38e2 bytes) 000988 1787550031.067171 write 0x023959 thru 0x024e3b (0x14e3 bytes) 000989 1787550031.068613 trunc from 0x0172a6 to 0x01e582 (0x72dd bytes) 000990 1787550031.082065 MIRROR_RESYNC 5 mirrors 000991 1787550032.434159 mapread 0x022da9 thru 0x024e3b (0x2093 bytes) 000992 1787550032.480986 punch from 0x0013e4 to 0x005fe0 (0x4bfd bytes) 000993 1787550032.538196 read 0x014502 thru 0x01daa2 (0x95a1 bytes) 000994 1787550032.554127 mapwrite 0x00a4f2 thru 0x0183e8 (0xdef7 bytes) 000995 1787550032.620536 mapread 0x0205b8 thru 0x022413 (0x1e5c bytes) 000996 1787550032.648986 trunc from 0x024e3c to 0x029ef6 (0x50bb bytes) 000997 1787550032.682463 read 0x025215 thru 0x029ef6 (0x4ce2 bytes) 000998 1787550032.736541 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29ef7): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ef7): Mirror 8: 0xcc420bda (stale) Mirror 9: 0xcc420bda (stale) Mirror 11: 0xcc420bda (stale) Mirror 13: 0xcc420bda (stale) CRC-32 checksum value for chunk [0, 0x29ef7): Mirror 1: 0xb5f163b0 Mirror 14: 0xb5f163b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ef7: skipped 000999 1787550034.029899 read 0x00f127 thru 0x0190af (0x9f89 bytes) 001000 1787550034.049266 trunc from 0x029ef7 to 0x004c8a (0x2526c bytes) All operations completed A-OK! PASS 70a (129s) == sanity-flr test 71: check mirror extend parent fid ==== 01:40:40 (1787550040) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x180:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 01:40:51 (1787550051) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295692 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131916 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x120:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x120:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:41:00 (1787550060) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1876256 .. Extending file size to 6007296 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5189536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6567936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8161440 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8450432 .. Extending file size to 9079008 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9089888 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9341184 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61425 61426 61427 61429 61430 done 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) 192.168.206.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:42:20 (1787550140) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5393024 .. Extending file size to 916992 .. Extending file size to 6998400 .. Extending file size to 6938112 .. Extending file size to 7692192 .. Extending file size to 8628256 .. Extending file size to 8570656 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8730464 .. Extending file size to 8575712 .. Extending file size to 8885696 .. Extending file size to 9029888 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9147712 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62963 62964 62965 62969 62971 done 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) 192.168.206.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:43:37 (1787550217) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1268876 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18908 3588112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43580 7170460 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1268876 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18908 3588112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43580 7170460 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136013 s, 15.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233272 s, 9.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf21f0d18 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:43:58 (1787550238) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111704 s, 18.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:44:05 (1787550245) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xae:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:44:13 (1787550253) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112224 s, 18.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:44:23 (1787550263) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108046 s, 19.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:44:32 (1787550272) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0625407 s, 33.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:45:07 (1787550307) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148419 s, 7.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:46:04 (1787550364) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116845 s, 17.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:46:57 (1787550417) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg632-server: Pool lustre.test_204f created Adding targets to pool oleg632-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg632-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146069 s, 14.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg632-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg632-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg632-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:48:14 (1787550494) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:48:22 (1787550502) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:48:30 (1787550510) oleg632-server: Pool lustre.test_206 created Adding targets to pool oleg632-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg632-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg632-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg632-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg632-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:48:58 (1787550538) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11735 s, 8.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382165 s, 27.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x880683b6 Mirror 3: 0x880683b6 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:49:09 (1787550549) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:49:12 (1787550552) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:49:15 (1787550555) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.107563 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1850000 Read this (4 bytes): BBBB 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:49:55 (1787550595) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0200329 s, 409 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:50:04 (1787550604) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557954 s, 18.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xcf lmm_fid: [0x200000405:0xcf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xcf lmm_fid: [0x200000405:0xcf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x145:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611343 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 01:50:05.000000000 -0400 Modify: 2026-08-24 01:50:05.000000000 -0400 Change: 2026-08-24 01:50:07.000000000 -0400 Birth: 2026-08-24 01:50:05.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:50:12 (1787550612) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509974 s, 20.6 MB/s oleg632-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg632-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg632-server: O/280000401/d6/166: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:50:36 (1787550636) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.3222 s, 127 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0485573 s, 84.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 01:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 01:50 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:50:45 (1787550645) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:50:53 (1787550653) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1868 sec =========== 01:51:03 (1787550663) === sanity-flr: start cleanup 01:51:05 (1787550665) === === sanity-flr: finish cleanup 01:51:15 (1787550675) ===