-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 03:19:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg624-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 03:19:21 (1787383161) === oleg624-client.virtnet: executing check_config_client /mnt/lustre oleg624-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg624-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95501077e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95501077e800.idle_timeout=debug disable quota as required oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:19:56 (1787383196) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8216] == sanity-flr test 0a: lfs mirror create with -N option == 03:19:58 (1787383198) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:20:13 (1787383213) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:20:16 (1787383216) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:20:18 (1787383218) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:20:40 (1787383240) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:20:43 (1787383243) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 03:20:45 (1787383245) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff95501077e800.stats=clear osc.lustre-OST0001-osc-ffff95501077e800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:20:52 (1787383252) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:21:03 (1787383263) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:21:12 (1787383272) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:21:14 (1787383274) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:21:22 (1787383282) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 03:21:32 (1787383292) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:21:40 (1787383300) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:21:49 (1787383309) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:21:58 (1787383318) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:22:10 (1787383330) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:22:16 (1787383336) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.663959 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960356 s, 10.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:22:25 (1787383345) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430064 s, 39.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:22:34 (1787383354) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:22:44 (1787383364) Stopping /mnt/lustre-ost1 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg624-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg624-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:24:00 (1787383440) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.441179 s, 26.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg624-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:24:44 (1787383484) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg624-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg624-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:26:01 (1787383561) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.5366 s, 20.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 03:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.910s user 0m0.000s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.106s user 0m0.000s sys 0m0.208s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg624-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg624-server oleg624-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.620s user 0m0.001s sys 0m0.186s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg624-server: oleg624-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg624-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95501077e800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:27:34 (1787383654) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:27:37 (1787383657) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:27:40 (1787383660) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 03:27:43 (1787383663) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 03:27:51 (1787383671) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.720087 s, 17.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:28:04 (1787383684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342662 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260528 s, 40.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:28:12 (1787383692) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:28:21 (1787383701) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00209811 s, 488 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000812835 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00488363 s, 210 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102039 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 03:28:31 (1787383711) create test files with size 15730900 .. 0c69b986087ebeb9f30f9eec6c4da21a - b5c11e54c5a09fd8fd22b1cec1f0e0f2 - f6da8cae634d86595f46d54935c2960a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 03:29:01 (1787383741) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:29:16 (1787383756) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:29:23 (1787383763) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103903 s, 10.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513618 s, 20.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 03:29:32 (1787383772) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143615 s, 14.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.513698 s, 8.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300258 s, 7.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 03:29:50 (1787383790) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:29:54 (1787383794) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:29:57 (1787383797) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234155 s, 44.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 03:30:04 (1787383804) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.312355 s, 10.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:30:15 (1787383815) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:30:25 (1787383825) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260954 s, 40.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:30:36 (1787383836) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff95500424d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.276421 s, 37.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff95500424f800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:30:50 (1787383850) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:31:28 (1787383888) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:32:07 (1787383927) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:32:10 (1787383930) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.321433 s, 9.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.110229 s, 28.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9581a56c (nosync,stale) Mirror 2: 0x9581a56c (nosync,stale) ** verify mirror 2 doesn't change original checksum: aaa78c4983fbb3f769de85112b4154de - mirror 1 checksum: bea46f068acc41c87421a468bba5b078 - mirror 2 checksum: aaa78c4983fbb3f769de85112b4154de - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T03:32:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T03:32:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: aaa78c4983fbb3f769de85112b4154de - mirror 1 checksum: bea46f068acc41c87421a468bba5b078 - mirror 2 checksum: bea46f068acc41c87421a468bba5b078 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T03:32:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T03:32:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:32:20 (1787383940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324412 s, 32.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:32:28 (1787383948) Failing mds1 on oleg624-server Stopping /mnt/lustre-mds1 (opts:) on oleg624-server 03:32:39 (1787383959) shut down facet: mds1 facet_host: oleg624-server facet_failover_host: oleg624-server Failover mds1 to oleg624-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 03:33:01 (1787383981) targets are mounted 03:33:01 (1787383981) facet_failover done oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299183 s, 35.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:33:17 (1787383997) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0576442 s, 355 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0367612 s, 557 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:33:26 (1787384006) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0379701 s, 539 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.01072 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:33:36 (1787384016) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00759909 s, 2.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:33:44 (1787384024) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0329615 s, 621 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:33:52 (1787384032) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00807083 s, 508 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248367 s, 165 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:33:59 (1787384039) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:34:20 (1787384060) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:34:48 (1787384088) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:35:12 (1787384112) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:35:37 (1787384137) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 03:35:43 (1787384143) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57369 mirror split pid 57370 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57369 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57370 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 03:36:52 (1787384212) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4214 000001 1787384214.492194 MIRROR_EXTEND to 2 mirrors skipping zero length zero range truncating to largest ever: 0x3e588 000003 1787384215.131974 trunc from 00000000 to 0x03e587 (0x3e588 bytes) 000004 1787384215.256625 read 0x000079 thru 0x00da34 (0xd9bc bytes) 000005 1787384215.267187 trunc from 0x02e8b3 to 0x0312b8 (0x2a06 bytes) 000006 1787384215.272206 mapwrite 0x02112e thru 0x028eeb (0x7dbe bytes) 000007 1787384215.320641 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e588): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e588): Mirror 2: 0x74d6f1f5 (stale) CRC-32 checksum value for chunk [0, 0x3e588): Mirror 1: 0x469866ae Mirror 3: 0x469866ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e588: skipped 000008 1787384216.350017 trunc from 0x03e588 to 0x021945 (0x1cc42 bytes) 000009 1787384216.493423 write 0x01c0ad thru 0x01c920 (0x874 bytes) 000010 1787384216.502985 write 0x02c30d thru 0x03c2a1 (0xff95 bytes) 000011 1787384216.505745 fallocate 0x00208a thru 0x00cf25 (0xae9c bytes) 000012 1787384216.510881 read 0x0272a9 thru 0x02e968 (0x76c0 bytes) 000013 1787384216.518014 trunc from 0x010752 to 0x013132 (0x29e1 bytes) 000014 1787384216.522586 punch from 0x01f526 to 0x02321e (0x3cf9 bytes) 000015 1787384216.528320 punch from 0x03ae7e to 0x03c2a1 (0x1424 bytes) 000017 1787384216.547766 trunc from 0x005b86 to 0x007e00 (0x227b bytes) 000018 1787384216.553049 write 0x035a79 thru 0x036278 (0x800 bytes) 000019 1787384216.554280 write 0x00247e thru 0x00914b (0x6cce bytes) 000020 1787384216.565127 trunc from 0x01cff5 to 0x01dd01 (0xd0d bytes) 000021 1787384216.570476 mapread 0x00bcc7 thru 0x01961a (0xd954 bytes) 000022 1787384216.598291 write 0x00da7a thru 0x00f9a9 (0x1f30 bytes) 000023 1787384216.600230 mapwrite 0x030eb1 thru 0x038439 (0x7589 bytes) 000024 1787384216.626072 trunc from 0x008b43 to 0x00d228 (0x46e6 bytes) 000026 1787384216.632091 write 0x03f593 thru 0x03ffff (0xa6d bytes) 000027 1787384216.633985 fallocate 0x003df2 thru 0x00beb3 (0x80c2 bytes) 000028 1787384216.644182 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x4a060703 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe1d3d7c8 Mirror 4: 0xe1d3d7c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000029 1787384218.083527 fallocate 0x007070 thru 0x00c4df (0x5470 bytes) 000030 1787384218.130741 punch from 0x02a071 to 0x02d0f4 (0x3084 bytes) 000031 1787384218.136557 mapwrite 0x007aa0 thru 0x00c20f (0x4770 bytes) 000032 1787384218.198311 read 0x03352a thru 0x036641 (0x3118 bytes) 000033 1787384218.203822 mapread 0x037a74 thru 0x03ffff (0x858c bytes) 000034 1787384218.232861 fallocate 0x000eda thru 0x008fdb (0x8102 bytes) 000035 1787384218.239769 trunc from 0x024190 to 0x026ec5 (0x2d36 bytes) 000036 1787384218.250243 read 0x0253a0 thru 0x02b1f3 (0x5e54 bytes) 000037 1787384218.273843 write 0x01a85f thru 0x01b150 (0x8f2 bytes) 000038 1787384218.286365 trunc from 0x00397e to 0x0063a4 (0x2a27 bytes) 000039 1787384218.297871 mapwrite 0x007879 thru 0x016434 (0xebbc bytes) 000040 1787384218.353838 trunc from 0x040000 to 0x02c529 (0x13ad6 bytes) 000041 1787384218.401617 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c52a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c52a): Mirror 2: 0x38dba7d5 (stale) Mirror 3: 0xcc05cfa4 (stale) Mirror 4: 0xf4b866df (stale) CRC-32 checksum value for chunk [0, 0x2c52a): Mirror 1: 0x940f88bd Mirror 5: 0x940f88bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c52a: skipped 000042 1787384219.921114 mapwrite 0x003b52 thru 0x006d92 (0x3241 bytes) 000044 1787384220.071781 trunc from 0x026446 to 0x02c529 (0x60e4 bytes) fallocating to largest ever: 0x40000 000045 1787384220.077719 fallocate 0x036e9b thru 0x03ffff (0x9165 bytes) 000046 1787384220.083093 write 0x03a0c3 thru 0x03ffff (0x5f3d bytes) 000047 1787384220.093931 fallocate 0x019f50 thru 0x025fa6 (0xc057 bytes) 000048 1787384220.099491 trunc from 0x040000 to 0x01ab13 (0x254ec bytes) 000049 1787384220.122213 fallocate 0x00b38f thru 0x010f35 (0x5ba7 bytes) 000050 1787384220.133343 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ab14): Mirror 3: 0xa381989f (stale) Mirror 4: 0x52365288 (stale) Mirror 5: 0xbf0ed4a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab14): Mirror 3: 0xa381989f (stale) Mirror 4: 0x52365288 (stale) Mirror 5: 0xbf0ed4a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab14: skipped 000052 1787384220.588505 mapread 0x017561 thru 0x0176b5 (0x155 bytes) 000053 1787384220.594270 trunc from 0x01ab14 to 0x034124 (0x19611 bytes) 000054 1787384220.619569 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34125): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34125): Mirror 3: 0x5c4985b1 (stale) Mirror 4: 0x60f34033 (stale) Mirror 5: 0xafc75f18 (stale) CRC-32 checksum value for chunk [0, 0x34125): Mirror 1: 0x22b7d04d Mirror 6: 0x22b7d04d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34125: skipped 000055 1787384221.860112 write 0x012bb7 thru 0x017314 (0x475e bytes) 000056 1787384221.924268 trunc from 0x02f541 to 0x033b9f (0x465f bytes) 000057 1787384221.930002 trunc from 0x018c77 to 0x01caac (0x3e36 bytes) 000058 1787384221.934690 mapread 0x0025e9 thru 0x006858 (0x4270 bytes) 000059 1787384221.966568 punch from 0x001eef to 0x00c48e (0xa5a0 bytes) 000060 1787384221.976026 read 0x00c7ac thru 0x013cd9 (0x752e bytes) 000061 1787384221.985792 write 0x0267d0 thru 0x0287e9 (0x201a bytes) 000063 1787384221.998567 mapwrite 0x00169e thru 0x00cd13 (0xb676 bytes) 000065 1787384222.031769 mapwrite 0x023db5 thru 0x025db2 (0x1ffe bytes) 000066 1787384222.057527 write 0x02b456 thru 0x02d881 (0x242c bytes) 000067 1787384222.066621 fallocate 0x03eb62 thru 0x03f5c6 (0xa65 bytes) 000068 1787384222.075594 mapread 0x002725 thru 0x00cf7c (0xa858 bytes) 000069 1787384222.077542 fallocate 0x0293e0 thru 0x02e8e1 (0x5502 bytes) 000070 1787384222.083130 fallocate 0x03f3b8 thru 0x03ffff (0xc48 bytes) 000072 1787384222.094590 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34125): Mirror 4: 0x60f34033 (stale) Mirror 5: 0xafc75f18 (stale) Mirror 6: 0x22b7d04d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34125): Mirror 4: 0x60f34033 (stale) Mirror 5: 0xafc75f18 (stale) Mirror 6: 0x22b7d04d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34125: skipped 000074 1787384222.571222 mapread 0x0151fa thru 0x015dba (0xbc1 bytes) 000075 1787384222.585158 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34125): Mirror 5: 0xafc75f18 (stale) Mirror 6: 0x22b7d04d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34125): Mirror 5: 0xafc75f18 (stale) Mirror 6: 0x22b7d04d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34125: skipped 000076 1787384223.021323 mapwrite 0x01a630 thru 0x0255b8 (0xaf89 bytes) 000077 1787384223.159675 punch from 0x022c5c to 0x0298bb (0x6c60 bytes) 000078 1787384223.169359 mapwrite 0x0078d1 thru 0x009cf4 (0x2424 bytes) 000079 1787384223.272136 punch from 0x017b5a to 0x026d54 (0xf1fb bytes) 000080 1787384223.282770 fallocate 0x031ed8 thru 0x03c11d (0xa246 bytes) 000081 1787384223.314535 MIRROR_RESYNC 3 mirrors 000082 1787384225.189316 read 0x01ece2 thru 0x02df30 (0xf24f bytes) 000083 1787384225.197251 mapread 0x01609c thru 0x017f57 (0x1ebc bytes) 000084 1787384225.207341 write 0x00a393 thru 0x0141dd (0x9e4b bytes) 000085 1787384225.244957 mapread 0x02e6c1 thru 0x034124 (0x5a64 bytes) 000086 1787384225.260663 read 0x021a9a thru 0x02d639 (0xbba0 bytes) 000087 1787384225.273961 punch from 0x02ee62 to 0x032b02 (0x3ca1 bytes) 000088 1787384225.278888 mapread 0x017eb8 thru 0x021ffe (0xa147 bytes) 000089 1787384225.281511 trunc from 0x034125 to 0x00e018 (0x2610c bytes) 000090 1787384225.307098 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe019): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe019): Mirror 5: 0x86b5b40b (stale) Mirror 6: 0x86b5b40b (stale) CRC-32 checksum value for chunk [0, 0xe019): Mirror 1: 0xec0e0420 Mirror 7: 0xec0e0420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe019: skipped 000091 1787384226.279798 read 0x006978 thru 0x00e018 (0x76a1 bytes) 000092 1787384226.326397 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe019): Mirror 6: 0x86b5b40b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe019): Mirror 6: 0x86b5b40b (stale) CRC-32 checksum value for chunk [0, 0xe019): Mirror 1: 0xec0e0420 Mirror 7: 0xec0e0420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe019: skipped 000093 1787384227.275134 mapwrite 0x001d71 thru 0x00ff38 (0xe1c8 bytes) 000094 1787384227.497360 punch from 0x00b729 to 0x00ff38 (0x4810 bytes) 000095 1787384227.506515 trunc from 0x00ff39 to 0x0340d3 (0x2419b bytes) 000097 1787384227.543080 write 0x02ea51 thru 0x03af9e (0xc54e bytes) 000098 1787384227.554872 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af9f): Mirror 7: 0x45a8c77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af9f): Mirror 7: 0x45a8c77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af9f: skipped 000099 1787384227.830962 mapwrite 0x026ba0 thru 0x032b2d (0xbf8e bytes) 000100 1787384227.880183 mapread 0x030daa thru 0x038249 (0x74a0 bytes) 000101 1787384227.881942 trunc from 0x0354a4 to 0x03af9e (0x5afb bytes) 000102 1787384227.888538 trunc from 0x02582a to 0x02b3c8 (0x5b9f bytes) 000103 1787384227.894867 punch from 0x0251a4 to 0x02efec (0x9e49 bytes) 000104 1787384227.900131 trunc from 0x03af9f to 0x0160fd (0x24ea1 bytes) 000105 1787384227.923201 punch from 0x0136a2 to 0x0156dc (0x203b bytes) 000106 1787384227.928702 write 0x02421f thru 0x026532 (0x2314 bytes) 000107 1787384227.930051 write 0x02750e thru 0x02a2b5 (0x2da8 bytes) 000108 1787384227.932152 trunc from 0x02a2b6 to 0x013669 (0x16c4c bytes) 000109 1787384227.949777 MIRROR_RESYNC 2 mirrors 000110 1787384228.655672 mapread 0x00c295 thru 0x0106c7 (0x4433 bytes) 000111 1787384228.681300 mapread 0x00973e thru 0x00ffa0 (0x6863 bytes) 000112 1787384228.682251 trunc from 0x002e11 to 0x0124ea (0xf6da bytes) 000113 1787384228.716207 punch from 0x000cfd to 0x01062b (0xf92f bytes) 000114 1787384228.722411 mapwrite 0x02c2f6 thru 0x034725 (0x8430 bytes) 000115 1787384228.789066 fallocate 0x02898d thru 0x0291ed (0x861 bytes) 000116 1787384228.813218 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000117 1787384229.055416 punch from 0x01951e to 0x021530 (0x8013 bytes) 000118 1787384229.060147 trunc from 0x034726 to 0x02757c (0xd1a9 bytes) 000120 1787384229.095753 write 0x03b5c9 thru 0x03c826 (0x125e bytes) 000121 1787384229.098607 read 0x01237e thru 0x02017a (0xddfd bytes) 000123 1787384229.119072 mapwrite 0x00f241 thru 0x0150e3 (0x5ea3 bytes) 000124 1787384229.166753 read 0x0034d8 thru 0x01316a (0xfc93 bytes) 000125 1787384229.186345 trunc from 0x02c31a to 0x0358d5 (0x95bc bytes) 000126 1787384229.192955 mapwrite 0x03b036 thru 0x03ffff (0x4fca bytes) 000127 1787384229.229698 punch from 0x0217dc to 0x02aa72 (0x9297 bytes) 000129 1787384229.236044 punch from 0x01d95c to 0x02bad0 (0xe175 bytes) 000130 1787384229.241556 mapwrite 0x0054ac thru 0x0093f0 (0x3f45 bytes) 000132 1787384229.261478 read 0x026921 thru 0x02e6c3 (0x7da3 bytes) 000133 1787384229.275465 mapwrite 0x0340d8 thru 0x03f1ad (0xb0d6 bytes) 000134 1787384229.305830 trunc from 0x040000 to 0x003192 (0x3ce6d bytes) 000135 1787384229.336555 read 0x001f1b thru 0x003192 (0x1278 bytes) 000136 1787384229.337366 mapwrite 0x00e63e thru 0x018e18 (0xa7db bytes) 000137 1787384229.401288 fallocate 0x013ffb thru 0x01a0cf (0x60d5 bytes) 000139 1787384229.408718 trunc from 0x014e86 to 0x01a0cf (0x524a bytes) 000140 1787384229.413512 fallocate 0x01d3bb thru 0x025bf9 (0x883f bytes) 000141 1787384229.444488 MIRROR_EXTEND to 2 mirrors 000142 1787384230.238792 mapread 0x00875d thru 0x00a027 (0x18cb bytes) 000143 1787384230.284269 fallocate 0x00d5b0 thru 0x018cf7 (0xb748 bytes) 000144 1787384230.320391 punch from 0x009d9b to 0x01974f (0xf9b5 bytes) 000145 1787384230.325801 read 0x00b53f thru 0x010e7c (0x593e bytes) 000146 1787384230.336711 mapwrite 0x029cd9 thru 0x0357bc (0xbae4 bytes) 000147 1787384230.452135 trunc from 0x0357bd to 0x00772a (0x2e092 bytes) 000148 1787384230.482420 trunc from 0x002038 to 0x00772a (0x56f3 bytes) 000149 1787384230.490407 fallocate 0x02cfe3 thru 0x032d0e (0x5d2c bytes) 000150 1787384230.502251 fallocate 0x021920 thru 0x026a4b (0x512c bytes) 000152 1787384230.519451 fallocate 0x019740 thru 0x01bbea (0x24ab bytes) 000153 1787384230.533749 mapwrite 0x027c56 thru 0x0290ca (0x1475 bytes) 000154 1787384230.577494 fallocate 0x03449c thru 0x03c14e (0x7cb3 bytes) 000155 1787384230.583558 read 0x01351f thru 0x0191e9 (0x5ccb bytes) 000156 1787384230.596477 read 0x02542c thru 0x0337b5 (0xe38a bytes) 000157 1787384230.597505 mapwrite 0x01b765 thru 0x021f80 (0x681c bytes) 000158 1787384230.625800 mapwrite 0x03e12e thru 0x03ffff (0x1ed2 bytes) 000159 1787384230.669466 write 0x0377d2 thru 0x03ffff (0x882e bytes) 000160 1787384230.678658 trunc from 0x040000 to 0x03e60b (0x19f4 bytes) 000161 1787384230.718579 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e60c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e60c): Mirror 2: 0xc1fd13d4 (stale) CRC-32 checksum value for chunk [0, 0x3e60c): Mirror 1: 0xa3223af3 Mirror 3: 0xa3223af3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e60c: skipped 000162 1787384231.641343 trunc from 0x034233 to 0x039591 (0x535f bytes) 000163 1787384231.698419 mapwrite 0x00a62c thru 0x00e899 (0x426e bytes) 000164 1787384231.759148 mapwrite 0x00d10c thru 0x0129bb (0x58b0 bytes) 000165 1787384231.779580 trunc from 0x03e60c to 0x03a57d (0x408e bytes) 000166 1787384231.812460 punch from 0x03a135 to 0x03a57d (0x449 bytes) 000167 1787384231.818535 fallocate 0x026d7e thru 0x028029 (0x12ac bytes) 000168 1787384231.826308 read 0x009bb5 thru 0x018d1f (0xf16b bytes) 000169 1787384231.836281 mapread 0x02a90f thru 0x03131a (0x6a0c bytes) 000170 1787384231.868874 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a57e): Mirror 3: 0xc6217168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a57e): Mirror 3: 0xc6217168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a57e: skipped 000172 1787384232.184085 trunc from 0x03a57e to 0x00dd86 (0x2c7f7 bytes) 000173 1787384232.226389 mapwrite 0x032d47 thru 0x03abfa (0x7eb4 bytes) 000174 1787384232.299644 punch from 0x0036a3 to 0x00a9c5 (0x7323 bytes) 000175 1787384232.306936 trunc from 0x01d315 to 0x02ca91 (0xf77d bytes) 000176 1787384232.317887 MIRROR_RESYNC 2 mirrors 000177 1787384233.194429 trunc from 0x03abfb to 0x013659 (0x275a1 bytes) 000178 1787384233.264532 fallocate 0x0180a5 thru 0x0248bb (0xc817 bytes) 000179 1787384233.270429 read 0x004ba4 thru 0x013659 (0xeab6 bytes) 000180 1787384233.283831 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000181 1787384233.416271 write 0x02fc89 thru 0x036fba (0x7332 bytes) 000182 1787384233.428437 mapwrite 0x038f24 thru 0x03ffff (0x70dc bytes) 000183 1787384233.491397 trunc from 0x040000 to 0x030a87 (0xf578 bytes) 000184 1787384233.545169 mapread 0x020f4e thru 0x02cc93 (0xbd46 bytes) 000185 1787384233.589038 punch from 0x0127ef to 0x018059 (0x586b bytes) 000186 1787384233.597171 fallocate 0x006c7e thru 0x007f1d (0x12a0 bytes) 000187 1787384233.606610 trunc from 0x030a88 to 0x0227e7 (0xe2a0 bytes) 000188 1787384233.635617 trunc from 0x0227e8 to 0x02b795 (0x8fae bytes) 000189 1787384233.665128 read 0x023326 thru 0x02b795 (0x8470 bytes) 000190 1787384233.692548 write 0x011198 thru 0x01fc89 (0xeaf2 bytes) 000191 1787384233.706890 trunc from 0x0266e4 to 0x02b795 (0x50b2 bytes) 000192 1787384233.718376 read 0x0122be thru 0x015e40 (0x3b83 bytes) 000193 1787384233.719226 write 0x011b9f thru 0x016075 (0x44d7 bytes) 000195 1787384233.722651 trunc from 0x00eacf to 0x013d0a (0x523c bytes) 000196 1787384233.758201 read 0x01b6f2 thru 0x01eb9d (0x34ac bytes) 000197 1787384233.761096 trunc from 0x013046 to 0x01e89c (0xb857 bytes) 000199 1787384233.771080 trunc from 0x02b796 to 0x026651 (0x5144 bytes) 000200 1787384233.795557 trunc from 0x026652 to 0x01f190 (0x74c1 bytes) 000201 1787384233.819142 trunc from 0x01f191 to 0x01c2d9 (0x2eb7 bytes) 000202 1787384233.846072 punch from 0x00a720 to 0x0192e1 (0xebc2 bytes) 000203 1787384233.866444 trunc from 0x00219f to 0x010435 (0xe297 bytes) 000204 1787384233.876051 mapwrite 0x00b82e thru 0x010883 (0x5056 bytes) 000205 1787384233.931402 mapread 0x005574 thru 0x012139 (0xcbc6 bytes) 000206 1787384233.936681 punch from 0x015934 to 0x01c2d9 (0x69a6 bytes) 000207 1787384233.945168 trunc from 0x00ff67 to 0x011842 (0x18dc bytes) 000208 1787384233.952216 fallocate 0x005bf7 thru 0x00e18f (0x8599 bytes) 000209 1787384233.962590 punch from 0x016f03 to 0x01c2d9 (0x53d7 bytes) 000211 1787384233.981115 punch from 0x0030e9 to 0x009489 (0x63a1 bytes) 000212 1787384234.001745 trunc from 0x01c2da to 0x00540d (0x16ecc bytes) 000213 1787384234.029079 trunc from 0x0023d6 to 0x00540d (0x3038 bytes) 000215 1787384234.045137 mapwrite 0x025ec8 thru 0x026a79 (0xbb2 bytes) 000216 1787384234.089846 fallocate 0x02b3f5 thru 0x035f48 (0xab54 bytes) 000217 1787384234.095106 mapread 0x015af7 thru 0x019214 (0x371e bytes) 000218 1787384234.126166 trunc from 0x035f49 to 0x029dbb (0xc18d bytes) 000220 1787384234.146588 mapread 0x00a226 thru 0x00af0d (0xce8 bytes) 000221 1787384234.153580 mapread 0x01127f thru 0x01b17d (0x9eff bytes) 000222 1787384234.184283 mapread 0x0033ff thru 0x0120a2 (0xeca4 bytes) 000224 1787384234.297609 trunc from 0x029dbc to 0x0210cd (0x8cee bytes) 000225 1787384234.326026 mapwrite 0x03eba0 thru 0x03ffff (0x1460 bytes) 000226 1787384234.376378 write 0x018778 thru 0x01ecfd (0x6586 bytes) 000227 1787384234.381607 read 0x025682 thru 0x026891 (0x1210 bytes) 000228 1787384234.429145 read 0x02a8b6 thru 0x0305c3 (0x5d0e bytes) 000229 1787384234.429973 trunc from 0x040000 to 0x02eb4b (0x114b4 bytes) 000230 1787384234.447501 trunc from 0x01edf1 to 0x02281f (0x3a2f bytes) 000231 1787384234.474167 read 0x0259a5 thru 0x02969f (0x3cfb bytes) 000233 1787384234.476417 trunc from 0x025fa3 to 0x02eb4b (0x8ba9 bytes) 000235 1787384234.494266 punch from 0x02aad9 to 0x02eb4b (0x4073 bytes) 000236 1787384234.505248 fallocate 0x007739 thru 0x012040 (0xa908 bytes) 000237 1787384234.518009 read 0x007378 thru 0x0108f0 (0x9579 bytes) 000238 1787384234.524152 trunc from 0x02eb4c to 0x03061b (0x1ad0 bytes) 000240 1787384234.568615 MIRROR_EXTEND to 2 mirrors 000241 1787384235.474698 read 0x021c98 thru 0x02d295 (0xb5fe bytes) 000242 1787384235.532220 trunc from 0x03061c to 0x02e62c (0x1fef bytes) 000243 1787384235.590432 write 0x006e3e thru 0x014bbc (0xdd7f bytes) 000244 1787384235.606679 punch from 0x01a7b8 to 0x0298e4 (0xf12d bytes) 000245 1787384235.617458 read 0x0281ae thru 0x02e62c (0x647f bytes) 000246 1787384235.634947 read 0x003cc0 thru 0x0098da (0x5c1b bytes) 000247 1787384235.636599 mapwrite 0x02806f thru 0x03084e (0x87e0 bytes) 000248 1787384235.731865 fallocate 0x03e69f thru 0x03ffff (0x1961 bytes) 000249 1787384235.740277 fallocate 0x016bfd thru 0x024ac0 (0xdec4 bytes) 000250 1787384235.758263 trunc from 0x02f358 to 0x03084e (0x14f7 bytes) 000251 1787384235.770060 write 0x019fe6 thru 0x01e809 (0x4824 bytes) 000252 1787384235.772210 trunc from 0x01c992 to 0x02b196 (0xe805 bytes) 000253 1787384235.799007 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3084f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3084f): Mirror 2: 0x757e355d (stale) CRC-32 checksum value for chunk [0, 0x3084f): Mirror 1: 0x92f59df8 Mirror 3: 0x92f59df8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3084f: skipped 000254 1787384237.141528 mapwrite 0x00f2ce thru 0x01c396 (0xd0c9 bytes) 000256 1787384237.342476 trunc from 0x03084f to 0x03d391 (0xcb43 bytes) 000257 1787384237.368881 punch from 0x00d882 to 0x00fd13 (0x2492 bytes) 000258 1787384237.374330 mapread 0x030eb5 thru 0x03d391 (0xc4dd bytes) 000259 1787384237.392532 trunc from 0x029c3c to 0x0352c7 (0xb68c bytes) 000260 1787384237.397260 read 0x022016 thru 0x02d735 (0xb720 bytes) 000261 1787384237.406166 mapread 0x00f8fe thru 0x014084 (0x4787 bytes) 000262 1787384237.417964 read 0x009a0e thru 0x0112eb (0x78de bytes) 000263 1787384237.418746 fallocate 0x0398ee thru 0x03fff6 (0x6709 bytes) 000264 1787384237.432180 fallocate 0x01515c thru 0x01608d (0xf32 bytes) 000265 1787384237.439759 mapwrite 0x013bce thru 0x021c0a (0xe03d bytes) 000266 1787384237.489099 trunc from 0x03b654 to 0x03fff6 (0x49a3 bytes) 000267 1787384237.508553 read 0x00f184 thru 0x01bb30 (0xc9ad bytes) 000268 1787384237.509486 write 0x022ab9 thru 0x030999 (0xdee1 bytes) 000269 1787384237.544832 fallocate 0x02b30a thru 0x02c686 (0x137d bytes) 000270 1787384237.568853 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fff7): Mirror 3: 0x677b0bf2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fff7): Mirror 3: 0x677b0bf2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fff7: skipped 000271 1787384237.871419 write 0x025f2b thru 0x0320f7 (0xc1cd bytes) 000272 1787384237.889927 fallocate 0x024574 thru 0x034297 (0xfd24 bytes) 000273 1787384237.907843 mapwrite 0x00bdeb thru 0x015291 (0x94a7 bytes) 000275 1787384237.964170 trunc from 0x03fff7 to 0x02395d (0x1c699 bytes) 000277 1787384237.984620 punch from 0x011af6 to 0x019461 (0x796c bytes) 000278 1787384237.991636 punch from 0x0037e2 to 0x00ac2d (0x744c bytes) 000279 1787384237.997648 trunc from 0x02395e to 0x03c27b (0x1891e bytes) 000280 1787384238.016343 punch from 0x014d6e to 0x016bb4 (0x1e47 bytes) 000281 1787384238.023418 read 0x014ac9 thru 0x01ff9a (0xb4d2 bytes) 000282 1787384238.038800 read 0x028a73 thru 0x030079 (0x7607 bytes) 000283 1787384238.067398 mapread 0x02d4ce thru 0x02e9c6 (0x14f9 bytes) 000284 1787384238.068443 trunc from 0x017995 to 0x018f6c (0x15d8 bytes) 000285 1787384238.075926 trunc from 0x03c27c to 0x03db31 (0x18b6 bytes) 000286 1787384238.103376 trunc from 0x03db32 to 0x02a399 (0x13798 bytes) 000287 1787384238.160495 fallocate 0x026367 thru 0x02e67d (0x8317 bytes) 000288 1787384238.166706 punch from 0x001ed6 to 0x003543 (0x166e bytes) 000289 1787384238.199644 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e67e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e67e): Mirror 3: 0xd4ab008d (stale) CRC-32 checksum value for chunk [0, 0x2e67e): Mirror 1: 0xf0f6b5e1 Mirror 4: 0xf0f6b5e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e67e: skipped 000290 1787384239.010557 mapread 0x0271a5 thru 0x02e67d (0x74d9 bytes) 000291 1787384239.040790 trunc from 0x02e67e to 0x02e600 (0x7d bytes) 000292 1787384239.114378 fallocate 0x02098d thru 0x02c896 (0xbf0a bytes) 000293 1787384239.121389 trunc from 0x02352a to 0x02983c (0x6313 bytes) 000294 1787384239.126809 punch from 0x01e163 to 0x02cb83 (0xea21 bytes) 000295 1787384239.132228 mapwrite 0x0007d3 thru 0x00cf39 (0xc767 bytes) 000296 1787384239.172107 fallocate 0x026cb5 thru 0x0369bb (0xfd07 bytes) 000297 1787384239.177493 write 0x0073b2 thru 0x0097f3 (0x2442 bytes) 000298 1787384239.179609 punch from 0x01f3e5 to 0x026193 (0x6daf bytes) 000299 1787384239.184293 read 0x025f73 thru 0x02e600 (0x868e bytes) 000300 1787384239.197570 MIRROR_RESYNC 3 mirrors 000301 1787384239.887661 MIRROR_RESYNC 3 mirrors 000302 1787384240.347301 punch from 0x00ed14 to 0x013a69 (0x4d56 bytes) 000303 1787384240.385511 trunc from 0x011fad to 0x017d03 (0x5d57 bytes) 000304 1787384240.391218 trunc from 0x02e601 to 0x00dd01 (0x208ff bytes) 000306 1787384240.456424 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xdd02): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd02): Mirror 3: 0x15aecb6f (stale) Mirror 4: 0x15aecb6f (stale) CRC-32 checksum value for chunk [0, 0xdd02): Mirror 1: 0x15aecb6f Mirror 5: 0x15aecb6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd02: skipped 000307 1787384241.360115 mapwrite 0x03b516 thru 0x03ffff (0x4aea bytes) 000308 1787384241.466066 fallocate 0x03279e thru 0x03f247 (0xcaaa bytes) 000309 1787384241.475978 fallocate 0x02f372 thru 0x03c103 (0xcd92 bytes) 000310 1787384241.486082 fallocate 0x003cae thru 0x005547 (0x189a bytes) 000311 1787384241.491375 write 0x031061 thru 0x0382fa (0x729a bytes) 000312 1787384241.499770 trunc from 0x02777a to 0x029be5 (0x246c bytes) 000313 1787384241.505775 mapwrite 0x006dc0 thru 0x007ba3 (0xde4 bytes) 000314 1787384241.552045 mapread 0x00ac0d thru 0x00e821 (0x3c15 bytes) 000315 1787384241.554069 trunc from 0x02fd59 to 0x032308 (0x25b0 bytes) 000316 1787384241.560994 read 0x029f2d thru 0x02d714 (0x37e8 bytes) 000317 1787384241.575432 punch from 0x02696f to 0x02d35f (0x69f1 bytes) 000318 1787384241.580174 trunc from 0x033fa7 to 0x038529 (0x4583 bytes) 000319 1787384241.590938 write 0x0191ab thru 0x02507a (0xbed0 bytes) 000321 1787384241.602967 trunc from 0x02ad46 to 0x02f583 (0x483e bytes) 000322 1787384241.613687 write 0x0388d1 thru 0x03ffff (0x772f bytes) 000323 1787384241.642859 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x19069bb4 (stale) Mirror 4: 0x19069bb4 (stale) Mirror 5: 0xc572c5ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b8dc543 Mirror 6: 0x9b8dc543 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000324 1787384242.849173 fallocate 0x021919 thru 0x022518 (0xc00 bytes) 000326 1787384242.905783 fallocate 0x0017f7 thru 0x00bd98 (0xa5a2 bytes) 000327 1787384242.912267 read 0x006a8c thru 0x008629 (0x1b9e bytes) 000328 1787384242.919207 punch from 0x012caa to 0x0142f8 (0x164f bytes) 000329 1787384242.925256 trunc from 0x040000 to 0x004a70 (0x3b58f bytes) 000330 1787384242.961328 punch from 0x0034b0 to 0x004a70 (0x15c1 bytes) 000331 1787384242.967503 read 0x001662 thru 0x004a70 (0x340f bytes) 000332 1787384242.977548 trunc from 0x000e8a to 0x004a70 (0x3be7 bytes) 000333 1787384242.984822 write 0x03f27d thru 0x03ffff (0xd83 bytes) 000334 1787384242.987164 mapread 0x000af7 thru 0x00e722 (0xdc2c bytes) 000335 1787384243.021367 trunc from 0x00654e to 0x0136fa (0xd1ad bytes) 000336 1787384243.027827 mapwrite 0x00c778 thru 0x0195d7 (0xce60 bytes) 000338 1787384243.113096 trunc from 0x038a7b to 0x03be3c (0x33c2 bytes) 000339 1787384243.118646 mapwrite 0x001304 thru 0x009d9b (0x8a98 bytes) 000340 1787384243.162220 mapwrite 0x010f06 thru 0x01b377 (0xa472 bytes) 000341 1787384243.189380 trunc from 0x040000 to 0x00b26e (0x34d91 bytes) 000342 1787384243.215771 write 0x01af2a thru 0x0238ae (0x8985 bytes) 000344 1787384243.220881 trunc from 0x0080b3 to 0x00f318 (0x7266 bytes) 000345 1787384243.233613 trunc from 0x0238af to 0x029058 (0x57aa bytes) 000346 1787384243.252166 punch from 0x01b37a to 0x02445e (0x90e5 bytes) 000347 1787384243.272880 mapread 0x012cbc thru 0x0177e9 (0x4b2e bytes) 000348 1787384243.301049 fallocate 0x035bc3 thru 0x03e527 (0x8965 bytes) 000349 1787384243.306178 mapread 0x02b327 thru 0x02e0e3 (0x2dbd bytes) 000350 1787384243.338440 mapread 0x00288c thru 0x00e2de (0xba53 bytes) 000351 1787384243.383073 punch from 0x030408 to 0x03bcce (0xb8c7 bytes) 000352 1787384243.389154 fallocate 0x013dc0 thru 0x022a07 (0xec48 bytes) 000353 1787384243.397330 mapread 0x00803a thru 0x00dc15 (0x5bdc bytes) 000354 1787384243.405069 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e528): Mirror 3: 0x9fc60fc3 (stale) Mirror 4: 0x9fc60fc3 (stale) Mirror 6: 0x9a156cae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e528): Mirror 3: 0x9fc60fc3 (stale) Mirror 4: 0x9fc60fc3 (stale) Mirror 6: 0x9a156cae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e528: skipped 000355 1787384243.813208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e528): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e528): Mirror 3: 0x9fc60fc3 (stale) Mirror 4: 0x9fc60fc3 (stale) Mirror 6: 0x9a156cae (stale) CRC-32 checksum value for chunk [0, 0x3e528): Mirror 1: 0x1326ba6b Mirror 7: 0x1326ba6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e528: skipped 000356 1787384244.911918 fallocate 0x029acd thru 0x037c6d (0xe1a1 bytes) 000357 1787384244.966118 trunc from 0x0000a7 to 0x000ab7 (0xa11 bytes) 000359 1787384244.974084 write 0x024cff thru 0x032e92 (0xe194 bytes) 000360 1787384244.992879 fallocate 0x0052fa thru 0x010050 (0xad57 bytes) 000361 1787384245.003303 mapwrite 0x03b155 thru 0x03ffff (0x4eab bytes) 000362 1787384245.128481 trunc from 0x040000 to 0x00850b (0x37af4 bytes) 000363 1787384245.151874 mapread 0x002b92 thru 0x005178 (0x25e7 bytes) 000364 1787384245.165624 fallocate 0x00d5f3 thru 0x016d32 (0x9740 bytes) 000365 1787384245.171637 fallocate 0x01d1c8 thru 0x027554 (0xa38d bytes) 000366 1787384245.178618 trunc from 0x016d33 to 0x003033 (0x13cff bytes) 000367 1787384245.206362 read 0x00113c thru 0x003033 (0x1ef8 bytes) 000368 1787384245.207514 mapread 0x002e7a thru 0x003033 (0x1ba bytes) 000369 1787384245.210524 mapread 0x000adc thru 0x003033 (0x2558 bytes) 000370 1787384245.220082 write 0x016010 thru 0x01c311 (0x6302 bytes) 000371 1787384245.223504 write 0x01c06c thru 0x01eee7 (0x2e7c bytes) 000372 1787384245.228353 mapread 0x01178d thru 0x012bca (0x143e bytes) 000373 1787384245.241973 mapwrite 0x0302b0 thru 0x030879 (0x5ca bytes) 000374 1787384245.300539 mapread 0x026748 thru 0x02bd61 (0x561a bytes) 000375 1787384245.320050 write 0x022e9a thru 0x024203 (0x136a bytes) 000376 1787384245.323558 write 0x039f6c thru 0x03ffff (0x6094 bytes) 000377 1787384245.328144 punch from 0x03dc40 to 0x03ffff (0x23c0 bytes) 000378 1787384245.355210 write 0x00f2d8 thru 0x012dec (0x3b15 bytes) 000379 1787384245.358330 read 0x00e316 thru 0x01b819 (0xd504 bytes) 000380 1787384245.359525 write 0x03ef8b thru 0x03ffff (0x1075 bytes) 000381 1787384245.378176 punch from 0x00619a to 0x00cf8f (0x6df6 bytes) 000382 1787384245.385326 write 0x022dff thru 0x026197 (0x3399 bytes) 000383 1787384245.386658 fallocate 0x01b167 thru 0x02af18 (0xfdb2 bytes) 000384 1787384245.400876 punch from 0x0382f8 to 0x03c3c8 (0x40d1 bytes) 000385 1787384245.416555 mapread 0x03ac44 thru 0x03cec0 (0x227d bytes) 000386 1787384245.494616 punch from 0x01e474 to 0x022ef9 (0x4a86 bytes) 000387 1787384245.536320 fallocate 0x008c8f thru 0x0172d2 (0xe644 bytes) 000388 1787384245.549422 mapread 0x027c3c thru 0x0365a9 (0xe96e bytes) 000389 1787384245.557784 trunc from 0x040000 to 0x024148 (0x1beb7 bytes) 000390 1787384245.591384 write 0x02b075 thru 0x02f1e8 (0x4174 bytes) 000391 1787384245.601335 write 0x007e95 thru 0x017636 (0xf7a2 bytes) 000392 1787384245.619061 mapwrite 0x02f372 thru 0x02f7a6 (0x435 bytes) 000394 1787384245.679022 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f7a7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f7a7): Mirror 3: 0x53c7e17f (stale) Mirror 4: 0x53c7e17f (stale) Mirror 6: 0xec0664e4 (stale) Mirror 7: 0x74de88dc (stale) CRC-32 checksum value for chunk [0, 0x2f7a7): Mirror 1: 0xb8d37c91 Mirror 8: 0xb8d37c91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f7a7: skipped 000395 1787384246.800803 write 0x00292d thru 0x003fc0 (0x1694 bytes) 000399 1787384246.871368 trunc from 0x02f7a7 to 0x029d2e (0x5a78 bytes) 000400 1787384246.902837 punch from 0x002ca7 to 0x0034ea (0x844 bytes) 000401 1787384246.916062 write 0x00fc31 thru 0x012575 (0x2945 bytes) 000402 1787384246.930298 mapwrite 0x0182d4 thru 0x027dbd (0xfaea bytes) 000403 1787384247.012528 fallocate 0x036667 thru 0x03715f (0xaf9 bytes) 000404 1787384247.019315 mapread 0x011a85 thru 0x018326 (0x68a2 bytes) 000405 1787384247.022333 fallocate 0x0323ef thru 0x03ffff (0xdc11 bytes) 000407 1787384247.030386 mapread 0x028c70 thru 0x029d2e (0x10bf bytes) 000409 1787384247.035177 mapread 0x00ffb3 thru 0x01604c (0x609a bytes) 000410 1787384247.037510 trunc from 0x029d2f to 0x0202ed (0x9a41 bytes) 000411 1787384247.062100 mapwrite 0x0085f0 thru 0x00c30a (0x3d1b bytes) 000412 1787384247.174293 trunc from 0x00bd4f to 0x00e8af (0x2b61 bytes) 000414 1787384247.193994 trunc from 0x017d48 to 0x01bc3e (0x3ef7 bytes) 000415 1787384247.213697 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x202ee): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202ee): Mirror 3: 0xe56d8820 (stale) Mirror 4: 0xe56d8820 (stale) Mirror 6: 0xefd35b90 (stale) Mirror 7: 0xa8d19f28 (stale) Mirror 8: 0x7ddb623d (stale) CRC-32 checksum value for chunk [0, 0x202ee): Mirror 1: 0xca070acf Mirror 9: 0xca070acf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202ee: skipped 000418 1787384248.266398 trunc from 0x011a2b to 0x0202ed (0xe8c3 bytes) 000419 1787384248.312004 trunc from 0x019943 to 0x01fd22 (0x63e0 bytes) 000420 1787384248.316998 punch from 0x017667 to 0x01817e (0xb18 bytes) 000421 1787384248.322342 mapwrite 0x001cb6 thru 0x0099ec (0x7d37 bytes) 000422 1787384248.359828 write 0x034805 thru 0x036e31 (0x262d bytes) 000423 1787384248.364423 fallocate 0x03d48b thru 0x03d828 (0x39e bytes) 000424 1787384248.369727 read 0x023c4e thru 0x02fce4 (0xc097 bytes) 000425 1787384248.377591 mapread 0x026c2a thru 0x032f7f (0xc356 bytes) 000426 1787384248.384212 punch from 0x025430 to 0x025460 (0x31 bytes) 000427 1787384248.390122 fallocate 0x00fb60 thru 0x019df6 (0xa297 bytes) 000428 1787384248.395122 trunc from 0x036e32 to 0x01b965 (0x1b4cc bytes) 000429 1787384248.440839 mapread 0x015791 thru 0x018761 (0x2fd1 bytes) 000430 1787384248.456619 fallocate 0x03ba46 thru 0x03d7e5 (0x1da0 bytes) 000431 1787384248.463133 mapread 0x00ee8c thru 0x014a12 (0x5b87 bytes) 000432 1787384248.465067 read 0x00eb61 thru 0x01cf5a (0xe3fa bytes) 000433 1787384248.479440 mapread 0x00cbc9 thru 0x01a30d (0xd745 bytes) 000434 1787384248.485528 MIRROR_RESYNC 7 mirrors 000435 1787384250.075413 write 0x01b87f thru 0x024e65 (0x95e7 bytes) 000436 1787384250.118463 read 0x014645 thru 0x021e75 (0xd831 bytes) 000437 1787384250.128633 punch from 0x00f4e3 to 0x0184a7 (0x8fc5 bytes) 000438 1787384250.134177 fallocate 0x03c8fa thru 0x03ffff (0x3706 bytes) 000439 1787384250.139232 mapwrite 0x027024 thru 0x0316d2 (0xa6af bytes) 000440 1787384250.191885 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d7e6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7e6): Mirror 3: 0xae8cb33b (stale) Mirror 4: 0xae8cb33b (stale) Mirror 6: 0xae8cb33b (stale) Mirror 7: 0xae8cb33b (stale) Mirror 8: 0xae8cb33b (stale) Mirror 9: 0xae8cb33b (stale) CRC-32 checksum value for chunk [0, 0x3d7e6): Mirror 1: 0x32c5f713 Mirror 10: 0x32c5f713 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7e6: skipped 000441 1787384251.368031 read 0x02c8f7 thru 0x02e676 (0x1d80 bytes) 000442 1787384251.389427 punch from 0x037410 to 0x03d7e5 (0x63d6 bytes) 000443 1787384251.453078 mapread 0x00dd28 thru 0x012beb (0x4ec4 bytes) 000444 1787384251.500568 write 0x01ced7 thru 0x0259da (0x8b04 bytes) 000447 1787384251.542714 mapwrite 0x01733f thru 0x023245 (0xbf07 bytes) 000448 1787384251.602605 write 0x00d091 thru 0x0153c9 (0x8339 bytes) 000449 1787384251.608148 read 0x001331 thru 0x0095d2 (0x82a2 bytes) 000451 1787384251.635430 fallocate 0x036a3e thru 0x03ffff (0x95c2 bytes) 000452 1787384251.647972 read 0x02b58c thru 0x038b78 (0xd5ed bytes) 000453 1787384251.649812 punch from 0x0246de to 0x02a894 (0x61b7 bytes) 000454 1787384251.660522 read 0x01ffb3 thru 0x028e17 (0x8e65 bytes) 000456 1787384251.687609 MIRROR_RESYNC 8 mirrors 000457 1787384254.802708 trunc from 0x03d7e6 to 0x003d14 (0x39ad1 bytes) 000458 1787384254.896201 mapwrite 0x006bf8 thru 0x011259 (0xa662 bytes) 000459 1787384255.052613 read 0x00a436 thru 0x011259 (0x6e24 bytes) 000460 1787384255.054621 punch from 0x002596 to 0x00bf63 (0x99ce bytes) 000461 1787384255.061545 trunc from 0x01125a to 0x015405 (0x41ac bytes) 000462 1787384255.080792 read 0x00beca thru 0x015405 (0x953c bytes) 000463 1787384255.096523 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x15406): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15406): Mirror 3: 0xa93dca5b (stale) Mirror 4: 0xa93dca5b (stale) Mirror 6: 0xa93dca5b (stale) Mirror 7: 0xa93dca5b (stale) Mirror 8: 0xa93dca5b (stale) Mirror 9: 0xa93dca5b (stale) Mirror 10: 0xa93dca5b (stale) CRC-32 checksum value for chunk [0, 0x15406): Mirror 1: 0x251931e2 Mirror 11: 0x251931e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15406: skipped 000464 1787384256.137977 write 0x03120b thru 0x03ffff (0xedf5 bytes) 000465 1787384256.181118 mapread 0x03eef0 thru 0x03ffff (0x1110 bytes) 000466 1787384256.184818 mapwrite 0x031e98 thru 0x039eb2 (0x801b bytes) 000467 1787384256.203563 mapwrite 0x028b20 thru 0x02939d (0x87e bytes) 000468 1787384256.228585 mapread 0x0247c2 thru 0x02cf14 (0x8753 bytes) 000469 1787384256.246261 fallocate 0x03cb63 thru 0x03d4ad (0x94b bytes) 000470 1787384256.252500 mapread 0x02bed2 thru 0x02e567 (0x2696 bytes) 000471 1787384256.260147 trunc from 0x040000 to 0x00a33c (0x35cc3 bytes) 000472 1787384256.326070 mapwrite 0x00bc29 thru 0x011710 (0x5ae8 bytes) 000473 1787384256.407499 fallocate 0x00a053 thru 0x00cb45 (0x2af3 bytes) 000474 1787384256.413662 read 0x00f101 thru 0x011710 (0x2610 bytes) 000475 1787384256.415853 mapread 0x0053bd thru 0x007445 (0x2089 bytes) 000476 1787384256.418346 mapread 0x011068 thru 0x011710 (0x6a9 bytes) 000477 1787384256.420569 punch from 0x00a822 to 0x011710 (0x6eef bytes) 000478 1787384256.430969 punch from 0x00e59c to 0x011710 (0x3175 bytes) 000479 1787384256.441674 mapwrite 0x038bbd thru 0x03ffff (0x7443 bytes) 000480 1787384256.509737 mapread 0x02e7ca thru 0x02eb52 (0x389 bytes) 000481 1787384256.519398 mapread 0x03804a thru 0x03ffff (0x7fb6 bytes) 000482 1787384256.524051 trunc from 0x03e380 to 0x03ffff (0x1c80 bytes) 000483 1787384256.534041 trunc from 0x013f98 to 0x01ceea (0x8f53 bytes) 000484 1787384256.548056 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe1b963d3 (stale) Mirror 4: 0xe1b963d3 (stale) Mirror 6: 0xe1b963d3 (stale) Mirror 7: 0xe1b963d3 (stale) Mirror 8: 0xe1b963d3 (stale) Mirror 9: 0xe1b963d3 (stale) Mirror 10: 0xe1b963d3 (stale) Mirror 11: 0x660abdac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x95112b8f Mirror 12: 0x95112b8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1787384258.043589 MIRROR_RESYNC 10 mirrors 000486 1787384260.519530 mapread 0x00b907 thru 0x0114f3 (0x5bed bytes) 000487 1787384260.545130 MIRROR_RESYNC 10 mirrors 000489 1787384261.274707 trunc from 0x040000 to 0x00f2f8 (0x30d07 bytes) 000490 1787384261.353700 mapread 0x00b105 thru 0x00f2f8 (0x41f4 bytes) 000491 1787384261.383964 trunc from 0x00bda9 to 0x00f2f8 (0x3550 bytes) 000492 1787384261.399087 mapwrite 0x00bd49 thru 0x00fc47 (0x3eff bytes) 000493 1787384261.467693 read 0x004117 thru 0x00f96d (0xb857 bytes) 000495 1787384261.474780 mapread 0x0059e3 thru 0x00dd63 (0x8381 bytes) 000496 1787384261.476499 mapwrite 0x0379cb thru 0x038d79 (0x13af bytes) 000498 1787384261.563788 trunc from 0x024260 to 0x030fb1 (0xcd52 bytes) 000499 1787384261.574732 mapwrite 0x015179 thru 0x01b937 (0x67bf bytes) 000500 1787384261.601743 mapwrite 0x00c1e4 thru 0x011003 (0x4e20 bytes) 000502 1787384261.625550 mapread 0x01fe2d thru 0x021525 (0x16f9 bytes) 000503 1787384261.640616 mapread 0x003e0f thru 0x0066b0 (0x28a2 bytes) 000504 1787384261.653232 mapwrite 0x003340 thru 0x012f0a (0xfbcb bytes) 000505 1787384261.696972 read 0x01bd0e thru 0x023245 (0x7538 bytes) 000506 1787384261.698186 write 0x03e81a thru 0x03ffff (0x17e6 bytes) 000507 1787384261.707774 fallocate 0x002aa2 thru 0x0066c8 (0x3c27 bytes) 000508 1787384261.719571 trunc from 0x017f59 to 0x01e2ff (0x63a7 bytes) 000509 1787384261.726127 fallocate 0x029506 thru 0x02fd8e (0x6889 bytes) 000510 1787384261.742305 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95112b8f (stale) Mirror 4: 0x95112b8f (stale) Mirror 6: 0x95112b8f (stale) Mirror 7: 0x95112b8f (stale) Mirror 8: 0x95112b8f (stale) Mirror 9: 0x95112b8f (stale) Mirror 10: 0x95112b8f (stale) Mirror 11: 0x95112b8f (stale) Mirror 12: 0x95112b8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x476e2090 Mirror 13: 0x476e2090 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1787384263.204088 trunc from 0x040000 to 0x035e80 (0xa17f bytes) 000512 1787384263.293843 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x35e81): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e81): Mirror 3: 0xcb8506d1 (stale) Mirror 4: 0xcb8506d1 (stale) Mirror 6: 0xcb8506d1 (stale) Mirror 7: 0xcb8506d1 (stale) Mirror 8: 0xcb8506d1 (stale) Mirror 9: 0xcb8506d1 (stale) Mirror 10: 0xcb8506d1 (stale) Mirror 11: 0xcb8506d1 (stale) Mirror 12: 0xcb8506d1 (stale) Mirror 13: 0x17022000 (stale) CRC-32 checksum value for chunk [0, 0x35e81): Mirror 1: 0x17022000 Mirror 14: 0x17022000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e81: skipped 000513 1787384264.453711 write 0x037c0d thru 0x03ffff (0x83f3 bytes) 000514 1787384264.528361 write 0x028a55 thru 0x033d8a (0xb336 bytes) 000515 1787384264.543813 trunc from 0x040000 to 0x031e78 (0xe187 bytes) 000516 1787384264.563563 mapwrite 0x020ca2 thru 0x030723 (0xfa82 bytes) 000517 1787384264.655293 mapread 0x01088d thru 0x01d7d2 (0xcf46 bytes) 000518 1787384264.687077 trunc from 0x02c303 to 0x031e78 (0x5b76 bytes) 000519 1787384264.705997 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e79): Mirror 3: 0x8ed0b8ec (stale) Mirror 4: 0x8ed0b8ec (stale) Mirror 6: 0x8ed0b8ec (stale) Mirror 7: 0x8ed0b8ec (stale) Mirror 8: 0x8ed0b8ec (stale) Mirror 9: 0x8ed0b8ec (stale) Mirror 10: 0x8ed0b8ec (stale) Mirror 12: 0x8ed0b8ec (stale) Mirror 13: 0xba2a4ec7 (stale) Mirror 14: 0xba2a4ec7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e79): Mirror 3: 0x8ed0b8ec (stale) Mirror 4: 0x8ed0b8ec (stale) Mirror 6: 0x8ed0b8ec (stale) Mirror 7: 0x8ed0b8ec (stale) Mirror 8: 0x8ed0b8ec (stale) Mirror 9: 0x8ed0b8ec (stale) Mirror 10: 0x8ed0b8ec (stale) Mirror 12: 0x8ed0b8ec (stale) Mirror 13: 0xba2a4ec7 (stale) Mirror 14: 0xba2a4ec7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e79: skipped 000520 1787384265.484210 punch from 0x02f84a to 0x031e78 (0x262f bytes) 000521 1787384265.491707 punch from 0x003be4 to 0x003cc3 (0xe0 bytes) 000522 1787384265.496952 mapread 0x0270b3 thru 0x02e50d (0x745b bytes) 000523 1787384265.514763 mapwrite 0x0351a1 thru 0x03ae21 (0x5c81 bytes) 000524 1787384265.563654 write 0x0151b3 thru 0x02022d (0xb07b bytes) 000525 1787384265.579388 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae22): Mirror 3: 0xe33d2f98 (stale) Mirror 4: 0xe33d2f98 (stale) Mirror 6: 0xe33d2f98 (stale) Mirror 7: 0xe33d2f98 (stale) Mirror 8: 0xe33d2f98 (stale) Mirror 9: 0xe33d2f98 (stale) Mirror 12: 0xe33d2f98 (stale) Mirror 13: 0x8bcfa7c9 (stale) Mirror 14: 0xdf9ec702 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae22): Mirror 3: 0xe33d2f98 (stale) Mirror 4: 0xe33d2f98 (stale) Mirror 6: 0xe33d2f98 (stale) Mirror 7: 0xe33d2f98 (stale) Mirror 8: 0xe33d2f98 (stale) Mirror 9: 0xe33d2f98 (stale) Mirror 12: 0xe33d2f98 (stale) Mirror 13: 0x8bcfa7c9 (stale) Mirror 14: 0xdf9ec702 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae22: skipped 000526 1787384266.257773 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae22): Mirror 3: 0xe33d2f98 (stale) Mirror 4: 0xe33d2f98 (stale) Mirror 6: 0xe33d2f98 (stale) Mirror 7: 0xe33d2f98 (stale) Mirror 8: 0xe33d2f98 (stale) Mirror 9: 0xe33d2f98 (stale) Mirror 13: 0x8bcfa7c9 (stale) Mirror 14: 0xdf9ec702 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae22): Mirror 3: 0xe33d2f98 (stale) Mirror 4: 0xe33d2f98 (stale) Mirror 6: 0xe33d2f98 (stale) Mirror 7: 0xe33d2f98 (stale) Mirror 8: 0xe33d2f98 (stale) Mirror 9: 0xe33d2f98 (stale) Mirror 13: 0x8bcfa7c9 (stale) Mirror 14: 0xdf9ec702 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae22: skipped 000527 1787384266.827043 fallocate 0x0057ca thru 0x011951 (0xc188 bytes) 000528 1787384266.832476 trunc from 0x03ae22 to 0x0098e8 (0x31539 bytes) 000530 1787384266.856871 mapread 0x001e24 thru 0x004fef (0x31cc bytes) 000531 1787384266.892774 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x98e9): Mirror 4: 0xd201efcb (stale) Mirror 6: 0xd201efcb (stale) Mirror 7: 0xd201efcb (stale) Mirror 8: 0xd201efcb (stale) Mirror 9: 0xd201efcb (stale) Mirror 13: 0x36625237 (stale) Mirror 14: 0x36625237 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x98e9): Mirror 4: 0xd201efcb (stale) Mirror 6: 0xd201efcb (stale) Mirror 7: 0xd201efcb (stale) Mirror 8: 0xd201efcb (stale) Mirror 9: 0xd201efcb (stale) Mirror 13: 0x36625237 (stale) Mirror 14: 0x36625237 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x98e9: skipped 000532 1787384267.341075 MIRROR_RESYNC 8 mirrors 000533 1787384268.358306 write 0x00bd4d thru 0x015cea (0x9f9e bytes) 000535 1787384268.402057 write 0x010d06 thru 0x012125 (0x1420 bytes) 000536 1787384268.403106 mapread 0x00d857 thru 0x015cea (0x8494 bytes) 000537 1787384268.413776 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x15ceb): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15ceb): Mirror 4: 0xbda8063e (stale) Mirror 6: 0xbda8063e (stale) Mirror 7: 0xbda8063e (stale) Mirror 8: 0xbda8063e (stale) Mirror 9: 0xbda8063e (stale) Mirror 13: 0xbda8063e (stale) Mirror 14: 0xbda8063e (stale) CRC-32 checksum value for chunk [0, 0x15ceb): Mirror 1: 0x19d44a26 Mirror 15: 0x19d44a26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15ceb: skipped 000538 1787384269.433513 MIRROR_RESYNC 9 mirrors 000539 1787384271.019724 trunc from 0x001b1a to 0x008f53 (0x743a bytes) 000540 1787384271.080786 write 0x028eb3 thru 0x038cb4 (0xfe02 bytes) 000541 1787384271.086509 mapwrite 0x00f8be thru 0x0191d2 (0x9915 bytes) 000542 1787384271.159670 trunc from 0x038cb5 to 0x012556 (0x2675e bytes) 000543 1787384271.209164 mapwrite 0x002f25 thru 0x003b54 (0xc30 bytes) 000544 1787384271.230777 trunc from 0x0068ea to 0x00ad21 (0x4438 bytes) 000545 1787384271.236869 trunc from 0x00462a to 0x00a6a5 (0x607c bytes) 000546 1787384271.242787 write 0x010b7d thru 0x0159d6 (0x4e5a bytes) 000547 1787384271.245390 write 0x011553 thru 0x012c58 (0x1706 bytes) 000548 1787384271.246688 mapwrite 0x033953 thru 0x03f05f (0xb70d bytes) 000549 1787384271.293787 mapread 0x002c89 thru 0x005c50 (0x2fc8 bytes) 000550 1787384271.312130 write 0x0071a8 thru 0x00ebc3 (0x7a1c bytes) 000551 1787384271.313616 fallocate 0x015619 thru 0x022c77 (0xd65f bytes) 000552 1787384271.325224 mapread 0x013892 thru 0x01bfa7 (0x8716 bytes) 000553 1787384271.348203 MIRROR_RESYNC 9 mirrors 000554 1787384273.187255 mapwrite 0x01badb thru 0x0280b9 (0xc5df bytes) 000555 1787384273.350509 mapwrite 0x03fccb thru 0x03ffff (0x335 bytes) 000556 1787384273.427064 trunc from 0x01e892 to 0x025d16 (0x7485 bytes) 000557 1787384273.434117 mapread 0x020b6b thru 0x027a99 (0x6f2f bytes) 000558 1787384273.451766 punch from 0x021748 to 0x025836 (0x40ef bytes) 000559 1787384273.463379 mapread 0x025bf3 thru 0x02c6d0 (0x6ade bytes) 000560 1787384273.480840 fallocate 0x00228f thru 0x010af6 (0xe868 bytes) 000561 1787384273.488465 mapread 0x03ad87 thru 0x03ffff (0x5279 bytes) 000562 1787384273.491854 read 0x005803 thru 0x0121dd (0xc9db bytes) 000563 1787384273.507329 read 0x0237f0 thru 0x025362 (0x1b73 bytes) 000564 1787384273.508652 read 0x02c725 thru 0x03c603 (0xfedf bytes) 000565 1787384273.528468 fallocate 0x00fa51 thru 0x013e57 (0x4407 bytes) 000566 1787384273.534730 mapwrite 0x03ba00 thru 0x03ffff (0x4600 bytes) 000568 1787384273.593987 punch from 0x013272 to 0x013f30 (0xcbf bytes) 000569 1787384273.612206 punch from 0x01a4e6 to 0x02811e (0xdc39 bytes) 000571 1787384273.622474 mapread 0x01a9b5 thru 0x01c150 (0x179c bytes) 000572 1787384273.655094 mapread 0x016744 thru 0x021de7 (0xb6a4 bytes) 000573 1787384273.690813 write 0x017b9a thru 0x019892 (0x1cf9 bytes) 000574 1787384273.692567 read 0x00f72b thru 0x01b977 (0xc24d bytes) 000575 1787384273.702635 read 0x010765 thru 0x018fee (0x888a bytes) 000576 1787384273.703674 trunc from 0x003e2b to 0x009c66 (0x5e3c bytes) 000577 1787384273.727141 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe37e37a8 (stale) Mirror 6: 0xe37e37a8 (stale) Mirror 7: 0xe37e37a8 (stale) Mirror 9: 0xe37e37a8 (stale) Mirror 13: 0xe37e37a8 (stale) Mirror 14: 0xe37e37a8 (stale) Mirror 15: 0xe37e37a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe37e37a8 (stale) Mirror 6: 0xe37e37a8 (stale) Mirror 7: 0xe37e37a8 (stale) Mirror 9: 0xe37e37a8 (stale) Mirror 13: 0xe37e37a8 (stale) Mirror 14: 0xe37e37a8 (stale) Mirror 15: 0xe37e37a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000578 1787384274.331312 trunc from 0x040000 to 0x03160f (0xe9f0 bytes) 000579 1787384274.363771 fallocate 0x026270 thru 0x02bbf5 (0x5986 bytes) 000580 1787384274.368939 mapwrite 0x00d237 thru 0x013cfe (0x6ac8 bytes) 000581 1787384274.413062 write 0x0225f6 thru 0x02ac3d (0x8648 bytes) 000582 1787384274.426953 mapread 0x00f94f thru 0x01a678 (0xad2a bytes) 000583 1787384274.453484 MIRROR_RESYNC 8 mirrors 000585 1787384276.634697 write 0x02afee thru 0x02f867 (0x487a bytes) 000586 1787384276.680790 trunc from 0x031610 to 0x02a2a3 (0x736c bytes) 000587 1787384276.711827 mapread 0x00dd05 thru 0x019b91 (0xbe8d bytes) 000589 1787384276.755440 fallocate 0x003776 thru 0x00db10 (0xa39b bytes) 000590 1787384276.762570 read 0x023c26 thru 0x02a2a3 (0x667e bytes) 000591 1787384276.789447 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2a4): Mirror 4: 0xa718757 (stale) Mirror 6: 0xa718757 (stale) Mirror 7: 0xa718757 (stale) Mirror 9: 0xa718757 (stale) Mirror 14: 0xa718757 (stale) Mirror 15: 0xa718757 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a4): Mirror 4: 0xa718757 (stale) Mirror 6: 0xa718757 (stale) Mirror 7: 0xa718757 (stale) Mirror 9: 0xa718757 (stale) Mirror 14: 0xa718757 (stale) Mirror 15: 0xa718757 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a4: skipped 000592 1787384277.241094 trunc from 0x0163a1 to 0x020a89 (0xa6e9 bytes) 000593 1787384277.247402 read 0x010e1c thru 0x014f53 (0x4138 bytes) 000594 1787384277.256572 mapread 0x01a691 thru 0x02142f (0x6d9f bytes) 000595 1787384277.297420 read 0x0279cc thru 0x02a2a3 (0x28d8 bytes) 000596 1787384277.308797 trunc from 0x020a19 to 0x028eaf (0x8497 bytes) 000597 1787384277.317440 trunc from 0x01739c to 0x02165f (0xa2c4 bytes) 000598 1787384277.333980 MIRROR_RESYNC 7 mirrors 000599 1787384278.898558 punch from 0x0233dd to 0x02a2a3 (0x6ec7 bytes) 000600 1787384278.942931 trunc from 0x01f726 to 0x027be5 (0x84c0 bytes) 000601 1787384278.951529 mapread 0x029255 thru 0x02a2a3 (0x104f bytes) 000602 1787384278.973138 fallocate 0x01dc02 thru 0x022663 (0x4a62 bytes) 000603 1787384278.995033 mapread 0x0206cd thru 0x0260a0 (0x59d4 bytes) 000604 1787384279.019735 fallocate 0x018589 thru 0x024fdb (0xca53 bytes) 000605 1787384279.028657 mapread 0x025d4e thru 0x02a2a3 (0x4556 bytes) 000607 1787384279.034842 trunc from 0x02a2a4 to 0x01eae0 (0xb7c3 bytes) 000608 1787384279.071395 trunc from 0x015e90 to 0x016598 (0x709 bytes) 000609 1787384279.078141 fallocate 0x00830b thru 0x00a6a9 (0x239f bytes) 000610 1787384279.096842 trunc from 0x010fbc to 0x01523a (0x427f bytes) 000611 1787384279.116353 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1eae1): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eae1): Mirror 4: 0x99a9875 (stale) Mirror 6: 0x99a9875 (stale) Mirror 7: 0x99a9875 (stale) Mirror 9: 0x99a9875 (stale) Mirror 14: 0x99a9875 (stale) Mirror 15: 0x99a9875 (stale) CRC-32 checksum value for chunk [0, 0x1eae1): Mirror 1: 0xfbb2a208 Mirror 16: 0xfbb2a208 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eae1: skipped 000612 1787384280.358223 trunc from 0x01eae1 to 0x0056c5 (0x1941b bytes) 000613 1787384280.415242 write 0x03e1fa thru 0x03ffff (0x1e06 bytes) 000614 1787384280.417927 trunc from 0x040000 to 0x02cd74 (0x1328b bytes) 000615 1787384280.438657 write 0x01c6d4 thru 0x02b717 (0xf044 bytes) 000616 1787384280.450635 punch from 0x0298ba to 0x02cd74 (0x34bb bytes) 000617 1787384280.466333 mapread 0x0257e8 thru 0x02cd74 (0x758d bytes) 000618 1787384280.482413 write 0x035ba5 thru 0x03ffff (0xa45b bytes) 000619 1787384280.485811 read 0x0328f9 thru 0x037b91 (0x5299 bytes) 000622 1787384280.492514 trunc from 0x033716 to 0x03e54d (0xae38 bytes) 000623 1787384280.510388 read 0x0130c8 thru 0x01aab0 (0x79e9 bytes) 000624 1787384280.523234 trunc from 0x040000 to 0x026294 (0x19d6b bytes) 000625 1787384280.537101 fallocate 0x0314e4 thru 0x03c0c2 (0xabdf bytes) 000626 1787384280.543364 fallocate 0x0291d6 thru 0x02c1fb (0x3026 bytes) 000627 1787384280.553740 fallocate 0x02332c thru 0x02d7e1 (0xa4b6 bytes) 000628 1787384280.564023 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c0c3): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0c3): Mirror 4: 0xda13c20e (stale) Mirror 6: 0xda13c20e (stale) Mirror 7: 0xda13c20e (stale) Mirror 9: 0xda13c20e (stale) Mirror 14: 0xda13c20e (stale) Mirror 15: 0xda13c20e (stale) Mirror 16: 0xf2fd126f (stale) CRC-32 checksum value for chunk [0, 0x3c0c3): Mirror 1: 0x36b01257 Mirror 17: 0x36b01257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0c3: skipped 000629 1787384281.597161 punch from 0x0017a5 to 0x00bea3 (0xa6ff bytes) 000631 1787384281.644849 read 0x01f32d thru 0x01f89d (0x571 bytes) 000632 1787384281.649851 read 0x02af0f thru 0x033e43 (0x8f35 bytes) 000633 1787384281.671641 trunc from 0x03c0c3 to 0x01dc44 (0x1e47e bytes) 000634 1787384281.695880 fallocate 0x0300e4 thru 0x03a357 (0xa274 bytes) 000635 1787384281.701687 read 0x01b849 thru 0x01dc44 (0x23fc bytes) 000636 1787384281.702410 mapwrite 0x0246c3 thru 0x029b79 (0x54b7 bytes) 000638 1787384281.756959 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b7a): Mirror 6: 0x326bba6c (stale) Mirror 7: 0x326bba6c (stale) Mirror 9: 0x326bba6c (stale) Mirror 14: 0x326bba6c (stale) Mirror 15: 0x326bba6c (stale) Mirror 16: 0x888126bd (stale) Mirror 17: 0xc9319c64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b7a): Mirror 6: 0x326bba6c (stale) Mirror 7: 0x326bba6c (stale) Mirror 9: 0x326bba6c (stale) Mirror 14: 0x326bba6c (stale) Mirror 15: 0x326bba6c (stale) Mirror 16: 0x888126bd (stale) Mirror 17: 0xc9319c64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b7a: skipped 000639 1787384282.354996 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29b7a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b7a): Mirror 6: 0x326bba6c (stale) Mirror 7: 0x326bba6c (stale) Mirror 9: 0x326bba6c (stale) Mirror 14: 0x326bba6c (stale) Mirror 15: 0x326bba6c (stale) Mirror 16: 0x888126bd (stale) Mirror 17: 0xc9319c64 (stale) CRC-32 checksum value for chunk [0, 0x29b7a): Mirror 1: 0x86eea039 Mirror 18: 0x86eea039 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b7a: skipped 000640 1787384283.668553 fallocate 0x03e5e1 thru 0x03ffff (0x1a1f bytes) 000641 1787384283.716754 trunc from 0x040000 to 0x015acf (0x2a530 bytes) 000642 1787384283.744757 mapwrite 0x007c6d thru 0x00dd3a (0x60ce bytes) 000643 1787384283.795647 trunc from 0x011cc5 to 0x012bfe (0xf3a bytes) 000644 1787384283.800054 mapread 0x011106 thru 0x015acf (0x49ca bytes) 000645 1787384283.808849 trunc from 0x015ad0 to 0x017764 (0x1c95 bytes) 000646 1787384283.828852 write 0x000ea8 thru 0x005b9d (0x4cf6 bytes) 000647 1787384283.845890 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17765): Mirror 6: 0x457384f2 (stale) Mirror 9: 0x457384f2 (stale) Mirror 14: 0x457384f2 (stale) Mirror 15: 0x457384f2 (stale) Mirror 16: 0xfc342a72 (stale) Mirror 17: 0x6b024542 (stale) Mirror 18: 0xf8054413 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17765): Mirror 6: 0x457384f2 (stale) Mirror 9: 0x457384f2 (stale) Mirror 14: 0x457384f2 (stale) Mirror 15: 0x457384f2 (stale) Mirror 16: 0xfc342a72 (stale) Mirror 17: 0x6b024542 (stale) Mirror 18: 0xf8054413 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17765: skipped 000649 1787384284.453589 mapread 0x014402 thru 0x017764 (0x3363 bytes) 000650 1787384284.481882 mapread 0x007a6c thru 0x012d1a (0xb2af bytes) 000651 1787384284.492992 read 0x01259d thru 0x017764 (0x51c8 bytes) 000652 1787384284.493760 mapwrite 0x00b529 thru 0x019342 (0xde1a bytes) 000654 1787384284.547194 mapwrite 0x033fee thru 0x03fcff (0xbd12 bytes) 000655 1787384284.600872 read 0x00432a thru 0x006bd0 (0x28a7 bytes) 000656 1787384284.602841 fallocate 0x03c9d6 thru 0x03ffff (0x362a bytes) 000657 1787384284.615225 read 0x016f61 thru 0x01c323 (0x53c3 bytes) 000658 1787384284.633429 write 0x00caea thru 0x01910b (0xc622 bytes) 000659 1787384284.635244 read 0x0079d1 thru 0x00888b (0xebb bytes) 000660 1787384284.636188 punch from 0x015df9 to 0x025a68 (0xfc70 bytes) 000661 1787384284.650875 mapwrite 0x039367 thru 0x03ffff (0x6c99 bytes) 000662 1787384284.674136 mapwrite 0x024058 thru 0x02ff44 (0xbeed bytes) 000663 1787384284.715019 fallocate 0x02f40a thru 0x036300 (0x6ef7 bytes) 000664 1787384284.725976 MIRROR_RESYNC 8 mirrors 000665 1787384286.973116 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000666 1787384287.456197 trunc from 0x02d78f to 0x033dcf (0x6641 bytes) 000667 1787384287.522167 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdeb85c9d (stale) Mirror 9: 0xdeb85c9d (stale) Mirror 14: 0xdeb85c9d (stale) Mirror 17: 0xdeb85c9d (stale) Mirror 18: 0xdeb85c9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdeb85c9d (stale) Mirror 9: 0xdeb85c9d (stale) Mirror 14: 0xdeb85c9d (stale) Mirror 17: 0xdeb85c9d (stale) Mirror 18: 0xdeb85c9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000668 1787384288.052740 trunc from 0x00cb67 to 0x018a97 (0xbf31 bytes) 000669 1787384288.059389 trunc from 0x019221 to 0x01aa48 (0x1828 bytes) 000670 1787384288.064797 trunc from 0x040000 to 0x03b6f7 (0x4908 bytes) 000671 1787384288.083142 trunc from 0x03b6f8 to 0x0328f5 (0x8e02 bytes) 000672 1787384288.101105 trunc from 0x00a625 to 0x0164a5 (0xbe81 bytes) 000673 1787384288.112651 trunc from 0x0328f6 to 0x00c34a (0x265ab bytes) 000674 1787384288.134155 read 0x00243d thru 0x00c34a (0x9f0e bytes) 000676 1787384288.149165 trunc from 0x00c34b to 0x02cb4e (0x20804 bytes) 000677 1787384288.171287 mapread 0x00e90c thru 0x01194b (0x3040 bytes) 000678 1787384288.201163 mapread 0x00e3bd thru 0x010c34 (0x2878 bytes) 000679 1787384288.209313 punch from 0x024a86 to 0x02868b (0x3c06 bytes) 000680 1787384288.239977 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cb4f): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb4f): Mirror 6: 0xf846ded9 (stale) Mirror 9: 0xf846ded9 (stale) Mirror 14: 0xf846ded9 (stale) Mirror 17: 0xf846ded9 (stale) Mirror 18: 0xf846ded9 (stale) CRC-32 checksum value for chunk [0, 0x2cb4f): Mirror 1: 0xcae4e222 Mirror 19: 0xcae4e222 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb4f: skipped 000681 1787384289.411284 write 0x019e23 thru 0x01b55b (0x1739 bytes) 000682 1787384289.462425 mapread 0x008290 thru 0x008605 (0x376 bytes) 000683 1787384289.472357 mapread 0x012d1f thru 0x015099 (0x237b bytes) 000684 1787384289.501231 MIRROR_RESYNC 7 mirrors 000685 1787384291.454252 trunc from 0x00701b to 0x0106fa (0x96e0 bytes) 000686 1787384291.503598 mapread 0x019ecd thru 0x023af5 (0x9c29 bytes) 000687 1787384291.527255 fallocate 0x01c976 thru 0x020a69 (0x40f4 bytes) 000688 1787384291.534730 write 0x004817 thru 0x008812 (0x3ffc bytes) 000689 1787384291.549197 trunc from 0x02cb4f to 0x036dee (0xa2a0 bytes) 000690 1787384291.579582 read 0x036017 thru 0x036dee (0xdd8 bytes) 000691 1787384291.584925 trunc from 0x027ba9 to 0x02aca0 (0x30f8 bytes) 000692 1787384291.593159 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36def): Mirror 6: 0xbb1c951e (stale) Mirror 9: 0xbb1c951e (stale) Mirror 17: 0xbb1c951e (stale) Mirror 18: 0xbb1c951e (stale) Mirror 19: 0xbb1c951e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36def): Mirror 6: 0xbb1c951e (stale) Mirror 9: 0xbb1c951e (stale) Mirror 17: 0xbb1c951e (stale) Mirror 18: 0xbb1c951e (stale) Mirror 19: 0xbb1c951e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36def: skipped 000693 1787384292.129401 mapwrite 0x01fb1e thru 0x022d1c (0x31ff bytes) 000694 1787384292.196601 read 0x029f48 thru 0x02b0d5 (0x118e bytes) 000695 1787384292.207039 mapwrite 0x00517d thru 0x006a0d (0x1891 bytes) 000696 1787384292.237294 trunc from 0x007fe1 to 0x00b3ee (0x340e bytes) 000697 1787384292.245749 punch from 0x01c808 to 0x01fd81 (0x357a bytes) 000698 1787384292.256097 fallocate 0x036373 thru 0x03ffff (0x9c8d bytes) 000699 1787384292.267522 write 0x0168d8 thru 0x01acbb (0x43e4 bytes) 000700 1787384292.282785 read 0x0363ac thru 0x036dee (0xa43 bytes) 000701 1787384292.299070 punch from 0x022464 to 0x02695a (0x44f7 bytes) 000702 1787384292.308857 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36def): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36def): Mirror 6: 0xbb1c951e (stale) Mirror 9: 0xbb1c951e (stale) Mirror 17: 0xbb1c951e (stale) Mirror 18: 0xbb1c951e (stale) Mirror 19: 0xbb1c951e (stale) CRC-32 checksum value for chunk [0, 0x36def): Mirror 1: 0x86f9d3a6 Mirror 20: 0x86f9d3a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36def: skipped 000704 1787384293.310622 fallocate 0x029768 thru 0x031176 (0x7a0f bytes) 000705 1787384293.360590 punch from 0x01bf17 to 0x027c84 (0xbd6e bytes) 000706 1787384293.367534 read 0x0325d2 thru 0x036dee (0x481d bytes) 000707 1787384293.377842 trunc from 0x010bc8 to 0x01d93a (0xcd73 bytes) 000708 1787384293.384686 read 0x00cd8e thru 0x014222 (0x7495 bytes) 000709 1787384293.397199 mapread 0x0069b9 thru 0x00e4a2 (0x7aea bytes) 000710 1787384293.398864 write 0x03b62b thru 0x03ffff (0x49d5 bytes) 000711 1787384293.406548 punch from 0x0191ed to 0x01dd9b (0x4baf bytes) 000713 1787384293.412841 trunc from 0x02d781 to 0x033238 (0x5ab8 bytes) 000715 1787384293.424339 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x60c20d2 (stale) Mirror 9: 0x60c20d2 (stale) Mirror 17: 0x60c20d2 (stale) Mirror 18: 0x60c20d2 (stale) Mirror 19: 0x60c20d2 (stale) Mirror 20: 0xe5347fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc5aa6de Mirror 21: 0xcc5aa6de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000716 1787384294.490292 punch from 0x01172a to 0x019d04 (0x85db bytes) 000717 1787384294.533102 trunc from 0x0273f3 to 0x02d57c (0x618a bytes) 000718 1787384294.545606 MIRROR_RESYNC 8 mirrors 000719 1787384296.051391 MIRROR_EXTEND to 9 mirrors 000720 1787384296.781884 read 0x006ff2 thru 0x00b7bf (0x47ce bytes) 000721 1787384296.798521 MIRROR_RESYNC 9 mirrors 000722 1787384297.410665 mapwrite 0x02ee62 thru 0x03597a (0x6b19 bytes) 000723 1787384297.543811 MIRROR_RESYNC 9 mirrors 000724 1787384299.491415 mapwrite 0x00a8cb thru 0x0171c1 (0xc8f7 bytes) 000725 1787384299.621217 write 0x02d386 thru 0x037190 (0x9e0b bytes) 000726 1787384299.643721 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x22465b13 (stale) Mirror 9: 0x22465b13 (stale) Mirror 17: 0x22465b13 (stale) Mirror 18: 0x22465b13 (stale) Mirror 19: 0x22465b13 (stale) Mirror 20: 0x22465b13 (stale) Mirror 21: 0x22465b13 (stale) Mirror 22: 0x22465b13 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb9ae916 Mirror 23: 0xbb9ae916 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000727 1787384300.663202 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x22465b13 (stale) Mirror 9: 0x22465b13 (stale) Mirror 17: 0x22465b13 (stale) Mirror 18: 0x22465b13 (stale) Mirror 19: 0x22465b13 (stale) Mirror 20: 0x22465b13 (stale) Mirror 21: 0x22465b13 (stale) Mirror 22: 0x22465b13 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb9ae916 Mirror 23: 0xbb9ae916 Mirror 24: 0xbb9ae916 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000728 1787384301.854793 punch from 0x018168 to 0x02069b (0x8534 bytes) 000729 1787384301.920434 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x22465b13 (stale) Mirror 9: 0x22465b13 (stale) Mirror 17: 0x22465b13 (stale) Mirror 18: 0x22465b13 (stale) Mirror 19: 0x22465b13 (stale) Mirror 20: 0x22465b13 (stale) Mirror 21: 0x22465b13 (stale) Mirror 22: 0x22465b13 (stale) Mirror 23: 0xbb9ae916 (stale) Mirror 24: 0xbb9ae916 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb9ae916 Mirror 25: 0xbb9ae916 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000730 1787384303.155197 mapread 0x00b4fe thru 0x01713f (0xbc42 bytes) 000731 1787384303.209802 mapwrite 0x00a2e4 thru 0x00de51 (0x3b6e bytes) 000732 1787384303.330006 mapread 0x00c6f2 thru 0x013510 (0x6e1f bytes) 000733 1787384303.357526 write 0x03fc63 thru 0x03ffff (0x39d bytes) 000734 1787384303.380636 mapwrite 0x02df5c thru 0x0366c6 (0x876b bytes) 000735 1787384303.458330 write 0x009916 thru 0x017b80 (0xe26b bytes) 000736 1787384303.463403 read 0x00e362 thru 0x017f95 (0x9c34 bytes) 000737 1787384303.464319 read 0x012e59 thru 0x013f22 (0x10ca bytes) 000738 1787384303.465839 punch from 0x01c09f to 0x0202a1 (0x4203 bytes) 000739 1787384303.477593 mapread 0x025c8d thru 0x02c0ed (0x6461 bytes) 000740 1787384303.499758 trunc from 0x016230 to 0x01b7b7 (0x5588 bytes) 000741 1787384303.523991 write 0x0049ce thru 0x0051ba (0x7ed bytes) 000742 1787384303.525383 mapwrite 0x03edea thru 0x03ffff (0x1216 bytes) 000743 1787384303.579039 mapwrite 0x018121 thru 0x01f82d (0x770d bytes) 000744 1787384303.654509 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x22465b13 (stale) Mirror 9: 0x22465b13 (stale) Mirror 17: 0x22465b13 (stale) Mirror 18: 0x22465b13 (stale) Mirror 19: 0x22465b13 (stale) Mirror 20: 0x22465b13 (stale) Mirror 21: 0x22465b13 (stale) Mirror 22: 0x22465b13 (stale) Mirror 23: 0xbb9ae916 (stale) Mirror 24: 0xbb9ae916 (stale) Mirror 25: 0xbb9ae916 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x94c49cd4 Mirror 26: 0x94c49cd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1787384305.035428 fallocate 0x02132f thru 0x02c64a (0xb31c bytes) 000747 1787384305.081018 mapwrite 0x02d34d thru 0x036805 (0x94b9 bytes) 000748 1787384305.130005 write 0x02b85a thru 0x034136 (0x88dd bytes) 000749 1787384305.143763 write 0x02eb4f thru 0x0386d7 (0x9b89 bytes) 000750 1787384305.154206 trunc from 0x040000 to 0x033cbd (0xc342 bytes) 000751 1787384305.196786 trunc from 0x02ace7 to 0x033cbd (0x8fd7 bytes) 000752 1787384305.232281 fallocate 0x027bbc thru 0x02f331 (0x7776 bytes) 000753 1787384305.245505 MIRROR_RESYNC 13 mirrors 000754 1787384308.739549 trunc from 0x003cf4 to 0x00c74b (0x8a58 bytes) 000755 1787384308.820181 write 0x03b503 thru 0x03ffff (0x4afd bytes) 000756 1787384308.822852 write 0x00dd1f thru 0x012aae (0x4d90 bytes) 000757 1787384308.837188 mapread 0x016de6 thru 0x021031 (0xa24c bytes) 000759 1787384308.876833 mapread 0x027557 thru 0x0292d9 (0x1d83 bytes) 000760 1787384308.878850 mapread 0x00cde2 thru 0x0152f1 (0x8510 bytes) 000761 1787384308.896077 fallocate 0x023049 thru 0x0283ab (0x5363 bytes) 000762 1787384308.908174 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x900d65ed (stale) Mirror 17: 0x900d65ed (stale) Mirror 18: 0x900d65ed (stale) Mirror 19: 0x900d65ed (stale) Mirror 20: 0x900d65ed (stale) Mirror 21: 0x900d65ed (stale) Mirror 22: 0x900d65ed (stale) Mirror 23: 0x900d65ed (stale) Mirror 24: 0x900d65ed (stale) Mirror 25: 0x900d65ed (stale) Mirror 26: 0x900d65ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x900d65ed (stale) Mirror 17: 0x900d65ed (stale) Mirror 18: 0x900d65ed (stale) Mirror 19: 0x900d65ed (stale) Mirror 20: 0x900d65ed (stale) Mirror 21: 0x900d65ed (stale) Mirror 22: 0x900d65ed (stale) Mirror 23: 0x900d65ed (stale) Mirror 24: 0x900d65ed (stale) Mirror 25: 0x900d65ed (stale) Mirror 26: 0x900d65ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1787384309.772880 trunc from 0x040000 to 0x00f74b (0x308b4 bytes) 000764 1787384309.831239 mapwrite 0x03c8f7 thru 0x03ffff (0x3709 bytes) 000765 1787384309.892821 trunc from 0x040000 to 0x019727 (0x268d8 bytes) 000766 1787384309.913196 read 0x00efa5 thru 0x018c54 (0x9cb0 bytes) 000767 1787384309.945633 trunc from 0x0034e5 to 0x00de61 (0xa97d bytes) 000768 1787384309.965204 MIRROR_RESYNC 12 mirrors 000769 1787384312.140646 write 0x036310 thru 0x038c82 (0x2973 bytes) 000770 1787384312.192274 mapread 0x024064 thru 0x0306b4 (0xc651 bytes) 000771 1787384312.238349 write 0x00fd6e thru 0x01a738 (0xa9cb bytes) 000772 1787384312.254423 trunc from 0x038c83 to 0x027c97 (0x10feb bytes) 000773 1787384312.287539 read 0x0156e6 thru 0x01779f (0x20ba bytes) 000774 1787384312.290739 trunc from 0x027c98 to 0x013c2d (0x1406a bytes) 000775 1787384312.310588 punch from 0x00ea52 to 0x013c2d (0x51dc bytes) 000776 1787384312.331333 trunc from 0x013c2e to 0x0031e5 (0x10a48 bytes) 000777 1787384312.353641 mapwrite 0x03f321 thru 0x03ffff (0xcdf bytes) 000778 1787384312.433990 read 0x03e293 thru 0x03ffff (0x1d6d bytes) 000780 1787384312.435467 read 0x017432 thru 0x01d306 (0x5ed5 bytes) 000781 1787384312.453583 fallocate 0x01f101 thru 0x0284be (0x93be bytes) 000782 1787384312.461607 fallocate 0x02c672 thru 0x0331c5 (0x6b54 bytes) 000783 1787384312.471216 mapwrite 0x028ae9 thru 0x02a155 (0x166d bytes) 000784 1787384312.510187 fallocate 0x0271f5 thru 0x02f0f5 (0x7f01 bytes) 000785 1787384312.522747 trunc from 0x040000 to 0x005c1d (0x3a3e2 bytes) 000786 1787384312.599086 trunc from 0x005c1e to 0x035846 (0x2fc29 bytes) 000788 1787384312.622553 mapread 0x0320e5 thru 0x034f0d (0x2e29 bytes) 000789 1787384312.654408 MIRROR_RESYNC 12 mirrors 000790 1787384314.483621 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000791 1787384314.972258 mapread 0x0208c6 thru 0x0226fb (0x1e36 bytes) 000792 1787384315.002282 fallocate 0x00978f thru 0x010da2 (0x7614 bytes) 000793 1787384315.088766 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: 0xe230ceeb (stale) Mirror 17: 0xe230ceeb (stale) Mirror 18: 0xe230ceeb (stale) Mirror 20: 0xe230ceeb (stale) Mirror 21: 0xe230ceeb (stale) Mirror 22: 0xe230ceeb (stale) Mirror 23: 0xe230ceeb (stale) Mirror 24: 0xe230ceeb (stale) Mirror 26: 0xe230ceeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: 0xe230ceeb (stale) Mirror 17: 0xe230ceeb (stale) Mirror 18: 0xe230ceeb (stale) Mirror 20: 0xe230ceeb (stale) Mirror 21: 0xe230ceeb (stale) Mirror 22: 0xe230ceeb (stale) Mirror 23: 0xe230ceeb (stale) Mirror 24: 0xe230ceeb (stale) Mirror 26: 0xe230ceeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35847: skipped 000795 1787384315.809729 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: 0xe230ceeb (stale) Mirror 17: 0xe230ceeb (stale) Mirror 18: 0xe230ceeb (stale) Mirror 20: 0xe230ceeb (stale) Mirror 21: 0xe230ceeb (stale) Mirror 22: 0xe230ceeb (stale) Mirror 23: 0xe230ceeb (stale) Mirror 24: 0xe230ceeb (stale) Mirror 26: 0xe230ceeb (stale) CRC-32 checksum value for chunk [0, 0x35847): Mirror 1: 0xe230ceeb Mirror 27: 0xe230ceeb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35847: skipped 000796 1787384316.836853 mapwrite 0x019dbd thru 0x022f15 (0x9159 bytes) 000798 1787384317.004917 read 0x01f076 thru 0x0278eb (0x8876 bytes) 000800 1787384317.016493 write 0x000dbe thru 0x0076db (0x691e bytes) 000801 1787384317.035559 fallocate 0x0281c8 thru 0x02c2e5 (0x411e bytes) 000802 1787384317.049377 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35847): Mirror 6: 0xe230ceeb (stale) Mirror 17: 0xe230ceeb (stale) Mirror 18: 0xe230ceeb (stale) Mirror 20: 0xe230ceeb (stale) Mirror 21: 0xe230ceeb (stale) Mirror 22: 0xe230ceeb (stale) Mirror 23: 0xe230ceeb (stale) Mirror 24: 0xe230ceeb (stale) Mirror 26: 0xe230ceeb (stale) Mirror 27: 0xe230ceeb (stale) CRC-32 checksum value for chunk [0, 0x35847): Mirror 1: 0xc61b6727 Mirror 28: 0xc61b6727 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35847: skipped 000803 1787384318.277377 mapwrite 0x009c8a thru 0x0153b0 (0xb727 bytes) 000804 1787384318.468034 trunc from 0x007fc1 to 0x01372c (0xb76c bytes) 000806 1787384318.485257 fallocate 0x03c23c thru 0x03ffff (0x3dc4 bytes) 000807 1787384318.494402 mapwrite 0x029eac thru 0x02b905 (0x1a5a bytes) 000808 1787384318.533555 read 0x02f316 thru 0x035846 (0x6531 bytes) 000809 1787384318.541772 mapwrite 0x009d6d thru 0x00b21d (0x14b1 bytes) 000810 1787384318.578615 trunc from 0x008e94 to 0x01858e (0xf6fb bytes) 000811 1787384318.584575 write 0x0033c9 thru 0x00ac86 (0x78be bytes) 000812 1787384318.595883 trunc from 0x035847 to 0x03c69a (0x6e54 bytes) 000813 1787384318.660489 mapwrite 0x03a0b4 thru 0x03ffff (0x5f4c bytes) 000814 1787384318.703516 mapread 0x007f07 thru 0x00cf59 (0x5053 bytes) 000815 1787384318.716800 punch from 0x03b393 to 0x03fe0b (0x4a79 bytes) 000816 1787384318.722181 mapread 0x0369d9 thru 0x03cd5c (0x6384 bytes) 000817 1787384318.733715 MIRROR_RESYNC 12 mirrors 000818 1787384322.563382 mapwrite 0x029483 thru 0x037eb7 (0xea35 bytes) 000819 1787384322.686411 write 0x03a963 thru 0x03ffff (0x569d bytes) 000820 1787384322.693081 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x62f06198 (stale) Mirror 17: 0x62f06198 (stale) Mirror 18: 0x62f06198 (stale) Mirror 20: 0x62f06198 (stale) Mirror 21: 0x62f06198 (stale) Mirror 22: 0x62f06198 (stale) Mirror 23: 0x62f06198 (stale) Mirror 24: 0x62f06198 (stale) Mirror 26: 0x62f06198 (stale) Mirror 27: 0x62f06198 (stale) Mirror 28: 0x62f06198 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb1c4a563 Mirror 29: 0xb1c4a563 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000821 1787384324.186393 trunc from 0x040000 to 0x02c05a (0x13fa5 bytes) 000822 1787384324.246450 fallocate 0x014b89 thru 0x01b901 (0x6d79 bytes) 000824 1787384324.252700 trunc from 0x02c05b to 0x017b7a (0x144e0 bytes) 000825 1787384324.273975 read 0x012249 thru 0x017b7a (0x5932 bytes) 000826 1787384324.290182 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x17b7b): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b7b): Mirror 6: 0xd1ee6224 (stale) Mirror 17: 0xd1ee6224 (stale) Mirror 18: 0xd1ee6224 (stale) Mirror 20: 0xd1ee6224 (stale) Mirror 21: 0xd1ee6224 (stale) Mirror 22: 0xd1ee6224 (stale) Mirror 23: 0xd1ee6224 (stale) Mirror 24: 0xd1ee6224 (stale) Mirror 26: 0xd1ee6224 (stale) Mirror 27: 0xd1ee6224 (stale) Mirror 28: 0xd1ee6224 (stale) Mirror 29: 0xd1ee6224 (stale) CRC-32 checksum value for chunk [0, 0x17b7b): Mirror 1: 0xd1ee6224 Mirror 30: 0xd1ee6224 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b7b: skipped 000827 1787384325.293411 trunc from 0x017b7b to 0x02710b (0xf591 bytes) 000828 1787384325.390398 punch from 0x00920a to 0x011349 (0x8140 bytes) 000829 1787384325.404621 punch from 0x0060f0 to 0x007b84 (0x1a95 bytes) 000830 1787384325.411275 fallocate 0x039628 thru 0x03ffff (0x69d8 bytes) 000831 1787384325.417845 read 0x026677 thru 0x02710b (0xa95 bytes) 000832 1787384325.442994 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2710c): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2710c): Mirror 6: 0x2bb13933 (stale) Mirror 17: 0x2bb13933 (stale) Mirror 18: 0x2bb13933 (stale) Mirror 20: 0x2bb13933 (stale) Mirror 21: 0x2bb13933 (stale) Mirror 22: 0x2bb13933 (stale) Mirror 23: 0x2bb13933 (stale) Mirror 24: 0x2bb13933 (stale) Mirror 26: 0x2bb13933 (stale) Mirror 27: 0x2bb13933 (stale) Mirror 28: 0x2bb13933 (stale) Mirror 29: 0x2bb13933 (stale) Mirror 30: 0x71d3a627 (stale) CRC-32 checksum value for chunk [0, 0x2710c): Mirror 1: 0x514e4d5d Mirror 31: 0x514e4d5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2710c: skipped 000833 1787384326.568781 mapwrite 0x0044ee thru 0x00a284 (0x5d97 bytes) 000834 1787384326.706718 trunc from 0x002254 to 0x008201 (0x5fae bytes) 000835 1787384326.717405 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2710c): Mirror 6: 0x2bb13933 (stale) Mirror 17: 0x2bb13933 (stale) Mirror 18: 0x2bb13933 (stale) Mirror 20: 0x2bb13933 (stale) Mirror 21: 0x2bb13933 (stale) Mirror 22: 0x2bb13933 (stale) Mirror 23: 0x2bb13933 (stale) Mirror 24: 0x2bb13933 (stale) Mirror 26: 0x2bb13933 (stale) Mirror 27: 0x2bb13933 (stale) Mirror 28: 0x2bb13933 (stale) Mirror 29: 0x2bb13933 (stale) Mirror 30: 0x71d3a627 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2710c): Mirror 6: 0x2bb13933 (stale) Mirror 17: 0x2bb13933 (stale) Mirror 18: 0x2bb13933 (stale) Mirror 20: 0x2bb13933 (stale) Mirror 21: 0x2bb13933 (stale) Mirror 22: 0x2bb13933 (stale) Mirror 23: 0x2bb13933 (stale) Mirror 24: 0x2bb13933 (stale) Mirror 26: 0x2bb13933 (stale) Mirror 27: 0x2bb13933 (stale) Mirror 28: 0x2bb13933 (stale) Mirror 29: 0x2bb13933 (stale) Mirror 30: 0x71d3a627 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2710c: skipped 000836 1787384327.382846 write 0x002039 thru 0x00b9b5 (0x997d bytes) 000837 1787384327.399638 fallocate 0x03da8e thru 0x03ffff (0x2572 bytes) 000838 1787384327.413090 fallocate 0x032ad3 thru 0x039f16 (0x7444 bytes) 000839 1787384327.420356 write 0x018fd4 thru 0x025a46 (0xca73 bytes) 000840 1787384327.444005 mapwrite 0x01e3ec thru 0x02c2c2 (0xded7 bytes) 000841 1787384327.563052 MIRROR_RESYNC 14 mirrors 000842 1787384330.494370 fallocate 0x01e4d2 thru 0x02d64e (0xf17d bytes) 000843 1787384330.541593 fallocate 0x00fdf2 thru 0x01837d (0x858c bytes) 000844 1787384330.552707 mapread 0x033de5 thru 0x03e09a (0xa2b6 bytes) 000845 1787384330.595023 fallocate 0x012902 thru 0x01f7b0 (0xceaf bytes) 000846 1787384330.604771 trunc from 0x040000 to 0x00c50b (0x33af4 bytes) 000847 1787384330.633397 trunc from 0x006675 to 0x00c50b (0x5e97 bytes) 000848 1787384330.639678 mapwrite 0x01f0a6 thru 0x02480a (0x5765 bytes) 000849 1787384330.690293 trunc from 0x02480b to 0x000ff8 (0x23812 bytes) 000850 1787384330.704364 mapread 0x000042 thru 0x000ff8 (0xfb7 bytes) 000851 1787384330.711657 trunc from 0x000ff9 to 0x026287 (0x2528f bytes) 000852 1787384330.732342 fallocate 0x0127cf thru 0x018e88 (0x66ba bytes) 000854 1787384330.739417 mapwrite 0x039796 thru 0x03a7af (0x101a bytes) 000855 1787384330.777070 trunc from 0x0089e2 to 0x0121e0 (0x97ff bytes) 000856 1787384330.783743 fallocate 0x007127 thru 0x00bc3a (0x4b14 bytes) 000857 1787384330.795169 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 6: 0xa724fc85 (stale) Mirror 17: 0xa724fc85 (stale) Mirror 18: 0xa724fc85 (stale) Mirror 20: 0xa724fc85 (stale) Mirror 21: 0xa724fc85 (stale) Mirror 22: 0xa724fc85 (stale) Mirror 23: 0xa724fc85 (stale) Mirror 24: 0xa724fc85 (stale) Mirror 26: 0xa724fc85 (stale) Mirror 27: 0xa724fc85 (stale) Mirror 28: 0xa724fc85 (stale) Mirror 29: 0xa724fc85 (stale) Mirror 30: 0xa724fc85 (stale) CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 1: 0x58fc83a6 Mirror 31: 0x58fc83a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7b0: skipped 000858 1787384332.093963 fallocate 0x02542e thru 0x02cbfc (0x77cf bytes) 000859 1787384332.179590 fallocate 0x01b978 thru 0x0296de (0xdd67 bytes) 000860 1787384332.199807 mapread 0x038af4 thru 0x03a7af (0x1cbc bytes) 000862 1787384332.248695 read 0x000121 thru 0x0019e2 (0x18c2 bytes) 000863 1787384332.263857 write 0x03827a thru 0x039d0d (0x1a94 bytes) 000864 1787384332.272285 fallocate 0x033af4 thru 0x03ffff (0xc50c bytes) 000865 1787384332.291652 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf31c269 (stale) Mirror 17: 0xbf31c269 (stale) Mirror 18: 0xbf31c269 (stale) Mirror 20: 0xbf31c269 (stale) Mirror 21: 0xbf31c269 (stale) Mirror 22: 0xbf31c269 (stale) Mirror 23: 0xbf31c269 (stale) Mirror 24: 0xbf31c269 (stale) Mirror 26: 0xbf31c269 (stale) Mirror 28: 0xbf31c269 (stale) Mirror 29: 0xbf31c269 (stale) Mirror 30: 0xbf31c269 (stale) Mirror 31: 0xb9ba7e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf31c269 (stale) Mirror 17: 0xbf31c269 (stale) Mirror 18: 0xbf31c269 (stale) Mirror 20: 0xbf31c269 (stale) Mirror 21: 0xbf31c269 (stale) Mirror 22: 0xbf31c269 (stale) Mirror 23: 0xbf31c269 (stale) Mirror 24: 0xbf31c269 (stale) Mirror 26: 0xbf31c269 (stale) Mirror 28: 0xbf31c269 (stale) Mirror 29: 0xbf31c269 (stale) Mirror 30: 0xbf31c269 (stale) Mirror 31: 0xb9ba7e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000866 1787384333.204433 mapwrite 0x00e451 thru 0x010bd0 (0x2780 bytes) 000867 1787384333.267386 mapread 0x0325cb thru 0x03d52f (0xaf65 bytes) 000868 1787384333.301205 MIRROR_RESYNC 14 mirrors 000869 1787384336.822820 read 0x01fdb6 thru 0x029d51 (0x9f9c bytes) 000870 1787384336.831542 trunc from 0x040000 to 0x03ad63 (0x529c bytes) 000871 1787384336.870526 trunc from 0x02bb44 to 0x03847c (0xc939 bytes) 000873 1787384336.876713 write 0x03de2e thru 0x03ffff (0x21d2 bytes) 000874 1787384336.878947 fallocate 0x03f200 thru 0x03ffc9 (0xdca bytes) 000875 1787384336.884253 punch from 0x007a53 to 0x0168da (0xee88 bytes) 000876 1787384336.890979 read 0x00f0b6 thru 0x017fd9 (0x8f24 bytes) 000877 1787384336.902407 fallocate 0x00ebe1 thru 0x011ed7 (0x32f7 bytes) 000878 1787384336.907637 mapwrite 0x003ef2 thru 0x006e71 (0x2f80 bytes) 000880 1787384336.926276 mapread 0x007256 thru 0x0103c0 (0x916b bytes) 000881 1787384336.932442 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf59a3cb5 (stale) Mirror 17: 0xf59a3cb5 (stale) Mirror 18: 0xf59a3cb5 (stale) Mirror 20: 0xf59a3cb5 (stale) Mirror 21: 0xf59a3cb5 (stale) Mirror 22: 0xf59a3cb5 (stale) Mirror 23: 0xf59a3cb5 (stale) Mirror 24: 0xf59a3cb5 (stale) Mirror 26: 0xf59a3cb5 (stale) Mirror 29: 0xf59a3cb5 (stale) Mirror 30: 0xf59a3cb5 (stale) Mirror 31: 0xf59a3cb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf59a3cb5 (stale) Mirror 17: 0xf59a3cb5 (stale) Mirror 18: 0xf59a3cb5 (stale) Mirror 20: 0xf59a3cb5 (stale) Mirror 21: 0xf59a3cb5 (stale) Mirror 22: 0xf59a3cb5 (stale) Mirror 23: 0xf59a3cb5 (stale) Mirror 24: 0xf59a3cb5 (stale) Mirror 26: 0xf59a3cb5 (stale) Mirror 29: 0xf59a3cb5 (stale) Mirror 30: 0xf59a3cb5 (stale) Mirror 31: 0xf59a3cb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000882 1787384337.738316 write 0x036f07 thru 0x03ffff (0x90f9 bytes) 000883 1787384337.760269 fallocate 0x01ec26 thru 0x020c8b (0x2066 bytes) 000886 1787384337.779406 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf59a3cb5 (stale) Mirror 17: 0xf59a3cb5 (stale) Mirror 18: 0xf59a3cb5 (stale) Mirror 20: 0xf59a3cb5 (stale) Mirror 21: 0xf59a3cb5 (stale) Mirror 22: 0xf59a3cb5 (stale) Mirror 23: 0xf59a3cb5 (stale) Mirror 24: 0xf59a3cb5 (stale) Mirror 26: 0xf59a3cb5 (stale) Mirror 29: 0xf59a3cb5 (stale) Mirror 30: 0xf59a3cb5 (stale) Mirror 31: 0xf59a3cb5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59917a5b Mirror 32: 0x59917a5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1787384339.017110 punch from 0x030d84 to 0x039ac1 (0x8d3e bytes) 000888 1787384339.059253 fallocate 0x01b103 thru 0x02a8f7 (0xf7f5 bytes) 000889 1787384339.068480 fallocate 0x014d8d thru 0x0239c7 (0xec3b bytes) 000890 1787384339.074879 trunc from 0x011cfe to 0x01cbf3 (0xaef6 bytes) 000891 1787384339.081095 mapwrite 0x01c812 thru 0x022323 (0x5b12 bytes) 000892 1787384339.139820 fallocate 0x03e4bd thru 0x03ffff (0x1b43 bytes) 000894 1787384339.154725 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf59a3cb5 (stale) Mirror 17: 0xf59a3cb5 (stale) Mirror 18: 0xf59a3cb5 (stale) Mirror 20: 0xf59a3cb5 (stale) Mirror 21: 0xf59a3cb5 (stale) Mirror 23: 0xf59a3cb5 (stale) Mirror 24: 0xf59a3cb5 (stale) Mirror 26: 0xf59a3cb5 (stale) Mirror 29: 0xf59a3cb5 (stale) Mirror 30: 0xf59a3cb5 (stale) Mirror 31: 0xf59a3cb5 (stale) Mirror 32: 0x59917a5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf59a3cb5 (stale) Mirror 17: 0xf59a3cb5 (stale) Mirror 18: 0xf59a3cb5 (stale) Mirror 20: 0xf59a3cb5 (stale) Mirror 21: 0xf59a3cb5 (stale) Mirror 23: 0xf59a3cb5 (stale) Mirror 24: 0xf59a3cb5 (stale) Mirror 26: 0xf59a3cb5 (stale) Mirror 29: 0xf59a3cb5 (stale) Mirror 30: 0xf59a3cb5 (stale) Mirror 31: 0xf59a3cb5 (stale) Mirror 32: 0x59917a5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000895 1787384340.076374 mapwrite 0x0247af thru 0x026dbc (0x260e bytes) 000896 1787384340.109844 write 0x000277 thru 0x00bc8f (0xba19 bytes) 000897 1787384340.123683 fallocate 0x0348f6 thru 0x03ffff (0xb70a bytes) 000898 1787384340.128705 mapwrite 0x0180c8 thru 0x01c7ef (0x4728 bytes) 000899 1787384340.161820 MIRROR_RESYNC 13 mirrors 000900 1787384344.048849 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000901 1787384344.781476 trunc from 0x040000 to 0x032fb2 (0xd04d bytes) 000902 1787384344.841766 fallocate 0x001579 thru 0x00f811 (0xe299 bytes) 000903 1787384344.856333 mapwrite 0x008a68 thru 0x00d16f (0x4708 bytes) 000904 1787384344.911185 trunc from 0x032fb3 to 0x00ba2c (0x27586 bytes) 000905 1787384344.957178 trunc from 0x00ba2d to 0x030041 (0x24615 bytes) 000906 1787384344.988827 write 0x03a218 thru 0x03ffff (0x5de8 bytes) 000907 1787384344.991461 punch from 0x0226a2 to 0x027252 (0x4bb1 bytes) 000908 1787384345.000466 mapread 0x014ea9 thru 0x0233fd (0xe555 bytes) 000909 1787384345.044400 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xcff69ee9 (stale) Mirror 17: 0xcff69ee9 (stale) Mirror 18: 0xcff69ee9 (stale) Mirror 20: 0xcff69ee9 (stale) Mirror 21: 0xcff69ee9 (stale) Mirror 23: 0xcff69ee9 (stale) Mirror 24: 0xcff69ee9 (stale) Mirror 29: 0xcff69ee9 (stale) Mirror 30: 0xcff69ee9 (stale) Mirror 31: 0xcff69ee9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xcff69ee9 (stale) Mirror 17: 0xcff69ee9 (stale) Mirror 18: 0xcff69ee9 (stale) Mirror 20: 0xcff69ee9 (stale) Mirror 21: 0xcff69ee9 (stale) Mirror 23: 0xcff69ee9 (stale) Mirror 24: 0xcff69ee9 (stale) Mirror 29: 0xcff69ee9 (stale) Mirror 30: 0xcff69ee9 (stale) Mirror 31: 0xcff69ee9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1787384345.675961 write 0x01a32d thru 0x01e106 (0x3dda bytes) 000911 1787384345.692302 trunc from 0x03cbc2 to 0x03ffff (0x343e bytes) 000912 1787384345.697286 read 0x031c4f thru 0x03ffff (0xe3b1 bytes) 000913 1787384345.708650 read 0x03f5af thru 0x03ffff (0xa51 bytes) 000914 1787384345.709867 read 0x022be5 thru 0x0323c7 (0xf7e3 bytes) 000915 1787384345.739881 fallocate 0x024031 thru 0x02a77e (0x674e bytes) 000916 1787384345.753922 read 0x010c4c thru 0x010f2c (0x2e1 bytes) 000917 1787384345.756503 fallocate 0x035351 thru 0x03d4ff (0x81af bytes) 000918 1787384345.767395 punch from 0x009b4c to 0x00f2d3 (0x5788 bytes) 000919 1787384345.776947 trunc from 0x040000 to 0x025d9f (0x1a260 bytes) 000920 1787384345.797789 punch from 0x0101e7 to 0x019ffd (0x9e17 bytes) 000921 1787384345.819252 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x25da0): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25da0): Mirror 6: 0x73ca4fe2 (stale) Mirror 17: 0x73ca4fe2 (stale) Mirror 18: 0x73ca4fe2 (stale) Mirror 20: 0x73ca4fe2 (stale) Mirror 21: 0x73ca4fe2 (stale) Mirror 23: 0x73ca4fe2 (stale) Mirror 24: 0x73ca4fe2 (stale) Mirror 29: 0x73ca4fe2 (stale) Mirror 30: 0x73ca4fe2 (stale) Mirror 31: 0x73ca4fe2 (stale) CRC-32 checksum value for chunk [0, 0x25da0): Mirror 1: 0x3ea7dad4 Mirror 32: 0x3ea7dad4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25da0: skipped 000923 1787384346.987074 write 0x026415 thru 0x035ded (0xf9d9 bytes) 000924 1787384347.053483 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x35dee): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35dee): Mirror 6: 0xb862772b (stale) Mirror 17: 0xb862772b (stale) Mirror 18: 0xb862772b (stale) Mirror 20: 0xb862772b (stale) Mirror 21: 0xb862772b (stale) Mirror 23: 0xb862772b (stale) Mirror 24: 0xb862772b (stale) Mirror 29: 0xb862772b (stale) Mirror 30: 0xb862772b (stale) Mirror 31: 0xb862772b (stale) Mirror 32: 0xc5dd0705 (stale) CRC-32 checksum value for chunk [0, 0x35dee): Mirror 1: 0xb991b973 Mirror 33: 0xb991b973 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35dee: skipped 000925 1787384348.250346 mapread 0x00ceb8 thru 0x00fd71 (0x2eba bytes) 000926 1787384348.285636 mapread 0x032710 thru 0x035ded (0x36de bytes) 000927 1787384348.309722 trunc from 0x008aa5 to 0x017518 (0xea74 bytes) 000928 1787384348.376165 write 0x033a97 thru 0x03ffff (0xc569 bytes) 000929 1787384348.398193 fallocate 0x002e69 thru 0x00eab7 (0xbc4f bytes) 000930 1787384348.405335 trunc from 0x040000 to 0x039432 (0x6bcd bytes) 000933 1787384348.435649 write 0x039966 thru 0x03ff18 (0x65b3 bytes) 000934 1787384348.438886 fallocate 0x028a9e thru 0x035876 (0xcdd9 bytes) 000935 1787384348.449874 MIRROR_RESYNC 13 mirrors 000936 1787384351.457590 fallocate 0x016d67 thru 0x020e45 (0xa0df bytes) 000938 1787384351.503118 write 0x002651 thru 0x005775 (0x3125 bytes) 000939 1787384351.517735 write 0x025c92 thru 0x02fad6 (0x9e45 bytes) 000941 1787384351.529174 mapread 0x027b1e thru 0x031381 (0x9864 bytes) 000943 1787384351.543732 trunc from 0x03ff19 to 0x02f5f5 (0x10923 bytes) 000944 1787384351.580626 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: 0x44b963ff (stale) Mirror 17: 0x44b963ff (stale) Mirror 18: 0x44b963ff (stale) Mirror 20: 0x44b963ff (stale) Mirror 21: 0x44b963ff (stale) Mirror 23: 0x44b963ff (stale) Mirror 24: 0x44b963ff (stale) Mirror 29: 0x44b963ff (stale) Mirror 30: 0x44b963ff (stale) Mirror 31: 0x44b963ff (stale) Mirror 32: 0x44b963ff (stale) Mirror 33: 0x44b963ff (stale) CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 1: 0xd61b541b Mirror 34: 0xd61b541b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5f6: skipped 000945 1787384352.905388 mapread 0x00e298 thru 0x011c71 (0x39da bytes) 000946 1787384352.988220 trunc from 0x0260a7 to 0x02f19b (0x90f5 bytes) 000947 1787384353.062947 mapread 0x018612 thru 0x025434 (0xce23 bytes) 000948 1787384353.082350 punch from 0x006978 to 0x00fc85 (0x930e bytes) 000949 1787384353.088316 mapwrite 0x02259b thru 0x02b8c3 (0x9329 bytes) 000950 1787384353.112455 fallocate 0x023791 thru 0x02a758 (0x6fc8 bytes) 000951 1787384353.117844 write 0x0000b0 thru 0x00907f (0x8fd0 bytes) 000953 1787384353.133887 write 0x025c26 thru 0x02ab18 (0x4ef3 bytes) 000954 1787384353.135513 trunc from 0x00ebec to 0x0130bd (0x44d2 bytes) 000955 1787384353.145080 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: 0x44b963ff (stale) Mirror 17: 0x44b963ff (stale) Mirror 18: 0x44b963ff (stale) Mirror 20: 0x44b963ff (stale) Mirror 21: 0x44b963ff (stale) Mirror 23: 0x44b963ff (stale) Mirror 24: 0x44b963ff (stale) Mirror 29: 0x44b963ff (stale) Mirror 30: 0x44b963ff (stale) Mirror 31: 0x44b963ff (stale) Mirror 32: 0x44b963ff (stale) Mirror 33: 0x44b963ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: 0x44b963ff (stale) Mirror 17: 0x44b963ff (stale) Mirror 18: 0x44b963ff (stale) Mirror 20: 0x44b963ff (stale) Mirror 21: 0x44b963ff (stale) Mirror 23: 0x44b963ff (stale) Mirror 24: 0x44b963ff (stale) Mirror 29: 0x44b963ff (stale) Mirror 30: 0x44b963ff (stale) Mirror 31: 0x44b963ff (stale) Mirror 32: 0x44b963ff (stale) Mirror 33: 0x44b963ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5f6: skipped 000956 1787384353.990771 fallocate 0x00e048 thru 0x0160ca (0x8083 bytes) 000957 1787384353.998416 punch from 0x0181ef to 0x02621b (0xe02d bytes) 000958 1787384354.011739 punch from 0x003a9d to 0x012669 (0xebcd bytes) 000959 1787384354.019416 read 0x00f920 thru 0x01289a (0x2f7b bytes) 000961 1787384354.040851 punch from 0x022cd5 to 0x02612c (0x3458 bytes) 000962 1787384354.046432 write 0x01b67d thru 0x0269be (0xb342 bytes) 000963 1787384354.078259 punch from 0x002dd6 to 0x00e5a3 (0xb7ce bytes) 000964 1787384354.088693 fallocate 0x02ff27 thru 0x036350 (0x642a bytes) 000965 1787384354.101145 punch from 0x01643d to 0x023ab9 (0xd67d bytes) 000966 1787384354.136327 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 6: 0x44b963ff (stale) Mirror 17: 0x44b963ff (stale) Mirror 18: 0x44b963ff (stale) Mirror 20: 0x44b963ff (stale) Mirror 21: 0x44b963ff (stale) Mirror 23: 0x44b963ff (stale) Mirror 24: 0x44b963ff (stale) Mirror 29: 0x44b963ff (stale) Mirror 30: 0x44b963ff (stale) Mirror 31: 0x44b963ff (stale) Mirror 32: 0x44b963ff (stale) Mirror 33: 0x44b963ff (stale) CRC-32 checksum value for chunk [0, 0x2f5f6): Mirror 1: 0xeea57409 Mirror 34: 0xeea57409 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5f6: skipped 000967 1787384355.412502 mapwrite 0x039884 thru 0x03ffff (0x677c bytes) 000968 1787384355.575328 punch from 0x0125a5 to 0x01c913 (0xa36f bytes) 000969 1787384355.582846 fallocate 0x03c208 thru 0x03ffff (0x3df8 bytes) 000970 1787384355.589718 mapwrite 0x025f2c thru 0x031b7c (0xbc51 bytes) 000971 1787384355.634457 trunc from 0x040000 to 0x01feae (0x20151 bytes) 000972 1787384355.662326 MIRROR_RESYNC 14 mirrors 000973 1787384357.452590 trunc from 0x01feaf to 0x001531 (0x1e97d bytes) 000974 1787384357.495861 fallocate 0x022370 thru 0x02be4f (0x9ae0 bytes) 000976 1787384357.502023 trunc from 0x000219 to 0x001531 (0x1319 bytes) 000977 1787384357.508491 trunc from 0x001532 to 0x00cd52 (0xb821 bytes) 000978 1787384357.528723 punch from 0x003b09 to 0x00abea (0x70e2 bytes) 000979 1787384357.534595 write 0x0211f2 thru 0x0220f5 (0xf04 bytes) 000980 1787384357.536234 mapread 0x01f4b8 thru 0x0220f5 (0x2c3e bytes) 000981 1787384357.547200 trunc from 0x0220f6 to 0x03cfdf (0x1aeea bytes) 000982 1787384357.571120 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfe0): Mirror 17: 0xd8234ceb (stale) Mirror 18: 0xd8234ceb (stale) Mirror 20: 0xd8234ceb (stale) Mirror 21: 0xd8234ceb (stale) Mirror 23: 0xd8234ceb (stale) Mirror 24: 0xd8234ceb (stale) Mirror 29: 0xd8234ceb (stale) Mirror 30: 0xd8234ceb (stale) Mirror 31: 0xd8234ceb (stale) Mirror 32: 0xd8234ceb (stale) Mirror 33: 0xd8234ceb (stale) Mirror 34: 0xd8234ceb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfe0): Mirror 17: 0xd8234ceb (stale) Mirror 18: 0xd8234ceb (stale) Mirror 20: 0xd8234ceb (stale) Mirror 21: 0xd8234ceb (stale) Mirror 23: 0xd8234ceb (stale) Mirror 24: 0xd8234ceb (stale) Mirror 29: 0xd8234ceb (stale) Mirror 30: 0xd8234ceb (stale) Mirror 31: 0xd8234ceb (stale) Mirror 32: 0xd8234ceb (stale) Mirror 33: 0xd8234ceb (stale) Mirror 34: 0xd8234ceb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfe0: skipped 000984 1787384358.291963 trunc from 0x03cfe0 to 0x03889e (0x4741 bytes) 000985 1787384358.326042 write 0x003ab1 thru 0x012a6e (0xefbe bytes) 000986 1787384358.349828 mapread 0x0329bf thru 0x03889e (0x5ee0 bytes) 000987 1787384358.378371 mapread 0x00733d thru 0x00bee9 (0x4bad bytes) 000990 1787384358.388481 trunc from 0x03889f to 0x0351b9 (0x36e5 bytes) 000991 1787384358.423672 mapwrite 0x02ee85 thru 0x03a7c4 (0xb940 bytes) 000992 1787384358.509394 trunc from 0x03a7c5 to 0x03a475 (0x34f bytes) 000993 1787384358.542219 write 0x0156fb thru 0x016f2e (0x1834 bytes) 000994 1787384358.555587 read 0x00f011 thru 0x016a56 (0x7a46 bytes) 000995 1787384358.566305 trunc from 0x03a476 to 0x031dab (0x86ca bytes) 000996 1787384358.599997 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31dac): Mirror 17: 0x928ef655 (stale) Mirror 18: 0x928ef655 (stale) Mirror 20: 0x928ef655 (stale) Mirror 21: 0x928ef655 (stale) Mirror 23: 0x928ef655 (stale) Mirror 24: 0x928ef655 (stale) Mirror 29: 0x928ef655 (stale) Mirror 30: 0x928ef655 (stale) Mirror 31: 0x928ef655 (stale) Mirror 32: 0x928ef655 (stale) Mirror 34: 0x928ef655 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31dac): Mirror 17: 0x928ef655 (stale) Mirror 18: 0x928ef655 (stale) Mirror 20: 0x928ef655 (stale) Mirror 21: 0x928ef655 (stale) Mirror 23: 0x928ef655 (stale) Mirror 24: 0x928ef655 (stale) Mirror 29: 0x928ef655 (stale) Mirror 30: 0x928ef655 (stale) Mirror 31: 0x928ef655 (stale) Mirror 32: 0x928ef655 (stale) Mirror 34: 0x928ef655 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31dac: skipped 000997 1787384359.434079 read 0x01c4fa thru 0x022a13 (0x651a bytes) 000999 1787384359.447463 fallocate 0x018448 thru 0x019cc8 (0x1881 bytes) 001000 1787384359.456355 read 0x029f8b thru 0x02ca75 (0x2aeb bytes) All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 03:39:24 (1787384364) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x6b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 03:39:32 (1787384372) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204325 s, 10.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955994 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 03:39:40 (1787384380) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 739296 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5197920 .. Extending file size to 6663008 .. Extending file size to 4877376 .. Extending file size to 6982976 .. Extending file size to 6795680 .. Extending file size to 8941312 .. Extending file size to 8982592 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9221632 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9167488 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61228 61230 61231 61234 61235 done 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) 192.168.206.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:41:03 (1787384463) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4699360 .. Extending file size to 8201344 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5359008 .. Extending file size to 8713184 .. Extending file size to 6769984 .. Extending file size to 7239168 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7984608 .. Extending file size to 8332864 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8737760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62696 62697 62699 62700 62702 done 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) 192.168.206.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:42:24 (1787384544) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1269172 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18428 3588592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23780 3583240 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42208 7171832 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2132 1269172 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18428 3588592 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23780 3583240 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42208 7171832 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0969784 s, 21.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205052 s, 10.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8e8abe31 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:42:42 (1787384562) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118966 s, 17.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 03:42:48 (1787384568) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x310:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30f:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:42:54 (1787384574) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116593 s, 18.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x311:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:43:01 (1787384581) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0898532 s, 23.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x313:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:43:08 (1787384588) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1007 s, 20.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:43:42 (1787384622) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg624-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625212 s, 16.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:44:34 (1787384674) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg624-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132859 s, 15.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:45:22 (1787384722) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg624-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg624-server: Pool lustre.test_204f created Adding targets to pool oleg624-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg624-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0844029 s, 24.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg624-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg624-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg624-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:46:31 (1787384791) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:46:37 (1787384797) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:46:44 (1787384804) oleg624-server: Pool lustre.test_206 created Adding targets to pool oleg624-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg624-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg624-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg624-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg624-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:47:10 (1787384830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766913 s, 13.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763167 s, 13.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2f9904dc Mirror 3: 0x2f9904dc PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:47:21 (1787384841) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:47:24 (1787384844) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:47:27 (1787384847) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0578522 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x710000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc80000 Read this (4 bytes): BBBB 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:48:00 (1787384880) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0181837 s, 451 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:48:07 (1787384887) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468602 s, 22.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f6 lmm_fid: [0x200000405:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f6 lmm_fid: [0x200000405:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611638 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 03:48:09.000000000 -0400 Modify: 2026-08-22 03:48:09.000000000 -0400 Change: 2026-08-22 03:48:10.000000000 -0400 Birth: 2026-08-22 03:48:09.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:48:16 (1787384896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327839 s, 32.0 MB/s oleg624-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg624-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg624-server: O/280000401/d1/321: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:48:36 (1787384916) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.270671 s, 151 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203557 s, 201 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 03:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 03:48 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:48:44 (1787384924) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:48:52 (1787384932) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1797 sec =========== 03:49:02 (1787384942) === sanity-flr: start cleanup 03:49:04 (1787384944) === === sanity-flr: finish cleanup 03:49:13 (1787384953) ===