-----============= acceptance-small: sanity-flr ============----- Tue Sep 15 14:53:57 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg603-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:54:07 (1789498447) === oleg603-client.virtnet: executing check_config_client /mnt/lustre oleg603-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg603-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5506448800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5506448800.idle_timeout=debug disable quota as required oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:54:31 (1789498471) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8203] == sanity-flr test 0a: lfs mirror create with -N option == 14:54:32 (1789498472) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:54:41 (1789498481) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:54:42 (1789498482) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:54:44 (1789498484) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:54:56 (1789498496) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:54:57 (1789498497) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:54:59 (1789498499) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b5506448800.stats=clear osc.lustre-OST0001-osc-ffff9b5506448800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:55:03 (1789498503) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:55:09 (1789498509) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:55:13 (1789498513) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:55:15 (1789498515) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:55:19 (1789498519) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 14:55:24 (1789498524) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:55:28 (1789498528) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:55:32 (1789498532) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:55:35 (1789498535) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:55:41 (1789498541) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:55:45 (1789498545) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.688461 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643922 s, 16.3 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:55:51 (1789498551) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.27197 s, 61.7 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:55:56 (1789498556) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:56:02 (1789498562) Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (40s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:56:42 (1789498602) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.172287 s, 36.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (23s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:57:05 (1789498625) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (45s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:57:50 (1789498670) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63134 s, 32.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 15 14:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.512s user 0m0.000s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.580s user 0m0.000s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.513s user 0m0.001s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg603-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5506448800.ost_server_uuid in FULL state after 0 sec PASS 33b (46s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:58:37 (1789498717) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:58:39 (1789498719) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:58:41 (1789498721) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 14:58:43 (1789498723) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 14:58:46 (1789498726) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.286006 s, 44.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:58:53 (1789498733) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165476 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165468 s, 63.4 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:58:57 (1789498737) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:59:02 (1789498742) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000749009 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000686738 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103694 s, 988 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000914952 s, 1.1 MB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 14:59:09 (1789498749) create test files with size 15735140 .. c698d8d94f205c80b1601cf661b6fd20 - 369a9d47d1a07d8b4af7b5962a37433a - cff4b0e613c94e963bc4859a5d0fd12f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 14:59:21 (1789498761) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:59:27 (1789498767) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:59:31 (1789498771) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064119 s, 16.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203985 s, 51.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 14:59:35 (1789498775) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0579124 s, 36.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.157147 s, 26.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.068187 s, 30.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 14:59:42 (1789498782) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 14:59:44 (1789498784) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:59:46 (1789498786) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314518 s, 33.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 14:59:49 (1789498789) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.145271 s, 21.7 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:59:54 (1789498794) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:00:00 (1789498800) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.168055 s, 62.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:00:11 (1789498811) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b55182f9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.143848 s, 72.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b55052b6800.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:00:19 (1789498819) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (21s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:00:40 (1789498840) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:00:58 (1789498858) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 15:01:00 (1789498860) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.148931 s, 21.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.110835 s, 28.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x14c34db4 (nosync,stale) Mirror 2: 0x14c34db4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5abb380fd5d0d174b28a2d3a63609564 - mirror 1 checksum: c49fa5cae07e4368faa7c49836956c8b - mirror 2 checksum: 5abb380fd5d0d174b28a2d3a63609564 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T15:01:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T15:01:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5abb380fd5d0d174b28a2d3a63609564 - mirror 1 checksum: c49fa5cae07e4368faa7c49836956c8b - mirror 2 checksum: c49fa5cae07e4368faa7c49836956c8b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T15:01:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T15:01:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:01:05 (1789498865) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254767 s, 41.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:01:08 (1789498868) Failing mds1 on oleg603-server Stopping /mnt/lustre-mds1 (opts:) on oleg603-server 15:01:17 (1789498877) shut down facet: mds1 facet_host: oleg603-server facet_failover_host: oleg603-server Failover mds1 to oleg603-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0000 15:01:32 (1789498892) targets are mounted 15:01:33 (1789498893) facet_failover done oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182134 s, 57.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (35s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:01:43 (1789498903) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0201357 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106492 s, 1.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:01:48 (1789498908) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0293962 s, 697 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00515454 s, 4.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:01:53 (1789498913) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00500205 s, 3.3 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:01:57 (1789498917) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0286837 s, 714 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:02:01 (1789498921) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0059773 s, 685 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0240533 s, 170 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:02:05 (1789498925) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:02:17 (1789498937) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:02:40 (1789498960) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:02:59 (1789498979) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:03:18 (1789498998) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 15:03:21 (1789499001) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57488 mirror split pid 57489 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57488 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57489 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 15:04:25 (1789499065) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9066 skipping zero size read 000004 1789499066.523845 write 0x014056 thru 0x015ff2 (0x1f9d bytes) zero_range to largest ever: 0xf0b0 000005 1789499066.524958 trunc from 0x00d59d to 0x00f0af (0x1b13 bytes) 000006 1789499066.527728 trunc from 0x00220f to 0x008bf5 (0x69e7 bytes) 000007 1789499066.531001 write 0x0091b6 thru 0x01666b (0xd4b6 bytes) fallocating to largest ever: 0x31d11 000008 1789499066.536235 fallocate 0x02d505 thru 0x031d10 (0x480c bytes) 000009 1789499066.540053 mapread 0x00ea99 thru 0x01595f (0x6ec7 bytes) 000011 1789499066.541665 write 0x02c170 thru 0x03b3da (0xf26b bytes) 000012 1789499066.546520 trunc from 0x01b405 to 0x02759a (0xc196 bytes) 000013 1789499066.550189 read 0x032307 thru 0x03b3da (0x90d4 bytes) 000014 1789499066.550796 trunc from 0x014cba to 0x01ce0f (0x8156 bytes) 000016 1789499066.565351 fallocate 0x001907 thru 0x0096bf (0x7db9 bytes) truncating to largest ever: 0x3af1b 000017 1789499066.569739 trunc from 0x03b3db to 0x03af1a (0x4c0 bytes) 000018 1789499066.583866 MIRROR_EXTEND to 2 mirrors 000019 1789499066.941851 trunc from 0x03af1b to 0x0268f7 (0x14623 bytes) 000021 1789499067.003681 write 0x018ea1 thru 0x023c0d (0xad6d bytes) 000022 1789499067.010533 punch from 0x0211a3 to 0x0268f7 (0x5755 bytes) 000024 1789499067.023058 trunc from 0x0268f8 to 0x0227c6 (0x4131 bytes) 000025 1789499067.041447 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) 000026 1789499067.137393 fallocate 0x00f047 thru 0x0106a2 (0x165c bytes) 000027 1789499067.142059 mapwrite 0x01d61d thru 0x021e13 (0x47f7 bytes) 000028 1789499067.173681 punch from 0x00257b to 0x005177 (0x2bfd bytes) fallocating to largest ever: 0x3c37d 000029 1789499067.177353 fallocate 0x039762 thru 0x03c37c (0x2c1b bytes) truncating to largest ever: 0x3f449 000030 1789499067.180844 trunc from 0x03c37d to 0x03f448 (0x30cc bytes) 000031 1789499067.194902 read 0x02fa19 thru 0x030eb4 (0x149c bytes) 000032 1789499067.199091 write 0x004394 thru 0x011ad9 (0xd746 bytes) 000034 1789499067.209094 read 0x03867c thru 0x03c066 (0x39eb bytes) 000035 1789499067.219905 mapread 0x0180bc thru 0x021d7d (0x9cc2 bytes) 000037 1789499067.221945 read 0x01f3f2 thru 0x025801 (0x6410 bytes) 000038 1789499067.222778 trunc from 0x03f449 to 0x00e3d5 (0x31073 bytes) 000039 1789499067.240093 read 0x00a31f thru 0x00e3d5 (0x40b7 bytes) 000040 1789499067.240949 punch from 0x0002c5 to 0x008610 (0x834c bytes) 000041 1789499067.259501 mapread 0x00a89c thru 0x00e3d5 (0x3b3a bytes) 000042 1789499067.260548 mapread 0x0001b1 thru 0x00decb (0xdd1b bytes) 000043 1789499067.273180 mapread 0x004dbc thru 0x00e3d5 (0x961a bytes) 000044 1789499067.274187 read 0x00ce5b thru 0x00e3d5 (0x157b bytes) 000045 1789499067.275089 trunc from 0x005c6a to 0x008400 (0x2797 bytes) 000046 1789499067.281416 write 0x028fa4 thru 0x030b1d (0x7b7a bytes) 000047 1789499067.282812 mapread 0x01eed9 thru 0x023ca9 (0x4dd1 bytes) 000048 1789499067.293533 mapread 0x01706a thru 0x020902 (0x9899 bytes) 000049 1789499067.296388 trunc from 0x030b1e to 0x0287b7 (0x8366 bytes) 000050 1789499067.318949 write 0x014870 thru 0x01761f (0x2db0 bytes) 000051 1789499067.323908 MIRROR_EXTEND to 2 mirrors 000052 1789499067.804579 punch from 0x015cf0 to 0x01da37 (0x7d48 bytes) 000053 1789499067.830177 mapwrite 0x0341b1 thru 0x03c87d (0x86cd bytes) 000054 1789499067.908920 fallocate 0x02f191 thru 0x0329bc (0x382c bytes) 000055 1789499067.915296 fallocate 0x011ac0 thru 0x015a01 (0x3f42 bytes) 000056 1789499067.922611 trunc from 0x03c87e to 0x0374b1 (0x53cc bytes) 000057 1789499067.933421 mapread 0x001a28 thru 0x0086f6 (0x6ccf bytes) 000058 1789499067.940508 punch from 0x02e652 to 0x0374b1 (0x8e60 bytes) 000059 1789499067.945835 write 0x02bcba thru 0x030bd1 (0x4f18 bytes) 000060 1789499067.953913 read 0x0139cd thru 0x0156c2 (0x1cf6 bytes) 000061 1789499067.958327 punch from 0x00c0b1 to 0x01a5f7 (0xe547 bytes) 000062 1789499067.961888 mapread 0x027606 thru 0x028b5f (0x155a bytes) 000064 1789499067.970222 write 0x00cafd thru 0x01a93d (0xde41 bytes) 000065 1789499067.977706 read 0x01e8b9 thru 0x028f0d (0xa655 bytes) 000066 1789499067.984288 trunc from 0x02ef55 to 0x0374b1 (0x855d bytes) 000067 1789499067.994018 write 0x003cab thru 0x0054bd (0x1813 bytes) 000068 1789499067.995112 mapread 0x007496 thru 0x009bf8 (0x2763 bytes) 000069 1789499067.995992 mapread 0x01b1ff thru 0x029e74 (0xec76 bytes) 000070 1789499067.997352 read 0x031d56 thru 0x0374b1 (0x575c bytes) 000071 1789499068.002879 write 0x00e22f thru 0x01d9b8 (0xf78a bytes) 000074 1789499068.005596 fallocate 0x008374 thru 0x010587 (0x8214 bytes) 000075 1789499068.015828 trunc from 0x0374b2 to 0x037ec6 (0xa15 bytes) 000076 1789499068.025536 read 0x026e94 thru 0x033246 (0xc3b3 bytes) 000077 1789499068.026152 fallocate 0x007ca1 thru 0x0171a5 (0xf505 bytes) 000078 1789499068.030299 trunc from 0x037ec7 to 0x03af4a (0x3084 bytes) fallocating to largest ever: 0x40000 000079 1789499068.053968 fallocate 0x03ac08 thru 0x03ffff (0x53f8 bytes) 000080 1789499068.058094 read 0x031bc1 thru 0x031db0 (0x1f0 bytes) 000081 1789499068.065724 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c0a0076 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd056919f Mirror 3: 0xd056919f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000082 1789499068.649655 trunc from 0x040000 to 0x03917e (0x6e81 bytes) 000083 1789499068.688826 trunc from 0x03917f to 0x02c989 (0xc7f5 bytes) 000084 1789499068.702523 trunc from 0x01dd75 to 0x02247b (0x4707 bytes) 000085 1789499068.706459 punch from 0x01733b to 0x020e30 (0x9af6 bytes) 000086 1789499068.712495 mapwrite 0x010f05 thru 0x017707 (0x6803 bytes) 000087 1789499068.751876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 2: 0xf953f40d (stale) Mirror 3: 0xd7d09d19 (stale) CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 1: 0x107d42ab Mirror 4: 0x107d42ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c98a: skipped 000088 1789499069.395116 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 2: 0xf953f40d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 2: 0xf953f40d (stale) CRC-32 checksum value for chunk [0, 0x2c98a): Mirror 1: 0x107d42ab Mirror 4: 0x107d42ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c98a: skipped 000089 1789499069.783835 fallocate 0x021cf2 thru 0x02a1b2 (0x84c1 bytes) 000090 1789499069.829383 fallocate 0x017465 thru 0x017cd9 (0x875 bytes) 000091 1789499069.833466 trunc from 0x02c98a to 0x0277e2 (0x51a7 bytes) 000092 1789499069.858313 fallocate 0x02d098 thru 0x03202b (0x4f94 bytes) 000093 1789499069.862171 write 0x004e6f thru 0x00f16e (0xa300 bytes) 000095 1789499069.872563 write 0x00e0b2 thru 0x016412 (0x8361 bytes) 000096 1789499069.877879 fallocate 0x008be5 thru 0x00d3a6 (0x47c2 bytes) 000097 1789499069.882165 mapwrite 0x00f076 thru 0x01b6ac (0xc637 bytes) 000098 1789499069.918473 trunc from 0x02906d to 0x02be58 (0x2dec bytes) 000099 1789499069.921595 mapwrite 0x030c0e thru 0x038fa3 (0x8396 bytes) 000100 1789499069.969186 punch from 0x013cce to 0x019692 (0x59c5 bytes) 000101 1789499069.974676 write 0x022fd2 thru 0x03143b (0xe46a bytes) 000102 1789499069.983683 read 0x0108b6 thru 0x019002 (0x874d bytes) 000103 1789499069.994876 trunc from 0x03866d to 0x038fa3 (0x937 bytes) 000105 1789499069.999876 mapwrite 0x025356 thru 0x030f75 (0xbc20 bytes) 000106 1789499070.026502 read 0x005aef thru 0x010930 (0xae42 bytes) 000107 1789499070.027380 fallocate 0x01e323 thru 0x02d252 (0xef30 bytes) 000108 1789499070.034702 punch from 0x01d11c to 0x01d71c (0x601 bytes) 000109 1789499070.041723 read 0x029f11 thru 0x03769b (0xd78b bytes) 000110 1789499070.042619 trunc from 0x00b59d to 0x014e70 (0x98d4 bytes) 000111 1789499070.059855 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38fa4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fa4): Mirror 2: 0x6ed294b4 (stale) Mirror 4: 0x342df744 (stale) CRC-32 checksum value for chunk [0, 0x38fa4): Mirror 1: 0xae8ebdc0 Mirror 5: 0xae8ebdc0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fa4: skipped 000112 1789499070.763311 mapread 0x012497 thru 0x014e63 (0x29cd bytes) 000113 1789499070.814581 trunc from 0x038fa4 to 0x02f4b0 (0x9af3 bytes) 000114 1789499070.871478 fallocate 0x006561 thru 0x015a76 (0xf516 bytes) 000115 1789499070.878982 mapwrite 0x023ce9 thru 0x024dd2 (0x10ea bytes) 000116 1789499070.906070 write 0x0145af thru 0x017857 (0x32a9 bytes) 000117 1789499070.919784 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f4b1): Mirror 4: 0x2cdaaaf8 (stale) Mirror 5: 0x1465f6be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4b1): Mirror 4: 0x2cdaaaf8 (stale) Mirror 5: 0x1465f6be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4b1: skipped 000118 1789499071.191698 mapwrite 0x00215e thru 0x005147 (0x2fea bytes) 000119 1789499071.239064 trunc from 0x015d60 to 0x025d1b (0xffbc bytes) 000120 1789499071.261593 punch from 0x010f77 to 0x01c642 (0xb6cc bytes) 000121 1789499071.265899 punch from 0x00da67 to 0x015206 (0x77a0 bytes) 000122 1789499071.270099 read 0x012c60 thru 0x01e965 (0xbd06 bytes) 000123 1789499071.277854 mapwrite 0x00a6e3 thru 0x0161bd (0xbadb bytes) 000124 1789499071.313322 read 0x02a084 thru 0x02f4b0 (0x542d bytes) 000125 1789499071.320836 trunc from 0x02f4b1 to 0x03fd90 (0x108e0 bytes) 000127 1789499071.335683 write 0x017eac thru 0x019833 (0x1988 bytes) 000129 1789499071.337667 fallocate 0x0190b2 thru 0x029088 (0xffd7 bytes) 000131 1789499071.344756 mapwrite 0x0285fd thru 0x0354d4 (0xced8 bytes) 000132 1789499071.388447 mapread 0x02d107 thru 0x02f799 (0x2693 bytes) 000133 1789499071.389353 trunc from 0x022367 to 0x02f00e (0xcca8 bytes) 000134 1789499071.394649 mapread 0x02c432 thru 0x02efb6 (0x2b85 bytes) 000135 1789499071.408825 write 0x01c82c thru 0x0296f0 (0xcec5 bytes) 000137 1789499071.417323 punch from 0x0162cc to 0x018293 (0x1fc8 bytes) 000138 1789499071.421912 mapread 0x00f19e thru 0x012d3c (0x3b9f bytes) 000139 1789499071.423987 read 0x00d055 thru 0x018756 (0xb702 bytes) 000140 1789499071.431680 trunc from 0x03c443 to 0x03fd90 (0x394e bytes) 000141 1789499071.436933 mapwrite 0x021a9c thru 0x02fcc0 (0xe225 bytes) 000143 1789499071.460819 read 0x00177c thru 0x0050f4 (0x3979 bytes) 000144 1789499071.465309 MIRROR_RESYNC 3 mirrors 000146 1789499072.090691 trunc from 0x029f71 to 0x0385b5 (0xe645 bytes) 000147 1789499072.136677 trunc from 0x012ad7 to 0x01eec5 (0xc3ef bytes) 000148 1789499072.140457 mapwrite 0x0139dd thru 0x017f72 (0x4596 bytes) 000149 1789499072.173604 trunc from 0x03fd91 to 0x02e62c (0x11764 bytes) 000150 1789499072.208880 punch from 0x00b00f to 0x00ca13 (0x1a05 bytes) 000151 1789499072.213656 mapread 0x00fd42 thru 0x01f5c4 (0xf883 bytes) 000152 1789499072.226188 read 0x0061de thru 0x007bfb (0x1a1e bytes) 000153 1789499072.236536 mapwrite 0x01922b thru 0x025347 (0xc11d bytes) 000154 1789499072.249660 punch from 0x018e67 to 0x0217d8 (0x8972 bytes) 000155 1789499072.257884 trunc from 0x00b420 to 0x00b781 (0x362 bytes) 000156 1789499072.265122 trunc from 0x02e62d to 0x007896 (0x26d96 bytes) 000157 1789499072.284684 fallocate 0x01c4e7 thru 0x0264c6 (0x9fe0 bytes) 000158 1789499072.292944 read 0x000e8b thru 0x002268 (0x13de bytes) 000159 1789499072.304967 write 0x03434a thru 0x03f40b (0xb0c2 bytes) 000160 1789499072.308934 read 0x0189d0 thru 0x024055 (0xb686 bytes) 000161 1789499072.322449 punch from 0x02706f to 0x02f972 (0x8904 bytes) 000163 1789499072.339399 write 0x0224c9 thru 0x022a99 (0x5d1 bytes) 000164 1789499072.340690 write 0x03a614 thru 0x03e355 (0x3d42 bytes) 000165 1789499072.342123 trunc from 0x03f40c to 0x03778c (0x7c7f bytes) 000166 1789499072.358605 punch from 0x00e847 to 0x01bce3 (0xd49d bytes) 000167 1789499072.363072 read 0x012cbe thru 0x022a2c (0xfd6f bytes) 000168 1789499072.378610 trunc from 0x036dde to 0x03778c (0x9af bytes) 000169 1789499072.391079 read 0x022f03 thru 0x02d0bd (0xa1bb bytes) 000170 1789499072.392286 fallocate 0x024621 thru 0x031b07 (0xd4e7 bytes) 000171 1789499072.396041 read 0x02753e thru 0x02a01e (0x2ae1 bytes) 000172 1789499072.404842 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3778d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3778d): Mirror 4: 0xf824afe1 (stale) Mirror 5: 0xf824afe1 (stale) CRC-32 checksum value for chunk [0, 0x3778d): Mirror 1: 0xea1c560d Mirror 6: 0xea1c560d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3778d: skipped 000174 1789499073.070685 read 0x022d63 thru 0x029a18 (0x6cb6 bytes) 000176 1789499073.104609 trunc from 0x004d07 to 0x0097a6 (0x4aa0 bytes) 000177 1789499073.149760 mapread 0x002e09 thru 0x00bc10 (0x8e08 bytes) 000178 1789499073.175443 trunc from 0x03778d to 0x031da8 (0x59e4 bytes) 000179 1789499073.215093 trunc from 0x0111a2 to 0x012a1d (0x187c bytes) 000180 1789499073.219022 write 0x01b1fe thru 0x026bba (0xb9bd bytes) 000182 1789499073.234578 fallocate 0x03f3ce thru 0x03ffff (0xc32 bytes) 000183 1789499073.241096 trunc from 0x031da9 to 0x01d9e2 (0x143c6 bytes) 000184 1789499073.260749 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: 0xe31721bd (stale) Mirror 5: 0xe31721bd (stale) Mirror 6: 0xc654b596 (stale) CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 1: 0xcf2e2f53 Mirror 7: 0xcf2e2f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d9e3: skipped 000185 1789499073.979676 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: 0xe31721bd (stale) Mirror 5: 0xe31721bd (stale) Mirror 6: 0xc654b596 (stale) CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 1: 0xcf2e2f53 Mirror 7: 0xcf2e2f53 Mirror 8: 0xcf2e2f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d9e3: skipped 000186 1789499074.571081 fallocate 0x006518 thru 0x00a7f2 (0x42db bytes) 000187 1789499074.606442 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 4: 0xe31721bd (stale) Mirror 5: 0xe31721bd (stale) Mirror 6: 0xc654b596 (stale) Mirror 7: 0xcf2e2f53 (stale) Mirror 8: 0xcf2e2f53 (stale) CRC-32 checksum value for chunk [0, 0x1d9e3): Mirror 1: 0xcf2e2f53 Mirror 9: 0xcf2e2f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d9e3: skipped 000188 1789499075.223081 punch from 0x018cb4 to 0x01d9e2 (0x4d2f bytes) 000189 1789499075.248108 trunc from 0x00d87f to 0x01ac14 (0xd396 bytes) 000190 1789499075.251350 punch from 0x0002b9 to 0x00faa3 (0xf7eb bytes) 000191 1789499075.255720 fallocate 0x00b252 thru 0x010d8f (0x5b3e bytes) 000192 1789499075.259606 read 0x01b8cc thru 0x01d9e2 (0x2117 bytes) 000193 1789499075.263963 mapwrite 0x014c64 thru 0x01e84d (0x9bea bytes) 000194 1789499075.321676 mapwrite 0x02a24e thru 0x035091 (0xae44 bytes) 000195 1789499075.373183 fallocate 0x00a929 thru 0x010e52 (0x652a bytes) 000196 1789499075.382065 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35092): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35092): Mirror 4: 0x3d9369b8 (stale) Mirror 5: 0x3d9369b8 (stale) Mirror 6: 0xc6b698d6 (stale) Mirror 7: 0xb4476970 (stale) Mirror 8: 0xb4476970 (stale) Mirror 9: 0xb4476970 (stale) CRC-32 checksum value for chunk [0, 0x35092): Mirror 1: 0xc9f2cf08 Mirror 10: 0xc9f2cf08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35092: skipped 000197 1789499075.917470 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35092): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35092): Mirror 4: 0x3d9369b8 (stale) Mirror 5: 0x3d9369b8 (stale) Mirror 6: 0xc6b698d6 (stale) Mirror 7: 0xb4476970 (stale) Mirror 8: 0xb4476970 (stale) Mirror 9: 0xb4476970 (stale) CRC-32 checksum value for chunk [0, 0x35092): Mirror 1: 0xc9f2cf08 Mirror 10: 0xc9f2cf08 Mirror 11: 0xc9f2cf08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35092: skipped 000198 1789499076.506002 trunc from 0x00ee3c to 0x01c06d (0xd232 bytes) 000199 1789499076.541456 punch from 0x003ab5 to 0x00c212 (0x875e bytes) 000201 1789499076.545232 trunc from 0x02cfa2 to 0x035091 (0x80f0 bytes) 000202 1789499076.549734 read 0x022365 thru 0x0307cb (0xe467 bytes) 000203 1789499076.563531 trunc from 0x035092 to 0x0048a5 (0x307ec bytes) 000204 1789499076.613836 read 0x003d32 thru 0x0048a5 (0xb74 bytes) 000205 1789499076.622944 MIRROR_RESYNC 9 mirrors 000206 1789499077.335198 MIRROR_EXTEND to 10 mirrors 000208 1789499077.744182 fallocate 0x03b6d0 thru 0x03ffff (0x4930 bytes) 000209 1789499077.774050 punch from 0x0012e7 to 0x00aaee (0x9808 bytes) 000210 1789499077.777927 punch from 0x022701 to 0x02d62d (0xaf2d bytes) 000211 1789499077.782071 read 0x01e63d thru 0x02809d (0x9a61 bytes) 000212 1789499077.790851 read 0x010592 thru 0x0185c6 (0x8035 bytes) 000213 1789499077.800925 trunc from 0x008296 to 0x016075 (0xdde0 bytes) 000215 1789499077.805908 punch from 0x03b424 to 0x03ffff (0x4bdc bytes) 000216 1789499077.813796 MIRROR_RESYNC 10 mirrors 000217 1789499078.584868 trunc from 0x040000 to 0x030f5f (0xf0a0 bytes) 000218 1789499078.608048 trunc from 0x01040b to 0x01ce98 (0xca8e bytes) 000219 1789499078.612045 punch from 0x012394 to 0x013235 (0xea2 bytes) 000220 1789499078.615807 trunc from 0x005e45 to 0x00ca09 (0x6bc5 bytes) 000221 1789499078.619381 fallocate 0x008886 thru 0x00bff3 (0x376e bytes) 000222 1789499078.624345 mapwrite 0x02ccb1 thru 0x039598 (0xc8e8 bytes) 000223 1789499078.679883 punch from 0x005064 to 0x0087c4 (0x3761 bytes) 000224 1789499078.685093 punch from 0x0381fe to 0x039598 (0x139b bytes) 000225 1789499078.690384 write 0x003485 thru 0x009cae (0x682a bytes) 000226 1789499078.697085 read 0x0167bc thru 0x01a724 (0x3f69 bytes) 000227 1789499078.709990 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39599): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39599): Mirror 4: 0xbec73fc9 (stale) Mirror 5: 0xbec73fc9 (stale) Mirror 6: 0xbec73fc9 (stale) Mirror 7: 0xbec73fc9 (stale) Mirror 8: 0xbec73fc9 (stale) Mirror 9: 0xbec73fc9 (stale) Mirror 10: 0xbec73fc9 (stale) Mirror 11: 0xbec73fc9 (stale) Mirror 12: 0xbec73fc9 (stale) CRC-32 checksum value for chunk [0, 0x39599): Mirror 1: 0x357a38bc Mirror 13: 0x357a38bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39599: skipped 000228 1789499079.273054 fallocate 0x00c700 thru 0x01af45 (0xe846 bytes) 000229 1789499079.298522 fallocate 0x02cfcf thru 0x0351a6 (0x81d8 bytes) 000230 1789499079.303467 trunc from 0x011f5f to 0x018ea7 (0x6f49 bytes) 000231 1789499079.306371 trunc from 0x039599 to 0x021cfa (0x1789e bytes) 000232 1789499079.339979 punch from 0x016460 to 0x021cfa (0xb89b bytes) 000233 1789499079.344555 fallocate 0x026376 thru 0x0333f9 (0xd084 bytes) 000234 1789499079.349239 mapwrite 0x000ea6 thru 0x004a3e (0x3b99 bytes) 000235 1789499079.388844 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x333fa): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333fa): Mirror 4: 0xb99206c1 (stale) Mirror 5: 0xb99206c1 (stale) Mirror 6: 0xb99206c1 (stale) Mirror 7: 0xb99206c1 (stale) Mirror 8: 0xb99206c1 (stale) Mirror 9: 0xb99206c1 (stale) Mirror 10: 0xb99206c1 (stale) Mirror 11: 0xb99206c1 (stale) Mirror 12: 0xb99206c1 (stale) Mirror 13: 0xcfc9334a (stale) CRC-32 checksum value for chunk [0, 0x333fa): Mirror 1: 0xa3702761 Mirror 14: 0xa3702761 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333fa: skipped 000236 1789499079.937637 fallocate 0x03994b thru 0x03c7e9 (0x2e9f bytes) 000237 1789499079.963490 read 0x0372e4 thru 0x03c7e9 (0x5506 bytes) 000238 1789499079.968193 write 0x03f1fd thru 0x03ffff (0xe03 bytes) 000239 1789499079.972602 trunc from 0x001bb7 to 0x006bcf (0x5019 bytes) 000240 1789499079.977951 fallocate 0x017ee2 thru 0x0201b5 (0x82d4 bytes) 000241 1789499079.982075 mapwrite 0x00bd77 thru 0x00fef4 (0x417e bytes) 000242 1789499080.014615 fallocate 0x004a21 thru 0x00f804 (0xade4 bytes) 000243 1789499080.019074 fallocate 0x01c84f thru 0x022ff0 (0x67a2 bytes) 000244 1789499080.024315 write 0x02c6ea thru 0x02e6c9 (0x1fe0 bytes) 000245 1789499080.032045 read 0x000b55 thru 0x0066b5 (0x5b61 bytes) 000248 1789499080.042395 mapread 0x00c94e thru 0x014097 (0x774a bytes) 000249 1789499080.043209 fallocate 0x01b5b3 thru 0x026a72 (0xb4c0 bytes) 000251 1789499080.047603 mapread 0x01f00b thru 0x02de94 (0xee8a bytes) 000252 1789499080.048588 trunc from 0x040000 to 0x02075b (0x1f8a4 bytes) 000253 1789499080.074471 MIRROR_RESYNC 12 mirrors 000254 1789499081.314508 trunc from 0x00f6ed to 0x01c0d4 (0xc9e8 bytes) 000255 1789499081.335013 read 0x01b98e thru 0x02075b (0x4dce bytes) 000256 1789499081.343686 punch from 0x01b83b to 0x02075b (0x4f21 bytes) 000257 1789499081.349194 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2075c): Mirror 5: 0x83d6de72 (stale) Mirror 6: 0x83d6de72 (stale) Mirror 7: 0x83d6de72 (stale) Mirror 8: 0x83d6de72 (stale) Mirror 9: 0x83d6de72 (stale) Mirror 10: 0x83d6de72 (stale) Mirror 11: 0x83d6de72 (stale) Mirror 12: 0x83d6de72 (stale) Mirror 13: 0x83d6de72 (stale) Mirror 14: 0x83d6de72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2075c): Mirror 5: 0x83d6de72 (stale) Mirror 6: 0x83d6de72 (stale) Mirror 7: 0x83d6de72 (stale) Mirror 8: 0x83d6de72 (stale) Mirror 9: 0x83d6de72 (stale) Mirror 10: 0x83d6de72 (stale) Mirror 11: 0x83d6de72 (stale) Mirror 12: 0x83d6de72 (stale) Mirror 13: 0x83d6de72 (stale) Mirror 14: 0x83d6de72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2075c: skipped 000259 1789499081.617834 read 0x01a7bd thru 0x02075b (0x5f9f bytes) 000260 1789499081.623493 trunc from 0x02075c to 0x010d81 (0xf9da bytes) 000262 1789499081.635459 trunc from 0x0101d3 to 0x010d81 (0xbaf bytes) 000263 1789499081.637657 mapwrite 0x005991 thru 0x0091e2 (0x3852 bytes) 000264 1789499081.667379 mapread 0x00fd62 thru 0x010d81 (0x1020 bytes) 000267 1789499081.668790 mapwrite 0x03490a thru 0x03d8ee (0x8fe5 bytes) 000268 1789499081.712001 read 0x008ca1 thru 0x00d2c2 (0x4622 bytes) 000269 1789499081.712637 mapwrite 0x012e14 thru 0x017953 (0x4b40 bytes) 000270 1789499081.729953 write 0x02b324 thru 0x032c5d (0x793a bytes) 000271 1789499081.731226 write 0x02322f thru 0x0314e6 (0xe2b8 bytes) 000272 1789499081.732644 read 0x03be37 thru 0x03d8ee (0x1ab8 bytes) 000273 1789499081.733156 mapwrite 0x037d39 thru 0x03eaba (0x6d82 bytes) 000274 1789499081.755281 read 0x03d605 thru 0x03eaba (0x14b6 bytes) 000275 1789499081.755821 punch from 0x015612 to 0x015bf9 (0x5e8 bytes) 000276 1789499081.761151 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: 0xcfec74ca (stale) Mirror 6: 0xcfec74ca (stale) Mirror 7: 0xcfec74ca (stale) Mirror 8: 0xcfec74ca (stale) Mirror 9: 0xcfec74ca (stale) Mirror 10: 0xcfec74ca (stale) Mirror 11: 0xcfec74ca (stale) Mirror 12: 0xcfec74ca (stale) Mirror 13: 0xcfec74ca (stale) Mirror 14: 0xcfec74ca (stale) CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 1: 0xf7778a5a Mirror 15: 0xf7778a5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabb: skipped 000277 1789499082.283383 write 0x010ea2 thru 0x01e64d (0xd7ac bytes) 000278 1789499082.317314 fallocate 0x01b1b9 thru 0x021d54 (0x6b9c bytes) 000279 1789499082.324231 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: 0xcfec74ca (stale) Mirror 6: 0xcfec74ca (stale) Mirror 7: 0xcfec74ca (stale) Mirror 8: 0xcfec74ca (stale) Mirror 9: 0xcfec74ca (stale) Mirror 10: 0xcfec74ca (stale) Mirror 11: 0xcfec74ca (stale) Mirror 12: 0xcfec74ca (stale) Mirror 13: 0xcfec74ca (stale) Mirror 14: 0xcfec74ca (stale) Mirror 15: 0xf7778a5a (stale) CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 1: 0xedf6382c Mirror 16: 0xedf6382c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabb: skipped 000280 1789499082.932518 read 0x00a808 thru 0x00e3f6 (0x3bef bytes) 000281 1789499082.944306 MIRROR_RESYNC 13 mirrors 000283 1789499084.789631 read 0x006435 thru 0x00a8d1 (0x449d bytes) 000284 1789499084.797172 trunc from 0x036225 to 0x03eaba (0x8896 bytes) 000285 1789499084.831522 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: 0xedf6382c (stale) Mirror 6: 0xedf6382c (stale) Mirror 7: 0xedf6382c (stale) Mirror 8: 0xedf6382c (stale) Mirror 9: 0xedf6382c (stale) Mirror 10: 0xedf6382c (stale) Mirror 11: 0xedf6382c (stale) Mirror 12: 0xedf6382c (stale) Mirror 13: 0xedf6382c (stale) Mirror 14: 0xedf6382c (stale) Mirror 15: 0xedf6382c (stale) Mirror 16: 0xedf6382c (stale) CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 1: 0xcb265298 Mirror 17: 0xcb265298 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabb: skipped 000286 1789499085.501687 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 5: 0xedf6382c (stale) Mirror 6: 0xedf6382c (stale) Mirror 7: 0xedf6382c (stale) Mirror 8: 0xedf6382c (stale) Mirror 9: 0xedf6382c (stale) Mirror 10: 0xedf6382c (stale) Mirror 11: 0xedf6382c (stale) Mirror 12: 0xedf6382c (stale) Mirror 13: 0xedf6382c (stale) Mirror 14: 0xedf6382c (stale) Mirror 15: 0xedf6382c (stale) Mirror 16: 0xedf6382c (stale) CRC-32 checksum value for chunk [0, 0x3eabb): Mirror 1: 0xcb265298 Mirror 17: 0xcb265298 Mirror 18: 0xcb265298 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabb: skipped 000287 1789499086.182298 mapread 0x02ab3a thru 0x03594c (0xae13 bytes) 000288 1789499086.199142 trunc from 0x0378b4 to 0x03eaba (0x7207 bytes) 000289 1789499086.239211 mapread 0x010c54 thru 0x019557 (0x8904 bytes) 000290 1789499086.251448 mapread 0x0127c6 thru 0x017e03 (0x563e bytes) 000291 1789499086.252219 read 0x02d0f6 thru 0x03341f (0x632a bytes) 000292 1789499086.257142 trunc from 0x03eabb to 0x03e945 (0x175 bytes) 000293 1789499086.268863 fallocate 0x031f46 thru 0x0395dc (0x7697 bytes) 000294 1789499086.272359 mapread 0x012e3e thru 0x01e834 (0xb9f7 bytes) 000295 1789499086.273349 mapread 0x0266b0 thru 0x026fcd (0x91e bytes) 000296 1789499086.282988 read 0x0117f5 thru 0x021334 (0xfb40 bytes) 000297 1789499086.283606 mapwrite 0x010c7f thru 0x0160d3 (0x5455 bytes) 000298 1789499086.298446 trunc from 0x007460 to 0x00bc70 (0x4811 bytes) 000299 1789499086.302760 trunc from 0x002a55 to 0x003240 (0x7ec bytes) 000300 1789499086.306146 trunc from 0x034099 to 0x03e945 (0xa8ad bytes) 000301 1789499086.309833 mapread 0x018059 thru 0x027eef (0xfe97 bytes) 000302 1789499086.311629 mapwrite 0x0305fe thru 0x03df16 (0xd919 bytes) 000303 1789499086.336012 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 6: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 14: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 6: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 14: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e946: skipped 000304 1789499086.767910 punch from 0x00cfe4 to 0x00f936 (0x2953 bytes) 000305 1789499086.771402 write 0x02d729 thru 0x033df7 (0x66cf bytes) 000306 1789499086.784460 punch from 0x023623 to 0x03237e (0xed5c bytes) 000307 1789499086.792921 trunc from 0x0311db to 0x034552 (0x3378 bytes) 000308 1789499086.801500 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 6: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 6: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e946: skipped 000309 1789499087.251149 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e946): Mirror 5: 0xebf459c8 (stale) Mirror 7: 0xebf459c8 (stale) Mirror 9: 0xebf459c8 (stale) Mirror 10: 0xebf459c8 (stale) Mirror 11: 0xebf459c8 (stale) Mirror 12: 0xebf459c8 (stale) Mirror 13: 0xebf459c8 (stale) Mirror 15: 0xebf459c8 (stale) Mirror 16: 0xebf459c8 (stale) Mirror 17: 0x7f793fe6 (stale) Mirror 18: 0x7f793fe6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e946: skipped 000310 1789499087.615621 fallocate 0x039bfe thru 0x03ffff (0x6402 bytes) 000311 1789499087.618492 mapwrite 0x037ba8 thru 0x03ffff (0x8458 bytes) 000312 1789499087.657586 write 0x01ca76 thru 0x026311 (0x989c bytes) 000313 1789499087.665961 read 0x0202a0 thru 0x024932 (0x4693 bytes) 000314 1789499087.666602 trunc from 0x02acf3 to 0x033181 (0x848f bytes) 000317 1789499087.672597 fallocate 0x038aba thru 0x03ffff (0x7546 bytes) 000318 1789499087.675988 trunc from 0x040000 to 0x005c95 (0x3a36a bytes) 000319 1789499087.684155 punch from 0x000fbc to 0x005c95 (0x4cda bytes) 000322 1789499087.688314 write 0x0175f5 thru 0x02260c (0xb018 bytes) 000325 1789499087.690410 punch from 0x006de1 to 0x00dc4a (0x6e6a bytes) 000326 1789499087.693892 punch from 0x014265 to 0x014d9c (0xb38 bytes) 000327 1789499087.696808 write 0x0224d4 thru 0x031f9c (0xfac9 bytes) 000328 1789499087.698358 write 0x03e874 thru 0x03f6b2 (0xe3f bytes) 000329 1789499087.701769 MIRROR_RESYNC 12 mirrors 000330 1789499089.453348 trunc from 0x012556 to 0x013054 (0xaff bytes) 000331 1789499089.478979 mapread 0x004640 thru 0x006ab3 (0x2474 bytes) 000334 1789499089.503402 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f6b3): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6b3): Mirror 5: 0xc222d161 (stale) Mirror 7: 0xc222d161 (stale) Mirror 9: 0xc222d161 (stale) Mirror 10: 0xc222d161 (stale) Mirror 11: 0xc222d161 (stale) Mirror 12: 0xc222d161 (stale) Mirror 13: 0xc222d161 (stale) Mirror 15: 0xc222d161 (stale) Mirror 16: 0xc222d161 (stale) Mirror 17: 0xc222d161 (stale) Mirror 18: 0xc222d161 (stale) CRC-32 checksum value for chunk [0, 0x3f6b3): Mirror 1: 0xc222d161 Mirror 19: 0xc222d161 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6b3: skipped 000335 1789499090.133095 write 0x02fcdf thru 0x03530f (0x5631 bytes) 000336 1789499090.168097 write 0x03c407 thru 0x03ffff (0x3bf9 bytes) 000338 1789499090.177493 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd0f40eb7 (stale) Mirror 7: 0xd0f40eb7 (stale) Mirror 9: 0xd0f40eb7 (stale) Mirror 10: 0xd0f40eb7 (stale) Mirror 11: 0xd0f40eb7 (stale) Mirror 12: 0xd0f40eb7 (stale) Mirror 13: 0xd0f40eb7 (stale) Mirror 15: 0xd0f40eb7 (stale) Mirror 16: 0xd0f40eb7 (stale) Mirror 17: 0xd0f40eb7 (stale) Mirror 19: 0xd0f40eb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd0f40eb7 (stale) Mirror 7: 0xd0f40eb7 (stale) Mirror 9: 0xd0f40eb7 (stale) Mirror 10: 0xd0f40eb7 (stale) Mirror 11: 0xd0f40eb7 (stale) Mirror 12: 0xd0f40eb7 (stale) Mirror 13: 0xd0f40eb7 (stale) Mirror 15: 0xd0f40eb7 (stale) Mirror 16: 0xd0f40eb7 (stale) Mirror 17: 0xd0f40eb7 (stale) Mirror 19: 0xd0f40eb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000340 1789499090.450614 read 0x01cd98 thru 0x022f22 (0x618b bytes) 000341 1789499090.455329 fallocate 0x02a332 thru 0x0396d2 (0xf3a1 bytes) 000342 1789499090.460148 mapwrite 0x008d06 thru 0x018a01 (0xfcfc bytes) 000343 1789499090.490872 trunc from 0x00e1db to 0x015133 (0x6f59 bytes) 000344 1789499090.493963 fallocate 0x00f818 thru 0x013747 (0x3f30 bytes) 000345 1789499090.497507 mapread 0x0029ef thru 0x0044f4 (0x1b06 bytes) 000346 1789499090.502764 mapread 0x0293d5 thru 0x02fed3 (0x6aff bytes) 000347 1789499090.513670 trunc from 0x040000 to 0x02836d (0x17c92 bytes) 000349 1789499090.532066 trunc from 0x02836e to 0x02a6a6 (0x2339 bytes) 000350 1789499090.541606 trunc from 0x027eae to 0x02a6a6 (0x27f9 bytes) 000352 1789499090.548106 read 0x0092f3 thru 0x00b9f4 (0x2702 bytes) 000353 1789499090.548721 mapread 0x0228ca thru 0x02a6a6 (0x7ddd bytes) 000354 1789499090.553890 trunc from 0x02a6a7 to 0x0244ff (0x61a7 bytes) 000356 1789499090.569774 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24500): Mirror 5: 0x7e9ab77e (stale) Mirror 7: 0x7e9ab77e (stale) Mirror 9: 0x7e9ab77e (stale) Mirror 10: 0x7e9ab77e (stale) Mirror 11: 0x7e9ab77e (stale) Mirror 12: 0x7e9ab77e (stale) Mirror 13: 0x7e9ab77e (stale) Mirror 15: 0x7e9ab77e (stale) Mirror 16: 0x7e9ab77e (stale) Mirror 19: 0x7e9ab77e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24500): Mirror 5: 0x7e9ab77e (stale) Mirror 7: 0x7e9ab77e (stale) Mirror 9: 0x7e9ab77e (stale) Mirror 10: 0x7e9ab77e (stale) Mirror 11: 0x7e9ab77e (stale) Mirror 12: 0x7e9ab77e (stale) Mirror 13: 0x7e9ab77e (stale) Mirror 15: 0x7e9ab77e (stale) Mirror 16: 0x7e9ab77e (stale) Mirror 19: 0x7e9ab77e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24500: skipped 000358 1789499090.851816 read 0x01bf26 thru 0x0239d9 (0x7ab4 bytes) 000359 1789499090.856157 trunc from 0x0090d9 to 0x013909 (0xa831 bytes) 000360 1789499090.860091 fallocate 0x031e4c thru 0x03388b (0x1a40 bytes) 000361 1789499090.863840 trunc from 0x03388c to 0x017cf4 (0x1bb97 bytes) 000362 1789499090.883875 mapwrite 0x012523 thru 0x0182df (0x5dbd bytes) 000363 1789499090.932196 MIRROR_RESYNC 11 mirrors 000364 1789499092.616521 mapread 0x00d651 thru 0x01752c (0x9edc bytes) 000365 1789499092.632698 trunc from 0x0182e0 to 0x02b1f7 (0x12f18 bytes) 000367 1789499092.673932 punch from 0x00ea78 to 0x0119af (0x2f38 bytes) 000369 1789499092.679842 trunc from 0x02b1f8 to 0x03798a (0xc793 bytes) 000370 1789499092.698097 fallocate 0x020265 thru 0x020e79 (0xc15 bytes) 000371 1789499092.704119 mapwrite 0x016a05 thru 0x0224dd (0xbad9 bytes) 000372 1789499092.743889 mapwrite 0x032988 thru 0x0380da (0x5753 bytes) 000373 1789499092.788176 write 0x03eb51 thru 0x03ffff (0x14af bytes) 000374 1789499092.789138 trunc from 0x0283f8 to 0x02b53c (0x3145 bytes) 000375 1789499092.799846 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7d4916d3 (stale) Mirror 7: 0x7d4916d3 (stale) Mirror 9: 0x7d4916d3 (stale) Mirror 10: 0x7d4916d3 (stale) Mirror 11: 0x7d4916d3 (stale) Mirror 12: 0x7d4916d3 (stale) Mirror 13: 0x7d4916d3 (stale) Mirror 15: 0x7d4916d3 (stale) Mirror 16: 0x7d4916d3 (stale) Mirror 19: 0x7d4916d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2a67ff0 Mirror 20: 0xd2a67ff0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1789499093.534724 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7d4916d3 (stale) Mirror 7: 0x7d4916d3 (stale) Mirror 9: 0x7d4916d3 (stale) Mirror 10: 0x7d4916d3 (stale) Mirror 11: 0x7d4916d3 (stale) Mirror 13: 0x7d4916d3 (stale) Mirror 15: 0x7d4916d3 (stale) Mirror 16: 0x7d4916d3 (stale) Mirror 19: 0x7d4916d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7d4916d3 (stale) Mirror 7: 0x7d4916d3 (stale) Mirror 9: 0x7d4916d3 (stale) Mirror 10: 0x7d4916d3 (stale) Mirror 11: 0x7d4916d3 (stale) Mirror 13: 0x7d4916d3 (stale) Mirror 15: 0x7d4916d3 (stale) Mirror 16: 0x7d4916d3 (stale) Mirror 19: 0x7d4916d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2a67ff0 Mirror 20: 0xd2a67ff0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000377 1789499094.138075 trunc from 0x018b2a to 0x023fb5 (0xb48c bytes) 000378 1789499094.171840 trunc from 0x040000 to 0x03937f (0x6c80 bytes) 000379 1789499094.187500 trunc from 0x039380 to 0x009ec8 (0x2f4b7 bytes) 000380 1789499094.201173 mapread 0x00074b thru 0x004311 (0x3bc7 bytes) 000381 1789499094.215854 write 0x012b82 thru 0x020d31 (0xe1b0 bytes) 000383 1789499094.217935 fallocate 0x013463 thru 0x020dd7 (0xd975 bytes) 000384 1789499094.221963 trunc from 0x020dd8 to 0x03710f (0x16338 bytes) 000385 1789499094.232175 write 0x01a9f8 thru 0x01f464 (0x4a6d bytes) 000386 1789499094.233227 punch from 0x018ad4 to 0x028a35 (0xff62 bytes) 000387 1789499094.242493 mapread 0x00120d thru 0x00a4ff (0x92f3 bytes) 000388 1789499094.247098 write 0x01f5b4 thru 0x020cb8 (0x1705 bytes) 000389 1789499094.256888 write 0x024202 thru 0x030d1e (0xcb1d bytes) 000390 1789499094.266101 mapwrite 0x029fe5 thru 0x037a9f (0xdabb bytes) 000391 1789499094.293692 write 0x025f1a thru 0x02751b (0x1602 bytes) 000392 1789499094.297105 punch from 0x007150 to 0x00a7f9 (0x36aa bytes) 000393 1789499094.301211 write 0x0184fc thru 0x024cf4 (0xc7f9 bytes) 000394 1789499094.307976 read 0x00f441 thru 0x01ee97 (0xfa57 bytes) 000395 1789499094.308582 mapread 0x0238b3 thru 0x02dc4f (0xa39d bytes) 000396 1789499094.309387 mapread 0x00f5e5 thru 0x0106dd (0x10f9 bytes) 000397 1789499094.312685 trunc from 0x00be3a to 0x01419f (0x8366 bytes) 000398 1789499094.316791 fallocate 0x022f11 thru 0x02da76 (0xab66 bytes) 000399 1789499094.320671 fallocate 0x02d3ba thru 0x03a256 (0xce9d bytes) 000400 1789499094.323979 punch from 0x031f05 to 0x03630c (0x4408 bytes) 000401 1789499094.327369 mapread 0x02022e thru 0x02555a (0x532d bytes) 000402 1789499094.328306 punch from 0x01c465 to 0x02ba42 (0xf5de bytes) 000403 1789499094.338680 mapwrite 0x03860d thru 0x039474 (0xe68 bytes) 000404 1789499094.381546 fallocate 0x000a0f thru 0x0048a6 (0x3e98 bytes) 000405 1789499094.388717 mapread 0x013898 thru 0x01d9fa (0xa163 bytes) 000406 1789499094.408233 mapread 0x013b65 thru 0x01ea83 (0xaf1f bytes) 000407 1789499094.412180 MIRROR_RESYNC 11 mirrors 000408 1789499096.533516 trunc from 0x02b985 to 0x02c7f5 (0xe71 bytes) 000409 1789499096.555684 fallocate 0x02de7c thru 0x03b30b (0xd490 bytes) 000410 1789499096.561212 trunc from 0x007a05 to 0x00f21f (0x781b bytes) 000411 1789499096.565405 read 0x005caa thru 0x00f8c8 (0x9c1f bytes) 000412 1789499096.578773 trunc from 0x00af3b to 0x00b210 (0x2d6 bytes) 000413 1789499096.583038 mapwrite 0x02a605 thru 0x032257 (0x7c53 bytes) 000414 1789499096.614704 fallocate 0x00fb28 thru 0x01c8f1 (0xcdca bytes) 000415 1789499096.619313 trunc from 0x0283c5 to 0x037d29 (0xf965 bytes) 000416 1789499096.623457 trunc from 0x039475 to 0x037ba1 (0x18d3 bytes) 000417 1789499096.645660 punch from 0x0271f6 to 0x0292ed (0x20f8 bytes) 000419 1789499096.653465 mapread 0x00e802 thru 0x01864f (0x9e4e bytes) 000420 1789499096.664390 mapread 0x003fac thru 0x009994 (0x59e9 bytes) 000422 1789499096.671774 trunc from 0x014ff6 to 0x01a0cc (0x50d7 bytes) 000423 1789499096.675865 mapread 0x0151b4 thru 0x0203ff (0xb24c bytes) 000424 1789499096.690547 mapread 0x00badc thru 0x0169bf (0xaee4 bytes) 000425 1789499096.691470 trunc from 0x02744c to 0x027677 (0x22c bytes) 000426 1789499096.697081 write 0x00141e thru 0x007ba4 (0x6787 bytes) 000427 1789499096.698447 punch from 0x00be49 to 0x00efa1 (0x3159 bytes) 000428 1789499096.701748 read 0x00b7a8 thru 0x00d153 (0x19ac bytes) 000429 1789499096.709710 trunc from 0x004003 to 0x00acbc (0x6cba bytes) 000430 1789499096.726974 trunc from 0x037ba2 to 0x011893 (0x2630e bytes) 000431 1789499096.742848 fallocate 0x01697a thru 0x0209ed (0xa074 bytes) 000432 1789499096.746389 punch from 0x004689 to 0x00891e (0x4296 bytes) 000433 1789499096.756653 MIRROR_RESYNC 11 mirrors 000434 1789499098.057554 write 0x021008 thru 0x02de7c (0xce75 bytes) 000435 1789499098.092600 write 0x02b6d2 thru 0x039949 (0xe278 bytes) 000436 1789499098.095542 mapwrite 0x011648 thru 0x0117b6 (0x16f bytes) 000438 1789499098.131986 write 0x002333 thru 0x00705c (0x4d2a bytes) 000439 1789499098.146412 punch from 0x0165ea to 0x021468 (0xae7f bytes) 000440 1789499098.158549 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3994a): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3994a): Mirror 5: 0x22897cc5 (stale) Mirror 7: 0x22897cc5 (stale) Mirror 9: 0x22897cc5 (stale) Mirror 10: 0x22897cc5 (stale) Mirror 11: 0x22897cc5 (stale) Mirror 13: 0x22897cc5 (stale) Mirror 15: 0x22897cc5 (stale) Mirror 16: 0x22897cc5 (stale) Mirror 19: 0x22897cc5 (stale) Mirror 20: 0x22897cc5 (stale) CRC-32 checksum value for chunk [0, 0x3994a): Mirror 1: 0x90c596d4 Mirror 21: 0x90c596d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3994a: skipped 000441 1789499098.857983 trunc from 0x00ce94 to 0x017dd4 (0xaf41 bytes) 000442 1789499098.891016 mapread 0x027aee thru 0x02f3b3 (0x78c6 bytes) 000444 1789499098.905143 write 0x032a77 thru 0x039d2e (0x72b8 bytes) 000445 1789499098.913854 trunc from 0x039d2f to 0x027022 (0x12d0c bytes) 000446 1789499098.945470 mapwrite 0x028ecb thru 0x034cdf (0xbe15 bytes) 000447 1789499098.994472 write 0x025275 thru 0x02f38f (0xa11b bytes) 000448 1789499098.995738 mapread 0x019a5e thru 0x02384e (0x9df1 bytes) 000449 1789499099.003308 read 0x0140d0 thru 0x01cee4 (0x8e15 bytes) 000451 1789499099.006317 trunc from 0x034ce0 to 0x0039e3 (0x312fc bytes) 000452 1789499099.021485 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e4): Mirror 5: 0xd7c37312 (stale) Mirror 7: 0xd7c37312 (stale) Mirror 9: 0xd7c37312 (stale) Mirror 11: 0xd7c37312 (stale) Mirror 13: 0xd7c37312 (stale) Mirror 15: 0xd7c37312 (stale) Mirror 16: 0xd7c37312 (stale) Mirror 19: 0xd7c37312 (stale) Mirror 20: 0xd7c37312 (stale) Mirror 21: 0xbe16c3b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e4): Mirror 5: 0xd7c37312 (stale) Mirror 7: 0xd7c37312 (stale) Mirror 9: 0xd7c37312 (stale) Mirror 11: 0xd7c37312 (stale) Mirror 13: 0xd7c37312 (stale) Mirror 15: 0xd7c37312 (stale) Mirror 16: 0xd7c37312 (stale) Mirror 19: 0xd7c37312 (stale) Mirror 20: 0xd7c37312 (stale) Mirror 21: 0xbe16c3b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e4: skipped 000453 1789499099.299690 fallocate 0x025757 thru 0x029167 (0x3a11 bytes) 000455 1789499099.309615 trunc from 0x0039e4 to 0x00f9af (0xbfcc bytes) 000456 1789499099.330153 write 0x00e53a thru 0x01706e (0x8b35 bytes) 000457 1789499099.337333 mapwrite 0x01aeb6 thru 0x024c34 (0x9d7f bytes) 000458 1789499099.383182 mapread 0x024895 thru 0x024c34 (0x3a0 bytes) 000459 1789499099.384015 mapread 0x013037 thru 0x015495 (0x245f bytes) 000461 1789499099.385428 read 0x0010a3 thru 0x002199 (0x10f7 bytes) 000462 1789499099.392091 punch from 0x0237af to 0x024c34 (0x1486 bytes) 000464 1789499099.395930 read 0x02265b thru 0x024c34 (0x25da bytes) 000465 1789499099.403051 write 0x039c48 thru 0x03ffff (0x63b8 bytes) 000466 1789499099.404493 trunc from 0x01ebd2 to 0x021e08 (0x3237 bytes) 000467 1789499099.409505 trunc from 0x040000 to 0x02abda (0x15425 bytes) 000468 1789499099.419523 write 0x008ef5 thru 0x01051d (0x7629 bytes) 000469 1789499099.421329 trunc from 0x01aece to 0x027484 (0xc5b7 bytes) 000470 1789499099.426853 write 0x03bb49 thru 0x03ffff (0x44b7 bytes) 000471 1789499099.428088 punch from 0x0124cd to 0x01a989 (0x84bd bytes) 000472 1789499099.431799 fallocate 0x03c596 thru 0x03ffff (0x3a6a bytes) 000473 1789499099.435380 mapwrite 0x036839 thru 0x03f54e (0x8d16 bytes) 000474 1789499099.462822 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x270970d8 (stale) Mirror 7: 0x270970d8 (stale) Mirror 9: 0x270970d8 (stale) Mirror 11: 0x270970d8 (stale) Mirror 13: 0x270970d8 (stale) Mirror 15: 0x270970d8 (stale) Mirror 16: 0x270970d8 (stale) Mirror 19: 0x270970d8 (stale) Mirror 20: 0x270970d8 (stale) Mirror 21: 0x72080cd4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xffd7891a Mirror 22: 0xffd7891a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1789499100.070940 punch from 0x02dad9 to 0x035944 (0x7e6c bytes) 000476 1789499100.101605 fallocate 0x00982e thru 0x00c313 (0x2ae6 bytes) 000477 1789499100.107822 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x270970d8 (stale) Mirror 7: 0x270970d8 (stale) Mirror 9: 0x270970d8 (stale) Mirror 11: 0x270970d8 (stale) Mirror 13: 0x270970d8 (stale) Mirror 15: 0x270970d8 (stale) Mirror 16: 0x270970d8 (stale) Mirror 19: 0x270970d8 (stale) Mirror 20: 0x270970d8 (stale) Mirror 21: 0x72080cd4 (stale) Mirror 22: 0xffd7891a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xffd7891a Mirror 23: 0xffd7891a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000478 1789499100.727480 MIRROR_RESYNC 13 mirrors 000480 1789499102.719633 mapread 0x0145b6 thru 0x018459 (0x3ea4 bytes) 000481 1789499102.736174 fallocate 0x00b817 thru 0x01ac8f (0xf479 bytes) 000482 1789499102.770162 trunc from 0x033a44 to 0x03ffff (0xc5bc bytes) 000483 1789499102.773438 trunc from 0x040000 to 0x02bac8 (0x14537 bytes) 000485 1789499102.788751 trunc from 0x02bac9 to 0x017cc2 (0x13e06 bytes) 000486 1789499102.799752 mapread 0x00fadb thru 0x017cc2 (0x81e8 bytes) 000487 1789499102.807790 fallocate 0x004f95 thru 0x009849 (0x48b5 bytes) 000488 1789499102.811419 punch from 0x00573b to 0x00e8ae (0x9174 bytes) 000489 1789499102.816021 mapwrite 0x036370 thru 0x03ff70 (0x9c01 bytes) 000490 1789499102.861887 write 0x00c43a thru 0x01a320 (0xdee7 bytes) 000491 1789499102.871598 trunc from 0x03ff71 to 0x039d5c (0x6214 bytes) 000492 1789499102.882406 trunc from 0x039d5d to 0x01fde6 (0x19f76 bytes) 000493 1789499102.902836 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1fde7): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fde7): Mirror 5: 0x49fd359f (stale) Mirror 7: 0x49fd359f (stale) Mirror 9: 0x49fd359f (stale) Mirror 11: 0x49fd359f (stale) Mirror 13: 0x49fd359f (stale) Mirror 15: 0x49fd359f (stale) Mirror 16: 0x49fd359f (stale) Mirror 19: 0x49fd359f (stale) Mirror 20: 0x49fd359f (stale) Mirror 21: 0x49fd359f (stale) Mirror 22: 0x49fd359f (stale) Mirror 23: 0x49fd359f (stale) CRC-32 checksum value for chunk [0, 0x1fde7): Mirror 1: 0x976b01f6 Mirror 24: 0x976b01f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fde7: skipped 000494 1789499103.610117 punch from 0x015a5f to 0x01f06a (0x960c bytes) 000495 1789499103.634464 fallocate 0x01acd7 thru 0x022cd7 (0x8001 bytes) 000496 1789499103.638497 mapread 0x012009 thru 0x01fde6 (0xddde bytes) 000497 1789499103.661013 mapwrite 0x01e7fe thru 0x02403e (0x5841 bytes) 000498 1789499103.693292 mapwrite 0x0133f7 thru 0x01dcc5 (0xa8cf bytes) 000499 1789499103.715794 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2403f): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2403f): Mirror 5: 0xe41ce56 (stale) Mirror 7: 0xe41ce56 (stale) Mirror 9: 0xe41ce56 (stale) Mirror 11: 0xe41ce56 (stale) Mirror 13: 0xe41ce56 (stale) Mirror 15: 0xe41ce56 (stale) Mirror 16: 0xe41ce56 (stale) Mirror 19: 0xe41ce56 (stale) Mirror 20: 0xe41ce56 (stale) Mirror 21: 0xe41ce56 (stale) Mirror 22: 0xe41ce56 (stale) Mirror 23: 0xe41ce56 (stale) Mirror 24: 0xf0d4a60d (stale) CRC-32 checksum value for chunk [0, 0x2403f): Mirror 1: 0xbf87ba3a Mirror 25: 0xbf87ba3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2403f: skipped 000500 1789499104.256526 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2403f): Mirror 5: 0xe41ce56 (stale) Mirror 7: 0xe41ce56 (stale) Mirror 9: 0xe41ce56 (stale) Mirror 11: 0xe41ce56 (stale) Mirror 13: 0xe41ce56 (stale) Mirror 15: 0xe41ce56 (stale) Mirror 16: 0xe41ce56 (stale) Mirror 19: 0xe41ce56 (stale) Mirror 20: 0xe41ce56 (stale) Mirror 21: 0xe41ce56 (stale) Mirror 22: 0xe41ce56 (stale) Mirror 23: 0xe41ce56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2403f): Mirror 5: 0xe41ce56 (stale) Mirror 7: 0xe41ce56 (stale) Mirror 9: 0xe41ce56 (stale) Mirror 11: 0xe41ce56 (stale) Mirror 13: 0xe41ce56 (stale) Mirror 15: 0xe41ce56 (stale) Mirror 16: 0xe41ce56 (stale) Mirror 19: 0xe41ce56 (stale) Mirror 20: 0xe41ce56 (stale) Mirror 21: 0xe41ce56 (stale) Mirror 22: 0xe41ce56 (stale) Mirror 23: 0xe41ce56 (stale) CRC-32 checksum value for chunk [0, 0x2403f): Mirror 1: 0xbf87ba3a Mirror 25: 0xbf87ba3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2403f: skipped 000502 1789499104.966380 mapread 0x00531f thru 0x0083a4 (0x3086 bytes) 000503 1789499104.992542 mapwrite 0x00ad9a thru 0x00d664 (0x28cb bytes) 000504 1789499105.052840 trunc from 0x0012ff to 0x005fac (0x4cae bytes) 000505 1789499105.059761 mapread 0x0153cb thru 0x023588 (0xe1be bytes) 000506 1789499105.077717 write 0x025e90 thru 0x02bfd5 (0x6146 bytes) 000507 1789499105.088288 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfd6): Mirror 5: 0x8c9e2b1e (stale) Mirror 7: 0x8c9e2b1e (stale) Mirror 9: 0x8c9e2b1e (stale) Mirror 11: 0x8c9e2b1e (stale) Mirror 13: 0x8c9e2b1e (stale) Mirror 16: 0x8c9e2b1e (stale) Mirror 19: 0x8c9e2b1e (stale) Mirror 20: 0x8c9e2b1e (stale) Mirror 21: 0x8c9e2b1e (stale) Mirror 22: 0x8c9e2b1e (stale) Mirror 23: 0x8c9e2b1e (stale) Mirror 25: 0xf2c26a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfd6): Mirror 5: 0x8c9e2b1e (stale) Mirror 7: 0x8c9e2b1e (stale) Mirror 9: 0x8c9e2b1e (stale) Mirror 11: 0x8c9e2b1e (stale) Mirror 13: 0x8c9e2b1e (stale) Mirror 16: 0x8c9e2b1e (stale) Mirror 19: 0x8c9e2b1e (stale) Mirror 20: 0x8c9e2b1e (stale) Mirror 21: 0x8c9e2b1e (stale) Mirror 22: 0x8c9e2b1e (stale) Mirror 23: 0x8c9e2b1e (stale) Mirror 25: 0xf2c26a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfd6: skipped 000508 1789499105.626172 mapread 0x00631f thru 0x00d100 (0x6de2 bytes) 000509 1789499105.647918 trunc from 0x02bfd6 to 0x0178eb (0x146ea bytes) 000510 1789499105.695395 trunc from 0x0178ec to 0x01d125 (0x583a bytes) 000513 1789499105.717080 MIRROR_RESYNC 13 mirrors 000514 1789499107.497435 fallocate 0x000c1b thru 0x0012ce (0x6b4 bytes) 000515 1789499107.538464 punch from 0x017c6c to 0x01a6d2 (0x2a67 bytes) 000516 1789499107.545204 write 0x033a4f thru 0x03f82a (0xbddc bytes) 000517 1789499107.546639 mapwrite 0x00f13d thru 0x017198 (0x805c bytes) 000518 1789499107.586067 punch from 0x02ccf0 to 0x033a91 (0x6da2 bytes) 000519 1789499107.594066 read 0x003ed5 thru 0x0051a3 (0x12cf bytes) 000520 1789499107.601370 fallocate 0x037291 thru 0x03b806 (0x4576 bytes) 000521 1789499107.605467 punch from 0x00f7da to 0x019947 (0xa16e bytes) 000522 1789499107.609683 trunc from 0x0307e9 to 0x0318e8 (0x1100 bytes) 000523 1789499107.615071 mapwrite 0x024b59 thru 0x02dc58 (0x9100 bytes) 000524 1789499107.657508 write 0x02c0e4 thru 0x03bbbf (0xfadc bytes) 000526 1789499107.659398 write 0x017ff0 thru 0x01ab93 (0x2ba4 bytes) 000527 1789499107.672557 mapread 0x02c2b5 thru 0x02ffcd (0x3d19 bytes) 000528 1789499107.674194 trunc from 0x03e63f to 0x03f82a (0x11ec bytes) 000529 1789499107.678395 fallocate 0x015ad5 thru 0x024265 (0xe791 bytes) 000530 1789499107.683841 trunc from 0x03f82b to 0x0223ba (0x1d470 bytes) 000531 1789499107.700178 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x223bb): Mirror 5: 0x194bbd47 (stale) Mirror 7: 0x194bbd47 (stale) Mirror 9: 0x194bbd47 (stale) Mirror 11: 0x194bbd47 (stale) Mirror 13: 0x194bbd47 (stale) Mirror 16: 0x194bbd47 (stale) Mirror 20: 0x194bbd47 (stale) Mirror 21: 0x194bbd47 (stale) Mirror 22: 0x194bbd47 (stale) Mirror 23: 0x194bbd47 (stale) Mirror 25: 0x194bbd47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x223bb): Mirror 5: 0x194bbd47 (stale) Mirror 7: 0x194bbd47 (stale) Mirror 9: 0x194bbd47 (stale) Mirror 11: 0x194bbd47 (stale) Mirror 13: 0x194bbd47 (stale) Mirror 16: 0x194bbd47 (stale) Mirror 20: 0x194bbd47 (stale) Mirror 21: 0x194bbd47 (stale) Mirror 22: 0x194bbd47 (stale) Mirror 23: 0x194bbd47 (stale) Mirror 25: 0x194bbd47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x223bb: skipped 000532 1789499108.197241 trunc from 0x0223bb to 0x01cca2 (0x5718 bytes) 000533 1789499108.220164 mapwrite 0x01e99d thru 0x021afe (0x3162 bytes) 000534 1789499108.275449 trunc from 0x01910c to 0x01bec5 (0x2dba bytes) 000535 1789499108.285020 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x21aff): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21aff): Mirror 5: 0x1bead248 (stale) Mirror 7: 0x1bead248 (stale) Mirror 9: 0x1bead248 (stale) Mirror 11: 0x1bead248 (stale) Mirror 13: 0x1bead248 (stale) Mirror 16: 0x1bead248 (stale) Mirror 20: 0x1bead248 (stale) Mirror 21: 0x1bead248 (stale) Mirror 22: 0x1bead248 (stale) Mirror 23: 0x1bead248 (stale) Mirror 25: 0x1bead248 (stale) CRC-32 checksum value for chunk [0, 0x21aff): Mirror 1: 0xda7068d4 Mirror 26: 0xda7068d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21aff: skipped 000536 1789499108.957511 punch from 0x00a7ad to 0x014579 (0x9dcd bytes) 000537 1789499108.986401 punch from 0x0202e4 to 0x021afe (0x181b bytes) 000538 1789499108.993998 mapwrite 0x010c4a thru 0x0156ec (0x4aa3 bytes) 000539 1789499109.024632 read 0x005fad thru 0x00ad6b (0x4dbf bytes) 000540 1789499109.031674 mapread 0x00f2d7 thru 0x00f85e (0x588 bytes) 000542 1789499109.032666 mapwrite 0x036df4 thru 0x03ffff (0x920c bytes) 000543 1789499109.089657 write 0x0006af thru 0x007ee9 (0x783b bytes) 000544 1789499109.097163 mapwrite 0x031e9e thru 0x03616f (0x42d2 bytes) 000545 1789499109.126400 punch from 0x0310cb to 0x039c72 (0x8ba8 bytes) 000546 1789499109.136219 trunc from 0x01d082 to 0x028d74 (0xbcf3 bytes) 000547 1789499109.142824 mapwrite 0x005d5f thru 0x0110c1 (0xb363 bytes) 000548 1789499109.160063 mapread 0x002fce thru 0x00a70b (0x773e bytes) 000549 1789499109.161879 write 0x013651 thru 0x02354e (0xfefe bytes) 000550 1789499109.174911 MIRROR_RESYNC 13 mirrors 000551 1789499111.100114 fallocate 0x01248c thru 0x01e20c (0xbd81 bytes) 000553 1789499111.148538 mapwrite 0x01e1e2 thru 0x02a185 (0xbfa4 bytes) 000554 1789499111.195981 read 0x03b4e2 thru 0x03ffff (0x4b1e bytes) 000555 1789499111.208180 mapread 0x004640 thru 0x00b4be (0x6e7f bytes) 000556 1789499111.224482 mapread 0x00d6f8 thru 0x01411f (0x6a28 bytes) 000557 1789499111.251334 mapwrite 0x02acde thru 0x037e70 (0xd193 bytes) 000559 1789499111.278929 write 0x000e38 thru 0x007606 (0x67cf bytes) 000560 1789499111.288920 mapwrite 0x00a765 thru 0x012139 (0x79d5 bytes) 000561 1789499111.312291 mapread 0x0261d6 thru 0x02ceca (0x6cf5 bytes) 000562 1789499111.314030 mapread 0x01d40f thru 0x02b003 (0xdbf5 bytes) 000563 1789499111.315061 trunc from 0x0357e5 to 0x03a2b0 (0x4acc bytes) 000564 1789499111.324570 trunc from 0x02e66c to 0x039f4e (0xb8e3 bytes) 000565 1789499111.337589 punch from 0x007fdf to 0x00c625 (0x4647 bytes) 000566 1789499111.344340 trunc from 0x040000 to 0x01666a (0x29995 bytes) 000567 1789499111.367191 fallocate 0x032a0d thru 0x033954 (0xf48 bytes) 000568 1789499111.372890 trunc from 0x02156e to 0x0280a3 (0x6b36 bytes) 000569 1789499111.376873 mapwrite 0x02bc9d thru 0x031ea3 (0x6207 bytes) 000570 1789499111.400832 mapwrite 0x01d465 thru 0x0216d8 (0x4274 bytes) 000571 1789499111.424204 mapwrite 0x01415f thru 0x01674e (0x25f0 bytes) 000572 1789499111.435463 mapwrite 0x00c10e thru 0x00ebe4 (0x2ad7 bytes) 000573 1789499111.454452 trunc from 0x033955 to 0x0194bc (0x1a498 bytes) 000574 1789499111.479184 fallocate 0x033d0f thru 0x03e7b6 (0xaaa8 bytes) 000575 1789499111.485046 fallocate 0x036753 thru 0x0392e2 (0x2b90 bytes) 000576 1789499111.489716 read 0x013675 thru 0x0194bc (0x5e48 bytes) 000578 1789499111.490982 mapwrite 0x005c18 thru 0x013224 (0xd60d bytes) 000579 1789499111.505410 punch from 0x01004e to 0x01163c (0x15ef bytes) 000580 1789499111.508756 fallocate 0x00b500 thru 0x00c358 (0xe59 bytes) 000581 1789499111.512034 punch from 0x00f0f9 to 0x0121d5 (0x30dd bytes) 000583 1789499111.516740 mapread 0x00c755 thru 0x011ad8 (0x5384 bytes) 000584 1789499111.524445 punch from 0x00d58b to 0x013c49 (0x66bf bytes) 000585 1789499111.529240 fallocate 0x01ce31 thru 0x02bb23 (0xecf3 bytes) 000586 1789499111.535090 read 0x016985 thru 0x024e97 (0xe513 bytes) 000587 1789499111.543229 trunc from 0x02bb24 to 0x01e28f (0xd894 bytes) 000588 1789499111.553994 mapread 0x00be09 thru 0x016475 (0xa66d bytes) 000589 1789499111.555065 punch from 0x01da95 to 0x01e28f (0x7fb bytes) 000590 1789499111.563290 punch from 0x007293 to 0x00a2b6 (0x3024 bytes) 000591 1789499111.571490 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e290): Mirror 5: 0x395e2483 (stale) Mirror 7: 0x395e2483 (stale) Mirror 9: 0x395e2483 (stale) Mirror 11: 0x395e2483 (stale) Mirror 13: 0x395e2483 (stale) Mirror 20: 0x395e2483 (stale) Mirror 21: 0x395e2483 (stale) Mirror 22: 0x395e2483 (stale) Mirror 23: 0x395e2483 (stale) Mirror 25: 0x395e2483 (stale) Mirror 26: 0x395e2483 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e290): Mirror 5: 0x395e2483 (stale) Mirror 7: 0x395e2483 (stale) Mirror 9: 0x395e2483 (stale) Mirror 11: 0x395e2483 (stale) Mirror 13: 0x395e2483 (stale) Mirror 20: 0x395e2483 (stale) Mirror 21: 0x395e2483 (stale) Mirror 22: 0x395e2483 (stale) Mirror 23: 0x395e2483 (stale) Mirror 25: 0x395e2483 (stale) Mirror 26: 0x395e2483 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e290: skipped 000592 1789499111.958808 trunc from 0x01e290 to 0x0117db (0xcab4 bytes) 000593 1789499111.974159 fallocate 0x03fcd2 thru 0x03ffff (0x32e bytes) 000594 1789499111.978524 fallocate 0x02b11c thru 0x02d1bf (0x20a4 bytes) 000595 1789499111.982852 mapread 0x009b61 thru 0x0117db (0x7c7b bytes) 000596 1789499112.005216 fallocate 0x000dfc thru 0x0102e0 (0xf4e5 bytes) 000597 1789499112.009731 fallocate 0x014f4a thru 0x01acf5 (0x5dac bytes) 000598 1789499112.014049 read 0x015bb3 thru 0x01acf5 (0x5143 bytes) 000599 1789499112.021925 read 0x01573e thru 0x015fa4 (0x867 bytes) 000600 1789499112.022755 mapwrite 0x0000fe thru 0x00d9ed (0xd8f0 bytes) 000601 1789499112.063033 punch from 0x00b6bb to 0x0192ef (0xdc35 bytes) 000602 1789499112.069049 mapwrite 0x03825f thru 0x03ffff (0x7da1 bytes) 000604 1789499112.128319 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x935114a7 (stale) Mirror 7: 0x935114a7 (stale) Mirror 9: 0x935114a7 (stale) Mirror 11: 0x935114a7 (stale) Mirror 13: 0x935114a7 (stale) Mirror 20: 0x935114a7 (stale) Mirror 21: 0x935114a7 (stale) Mirror 22: 0x935114a7 (stale) Mirror 23: 0x935114a7 (stale) Mirror 25: 0x935114a7 (stale) Mirror 26: 0x935114a7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9fd9671a Mirror 27: 0x9fd9671a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1789499112.744063 trunc from 0x040000 to 0x03665f (0x99a0 bytes) 000607 1789499112.782604 punch from 0x02c601 to 0x02d491 (0xe91 bytes) 000608 1789499112.788017 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x36660): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36660): Mirror 5: 0x1ca5fde6 (stale) Mirror 7: 0x1ca5fde6 (stale) Mirror 9: 0x1ca5fde6 (stale) Mirror 11: 0x1ca5fde6 (stale) Mirror 13: 0x1ca5fde6 (stale) Mirror 20: 0x1ca5fde6 (stale) Mirror 21: 0x1ca5fde6 (stale) Mirror 22: 0x1ca5fde6 (stale) Mirror 23: 0x1ca5fde6 (stale) Mirror 25: 0x1ca5fde6 (stale) Mirror 26: 0x1ca5fde6 (stale) Mirror 27: 0x605a6c57 (stale) CRC-32 checksum value for chunk [0, 0x36660): Mirror 1: 0x605a6c57 Mirror 28: 0x605a6c57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36660: skipped 000610 1789499113.374614 punch from 0x0338fe to 0x03665f (0x2d62 bytes) 000611 1789499113.413110 fallocate 0x015225 thru 0x021618 (0xc3f4 bytes) 000612 1789499113.417366 fallocate 0x01727e thru 0x022499 (0xb21c bytes) 000613 1789499113.423932 MIRROR_RESYNC 14 mirrors 000615 1789499114.668501 mapwrite 0x00a6ef thru 0x01299a (0x82ac bytes) 000616 1789499114.730979 trunc from 0x036660 to 0x01b30a (0x1b355 bytes) 000617 1789499114.744299 punch from 0x0186d7 to 0x01b30a (0x2c34 bytes) 000619 1789499114.755671 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 5: 0x2e2e428d (stale) Mirror 7: 0x2e2e428d (stale) Mirror 9: 0x2e2e428d (stale) Mirror 11: 0x2e2e428d (stale) Mirror 13: 0x2e2e428d (stale) Mirror 20: 0x2e2e428d (stale) Mirror 21: 0x2e2e428d (stale) Mirror 22: 0x2e2e428d (stale) Mirror 23: 0x2e2e428d (stale) Mirror 25: 0x2e2e428d (stale) Mirror 26: 0x2e2e428d (stale) Mirror 27: 0x2e2e428d (stale) Mirror 28: 0x2e2e428d (stale) CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 1: 0xf57fccaf Mirror 29: 0xf57fccaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b30b: skipped 000620 1789499115.448006 trunc from 0x0150f4 to 0x01b30a (0x6217 bytes) 000621 1789499115.491453 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 5: 0x2e2e428d (stale) Mirror 7: 0x2e2e428d (stale) Mirror 9: 0x2e2e428d (stale) Mirror 11: 0x2e2e428d (stale) Mirror 13: 0x2e2e428d (stale) Mirror 20: 0x2e2e428d (stale) Mirror 21: 0x2e2e428d (stale) Mirror 22: 0x2e2e428d (stale) Mirror 23: 0x2e2e428d (stale) Mirror 25: 0x2e2e428d (stale) Mirror 26: 0x2e2e428d (stale) Mirror 27: 0x2e2e428d (stale) Mirror 28: 0x2e2e428d (stale) Mirror 29: 0xf57fccaf (stale) CRC-32 checksum value for chunk [0, 0x1b30b): Mirror 1: 0xf57fccaf Mirror 30: 0xf57fccaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b30b: skipped 000622 1789499116.298799 trunc from 0x0147b6 to 0x01b30a (0x6b55 bytes) 000623 1789499116.330385 fallocate 0x011d81 thru 0x0150fd (0x337d bytes) 000624 1789499116.338035 punch from 0x012fe4 to 0x01b30a (0x8327 bytes) 000625 1789499116.347595 mapwrite 0x016dda thru 0x020188 (0x93af bytes) 000626 1789499116.400970 read 0x005095 thru 0x006fdc (0x1f48 bytes) 000627 1789499116.405471 read 0x018e07 thru 0x01bae4 (0x2cde bytes) 000628 1789499116.406241 mapread 0x001eb2 thru 0x00537c (0x34cb bytes) 000629 1789499116.413386 read 0x00bfa3 thru 0x014b19 (0x8b77 bytes) 000630 1789499116.414021 mapread 0x00d515 thru 0x013449 (0x5f35 bytes) 000631 1789499116.414781 read 0x004502 thru 0x0141be (0xfcbd bytes) 000632 1789499116.415387 mapwrite 0x035b8c thru 0x03cc38 (0x70ad bytes) 000633 1789499116.452947 punch from 0x0027e2 to 0x003931 (0x1150 bytes) 000634 1789499116.458785 mapread 0x00dd9c thru 0x01765c (0x98c1 bytes) 000635 1789499116.460489 mapread 0x002974 thru 0x003d52 (0x13df bytes) 000636 1789499116.468498 fallocate 0x02cf87 thru 0x02f2aa (0x2324 bytes) 000637 1789499116.473328 mapread 0x03caba thru 0x03cc38 (0x17f bytes) 000638 1789499116.475094 mapwrite 0x0210fa thru 0x0233c6 (0x22cd bytes) 000639 1789499116.485162 trunc from 0x03cc39 to 0x012b8d (0x2a0ab bytes) 000640 1789499116.505227 fallocate 0x03ea2c thru 0x03ffff (0x15d4 bytes) 000641 1789499116.511674 mapread 0x032fb5 thru 0x035324 (0x2370 bytes) 000642 1789499116.543517 punch from 0x00349c to 0x00b741 (0x82a6 bytes) 000643 1789499116.551219 read 0x016faf thru 0x01c6b8 (0x570a bytes) 000644 1789499116.564528 mapwrite 0x02357e thru 0x02ff70 (0xc9f3 bytes) 000645 1789499116.594324 punch from 0x015cb0 to 0x02301a (0xd36b bytes) 000647 1789499116.601199 mapread 0x029a06 thru 0x0348b8 (0xaeb3 bytes) 000648 1789499116.602156 mapwrite 0x01158a thru 0x01456d (0x2fe4 bytes) 000649 1789499116.617048 mapwrite 0x019975 thru 0x01d52a (0x3bb6 bytes) 000650 1789499116.642920 write 0x02290f thru 0x029e88 (0x757a bytes) 000651 1789499116.652869 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x539b00d6 (stale) Mirror 7: 0x539b00d6 (stale) Mirror 9: 0x539b00d6 (stale) Mirror 11: 0x539b00d6 (stale) Mirror 13: 0x539b00d6 (stale) Mirror 20: 0x539b00d6 (stale) Mirror 21: 0x539b00d6 (stale) Mirror 22: 0x539b00d6 (stale) Mirror 23: 0x539b00d6 (stale) Mirror 26: 0x539b00d6 (stale) Mirror 27: 0x539b00d6 (stale) Mirror 28: 0x539b00d6 (stale) Mirror 29: 0xf808ab43 (stale) Mirror 30: 0xf808ab43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x539b00d6 (stale) Mirror 7: 0x539b00d6 (stale) Mirror 9: 0x539b00d6 (stale) Mirror 11: 0x539b00d6 (stale) Mirror 13: 0x539b00d6 (stale) Mirror 20: 0x539b00d6 (stale) Mirror 21: 0x539b00d6 (stale) Mirror 22: 0x539b00d6 (stale) Mirror 23: 0x539b00d6 (stale) Mirror 26: 0x539b00d6 (stale) Mirror 27: 0x539b00d6 (stale) Mirror 28: 0x539b00d6 (stale) Mirror 29: 0xf808ab43 (stale) Mirror 30: 0xf808ab43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000652 1789499117.246573 mapread 0x00b99a thru 0x019434 (0xda9b bytes) 000653 1789499117.263884 read 0x0089c9 thru 0x00ba01 (0x3039 bytes) 000655 1789499117.264945 write 0x00e772 thru 0x019808 (0xb097 bytes) 000656 1789499117.269578 mapwrite 0x004887 thru 0x00cd6a (0x84e4 bytes) 000657 1789499117.290905 mapread 0x004ba4 thru 0x009246 (0x46a3 bytes) 000658 1789499117.292103 mapread 0x021cb6 thru 0x02d470 (0xb7bb bytes) 000659 1789499117.308649 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x539b00d6 (stale) Mirror 7: 0x539b00d6 (stale) Mirror 9: 0x539b00d6 (stale) Mirror 11: 0x539b00d6 (stale) Mirror 13: 0x539b00d6 (stale) Mirror 20: 0x539b00d6 (stale) Mirror 21: 0x539b00d6 (stale) Mirror 22: 0x539b00d6 (stale) Mirror 23: 0x539b00d6 (stale) Mirror 26: 0x539b00d6 (stale) Mirror 28: 0x539b00d6 (stale) Mirror 29: 0xf808ab43 (stale) Mirror 30: 0xf808ab43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x539b00d6 (stale) Mirror 7: 0x539b00d6 (stale) Mirror 9: 0x539b00d6 (stale) Mirror 11: 0x539b00d6 (stale) Mirror 13: 0x539b00d6 (stale) Mirror 20: 0x539b00d6 (stale) Mirror 21: 0x539b00d6 (stale) Mirror 22: 0x539b00d6 (stale) Mirror 23: 0x539b00d6 (stale) Mirror 26: 0x539b00d6 (stale) Mirror 28: 0x539b00d6 (stale) Mirror 29: 0xf808ab43 (stale) Mirror 30: 0xf808ab43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1789499117.801057 mapwrite 0x006a59 thru 0x014a54 (0xdffc bytes) 000661 1789499117.862704 read 0x01a497 thru 0x01c1ac (0x1d16 bytes) 000662 1789499117.869516 trunc from 0x0180dc to 0x01b068 (0x2f8d bytes) 000663 1789499117.880666 fallocate 0x032b36 thru 0x034448 (0x1913 bytes) 000664 1789499117.887846 read 0x018814 thru 0x01fbcc (0x73b9 bytes) 000665 1789499117.898963 mapwrite 0x02bc19 thru 0x038dfe (0xd1e6 bytes) 000666 1789499117.914858 trunc from 0x035dbc to 0x03748a (0x16cf bytes) 000667 1789499117.917826 trunc from 0x01cfb6 to 0x01f622 (0x266d bytes) 000668 1789499117.921685 read 0x00a610 thru 0x012060 (0x7a51 bytes) 000669 1789499117.922515 mapread 0x01998f thru 0x028d9c (0xf40e bytes) 000670 1789499117.930329 fallocate 0x032dee thru 0x03b0be (0x82d1 bytes) 000671 1789499117.936042 mapread 0x00cd34 thru 0x01a4c5 (0xd792 bytes) 000672 1789499117.941031 write 0x03117e thru 0x037c65 (0x6ae8 bytes) 000673 1789499117.946368 fallocate 0x032f3a thru 0x0348dd (0x19a4 bytes) 000674 1789499117.950614 fallocate 0x032369 thru 0x03366d (0x1305 bytes) 000676 1789499117.958755 mapwrite 0x0293d9 thru 0x030051 (0x6c79 bytes) 000677 1789499117.985050 mapwrite 0x015173 thru 0x0214c1 (0xc34f bytes) 000678 1789499118.008610 mapwrite 0x01080e thru 0x02071f (0xff12 bytes) 000679 1789499118.027133 mapwrite 0x00cfb2 thru 0x01a37d (0xd3cc bytes) 000680 1789499118.040855 punch from 0x015f6f to 0x01f93e (0x99d0 bytes) 000681 1789499118.046585 mapread 0x02ff85 thru 0x035b76 (0x5bf2 bytes) 000682 1789499118.047473 fallocate 0x008ced thru 0x012524 (0x9838 bytes) 000683 1789499118.056210 punch from 0x03dda7 to 0x03ffff (0x2259 bytes) 000684 1789499118.060236 trunc from 0x040000 to 0x01008b (0x2ff74 bytes) 000686 1789499118.072167 read 0x00952d thru 0x01008b (0x6b5f bytes) 000687 1789499118.072771 trunc from 0x01008c to 0x035b82 (0x25af7 bytes) 000688 1789499118.082832 trunc from 0x027bb4 to 0x02f42c (0x7879 bytes) 000689 1789499118.086993 punch from 0x002dfe to 0x008dda (0x5fdd bytes) 000690 1789499118.092124 punch from 0x01574d to 0x018b6e (0x3422 bytes) 000691 1789499118.095903 read 0x024111 thru 0x0273bf (0x32af bytes) 000692 1789499118.106331 mapread 0x020a40 thru 0x023c02 (0x31c3 bytes) 000693 1789499118.107091 punch from 0x0229ec to 0x0244c1 (0x1ad6 bytes) 000694 1789499118.110120 trunc from 0x023075 to 0x0280f9 (0x5085 bytes) 000695 1789499118.113807 read 0x02d5d6 thru 0x035b82 (0x85ad bytes) 000696 1789499118.114823 fallocate 0x03579c thru 0x03e186 (0x89eb bytes) 000698 1789499118.124212 read 0x02dabf thru 0x02f7be (0x1d00 bytes) 000699 1789499118.125155 read 0x00a3da thru 0x01280e (0x8435 bytes) 000700 1789499118.126077 fallocate 0x0241d7 thru 0x025394 (0x11be bytes) 000701 1789499118.131173 trunc from 0x01b759 to 0x027a08 (0xc2b0 bytes) 000702 1789499118.136587 trunc from 0x03e187 to 0x028f4c (0x1523a bytes) 000704 1789499118.153852 fallocate 0x02ac4e thru 0x039c98 (0xf04b bytes) 000705 1789499118.163024 MIRROR_RESYNC 14 mirrors 000706 1789499119.966908 fallocate 0x0232d3 thru 0x02eb1a (0xb848 bytes) 000707 1789499120.006838 trunc from 0x02e188 to 0x02f676 (0x14ef bytes) 000708 1789499120.013574 trunc from 0x039c99 to 0x0202f1 (0x199a7 bytes) 000709 1789499120.038310 read 0x0188a5 thru 0x0202f1 (0x7a4d bytes) 000710 1789499120.048369 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x202f2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202f2): Mirror 5: 0x8fb092bb (stale) Mirror 7: 0x8fb092bb (stale) Mirror 9: 0x8fb092bb (stale) Mirror 11: 0x8fb092bb (stale) Mirror 13: 0x8fb092bb (stale) Mirror 20: 0x8fb092bb (stale) Mirror 21: 0x8fb092bb (stale) Mirror 22: 0x8fb092bb (stale) Mirror 23: 0x8fb092bb (stale) Mirror 26: 0x8fb092bb (stale) Mirror 28: 0x8fb092bb (stale) Mirror 29: 0x8fb092bb (stale) Mirror 30: 0x8fb092bb (stale) CRC-32 checksum value for chunk [0, 0x202f2): Mirror 1: 0x8fb092bb Mirror 31: 0x8fb092bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202f2: skipped 000711 1789499120.862878 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x202f2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202f2): Mirror 5: 0x8fb092bb (stale) Mirror 7: 0x8fb092bb (stale) Mirror 9: 0x8fb092bb (stale) Mirror 11: 0x8fb092bb (stale) Mirror 13: 0x8fb092bb (stale) Mirror 20: 0x8fb092bb (stale) Mirror 21: 0x8fb092bb (stale) Mirror 22: 0x8fb092bb (stale) Mirror 23: 0x8fb092bb (stale) Mirror 26: 0x8fb092bb (stale) Mirror 28: 0x8fb092bb (stale) Mirror 29: 0x8fb092bb (stale) Mirror 30: 0x8fb092bb (stale) CRC-32 checksum value for chunk [0, 0x202f2): Mirror 1: 0x8fb092bb Mirror 31: 0x8fb092bb Mirror 32: 0x8fb092bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202f2: skipped 000712 1789499121.614230 mapwrite 0x031aef thru 0x0377e1 (0x5cf3 bytes) 000713 1789499121.704810 write 0x02b6c6 thru 0x03779a (0xc0d5 bytes) 000714 1789499121.710132 trunc from 0x01bca7 to 0x028734 (0xca8e bytes) 000715 1789499121.719552 write 0x0275db thru 0x02e7cc (0x71f2 bytes) 000716 1789499121.725867 punch from 0x022cbc to 0x029b3a (0x6e7f bytes) 000717 1789499121.737216 punch from 0x00a7b4 to 0x01956e (0xedbb bytes) 000719 1789499121.743911 write 0x01c8db thru 0x01d047 (0x76d bytes) 000720 1789499121.750607 trunc from 0x0377e2 to 0x025574 (0x1226d bytes) 000721 1789499121.766009 fallocate 0x00ee49 thru 0x01a2b9 (0xb471 bytes) 000722 1789499121.770133 fallocate 0x039948 thru 0x03ffff (0x66b8 bytes) 000724 1789499121.775806 write 0x03a22e thru 0x03ffff (0x5dd2 bytes) 000725 1789499121.777000 mapread 0x03cf8b thru 0x03e28f (0x1305 bytes) 000726 1789499121.778389 write 0x02473b thru 0x02c9b1 (0x8277 bytes) 000727 1789499121.789931 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe0247c97 (stale) Mirror 9: 0xe0247c97 (stale) Mirror 11: 0xe0247c97 (stale) Mirror 13: 0xe0247c97 (stale) Mirror 20: 0xe0247c97 (stale) Mirror 21: 0xe0247c97 (stale) Mirror 22: 0xe0247c97 (stale) Mirror 23: 0xe0247c97 (stale) Mirror 26: 0xe0247c97 (stale) Mirror 28: 0xe0247c97 (stale) Mirror 29: 0xe0247c97 (stale) Mirror 30: 0xe0247c97 (stale) Mirror 31: 0xe0247c97 (stale) Mirror 32: 0xe0247c97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe0247c97 (stale) Mirror 9: 0xe0247c97 (stale) Mirror 11: 0xe0247c97 (stale) Mirror 13: 0xe0247c97 (stale) Mirror 20: 0xe0247c97 (stale) Mirror 21: 0xe0247c97 (stale) Mirror 22: 0xe0247c97 (stale) Mirror 23: 0xe0247c97 (stale) Mirror 26: 0xe0247c97 (stale) Mirror 28: 0xe0247c97 (stale) Mirror 29: 0xe0247c97 (stale) Mirror 30: 0xe0247c97 (stale) Mirror 31: 0xe0247c97 (stale) Mirror 32: 0xe0247c97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000728 1789499122.310603 mapread 0x037d53 thru 0x03ffff (0x82ad bytes) 000729 1789499122.331706 trunc from 0x040000 to 0x03aa8c (0x5573 bytes) 000730 1789499122.358783 read 0x0107a9 thru 0x01a73c (0x9f94 bytes) 000731 1789499122.371168 mapread 0x01b0d7 thru 0x021c02 (0x6b2c bytes) 000732 1789499122.383514 read 0x0245ca thru 0x032aef (0xe526 bytes) 000733 1789499122.394232 write 0x021c30 thru 0x0272cb (0x569c bytes) 000734 1789499122.397893 mapwrite 0x022014 thru 0x0263c4 (0x43b1 bytes) 000735 1789499122.414621 trunc from 0x013593 to 0x017acc (0x453a bytes) 000736 1789499122.419997 read 0x02b958 thru 0x039664 (0xdd0d bytes) 000737 1789499122.420891 mapwrite 0x014af8 thru 0x01dc9d (0x91a6 bytes) 000739 1789499122.467588 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3aa8d): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa8d): Mirror 7: 0xbeab6f06 (stale) Mirror 9: 0xbeab6f06 (stale) Mirror 11: 0xbeab6f06 (stale) Mirror 13: 0xbeab6f06 (stale) Mirror 20: 0xbeab6f06 (stale) Mirror 21: 0xbeab6f06 (stale) Mirror 22: 0xbeab6f06 (stale) Mirror 23: 0xbeab6f06 (stale) Mirror 26: 0xbeab6f06 (stale) Mirror 28: 0xbeab6f06 (stale) Mirror 29: 0xbeab6f06 (stale) Mirror 30: 0xbeab6f06 (stale) Mirror 31: 0xbeab6f06 (stale) Mirror 32: 0xbeab6f06 (stale) CRC-32 checksum value for chunk [0, 0x3aa8d): Mirror 1: 0x7204630e Mirror 33: 0x7204630e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa8d: skipped 000740 1789499123.306618 read 0x038184 thru 0x03aa8c (0x2909 bytes) 000741 1789499123.322546 mapwrite 0x0234b3 thru 0x023f80 (0xace bytes) 000743 1789499123.407013 trunc from 0x02fbb9 to 0x035898 (0x5ce0 bytes) 000744 1789499123.410439 mapwrite 0x02736c thru 0x02be3b (0x4ad0 bytes) 000745 1789499123.428471 trunc from 0x03aa8d to 0x039e16 (0xc76 bytes) 000746 1789499123.456733 mapread 0x02a3c0 thru 0x03486e (0xa4af bytes) 000747 1789499123.465198 punch from 0x01e66d to 0x023af0 (0x5484 bytes) 000749 1789499123.468991 trunc from 0x039e17 to 0x02ddb8 (0xc05e bytes) 000750 1789499123.479222 trunc from 0x0171be to 0x01df73 (0x6db6 bytes) 000751 1789499123.482955 mapread 0x0126fa thru 0x01d8fb (0xb202 bytes) 000752 1789499123.499362 write 0x013229 thru 0x017f8c (0x4d64 bytes) 000753 1789499123.501009 read 0x010427 thru 0x010ffa (0xbd4 bytes) 000754 1789499123.501614 trunc from 0x0079ed to 0x017458 (0xfa6c bytes) 000755 1789499123.511001 trunc from 0x00f096 to 0x01cd68 (0xdcd3 bytes) 000757 1789499123.514400 trunc from 0x02ddb9 to 0x023a56 (0xa362 bytes) 000758 1789499123.528073 mapread 0x014536 thru 0x0200c5 (0xbb90 bytes) 000759 1789499123.541449 punch from 0x00b53c to 0x0166e5 (0xb1aa bytes) 000761 1789499123.549049 mapwrite 0x02995c thru 0x037f1d (0xe5c2 bytes) 000762 1789499123.589934 write 0x023a33 thru 0x026ed9 (0x34a7 bytes) 000763 1789499123.590984 punch from 0x02a013 to 0x02d0e5 (0x30d3 bytes) 000766 1789499123.601163 trunc from 0x037f1e to 0x017ed7 (0x20046 bytes) 000768 1789499123.618049 fallocate 0x0246e0 thru 0x02dbf9 (0x951a bytes) 000769 1789499123.622887 trunc from 0x017ed8 to 0x007d65 (0x10172 bytes) 000770 1789499123.640022 punch from 0x000441 to 0x004a0c (0x45cc bytes) 000771 1789499123.643832 mapread 0x001b9d thru 0x0054b2 (0x3916 bytes) 000772 1789499123.657522 write 0x006862 thru 0x00ebd3 (0x8372 bytes) 000773 1789499123.658812 mapread 0x002c19 thru 0x003d9b (0x1183 bytes) 000774 1789499123.660171 mapwrite 0x006ae8 thru 0x00d0b9 (0x65d2 bytes) 000775 1789499123.673620 fallocate 0x00d629 thru 0x00d70c (0xe4 bytes) 000776 1789499123.678338 mapread 0x0038f6 thru 0x006b81 (0x328c bytes) 000777 1789499123.679433 punch from 0x0047ff to 0x00ebd3 (0xa3d5 bytes) 000778 1789499123.682647 mapwrite 0x01ac50 thru 0x01ffc1 (0x5372 bytes) 000781 1789499123.734150 mapread 0x0153fc thru 0x01ffc1 (0xabc6 bytes) 000782 1789499123.753195 MIRROR_RESYNC 16 mirrors 000783 1789499125.832473 mapwrite 0x024c0d thru 0x02b0ab (0x649f bytes) 000784 1789499125.917766 fallocate 0x028c81 thru 0x037fae (0xf32e bytes) 000786 1789499125.926266 fallocate 0x0162cb thru 0x024749 (0xe47f bytes) 000787 1789499125.937894 MIRROR_RESYNC 16 mirrors 000788 1789499128.849874 mapread 0x00bb49 thru 0x010dea (0x52a2 bytes) 000789 1789499128.863604 punch from 0x005a4d to 0x00cd72 (0x7326 bytes) 000790 1789499128.907485 read 0x0208a0 thru 0x029576 (0x8cd7 bytes) 000791 1789499128.917398 mapread 0x01036f thru 0x013acf (0x3761 bytes) 000792 1789499128.924610 write 0x00645d thru 0x012640 (0xc1e4 bytes) 000793 1789499128.929533 mapwrite 0x0368a9 thru 0x03ffff (0x9757 bytes) 000794 1789499128.973090 fallocate 0x00ddb9 thru 0x019eed (0xc135 bytes) 000795 1789499128.979162 write 0x03d44f thru 0x03ffff (0x2bb1 bytes) 000798 1789499128.983137 mapread 0x01d1e1 thru 0x0241c1 (0x6fe1 bytes) 000799 1789499128.989981 mapread 0x003900 thru 0x00c056 (0x8757 bytes) 000800 1789499129.000097 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xba24b931 (stale) Mirror 11: 0xba24b931 (stale) Mirror 13: 0xba24b931 (stale) Mirror 20: 0xba24b931 (stale) Mirror 21: 0xba24b931 (stale) Mirror 22: 0xba24b931 (stale) Mirror 23: 0xba24b931 (stale) Mirror 26: 0xba24b931 (stale) Mirror 28: 0xba24b931 (stale) Mirror 29: 0xba24b931 (stale) Mirror 30: 0xba24b931 (stale) Mirror 31: 0xba24b931 (stale) Mirror 32: 0xba24b931 (stale) Mirror 33: 0xba24b931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xba24b931 (stale) Mirror 11: 0xba24b931 (stale) Mirror 13: 0xba24b931 (stale) Mirror 20: 0xba24b931 (stale) Mirror 21: 0xba24b931 (stale) Mirror 22: 0xba24b931 (stale) Mirror 23: 0xba24b931 (stale) Mirror 26: 0xba24b931 (stale) Mirror 28: 0xba24b931 (stale) Mirror 29: 0xba24b931 (stale) Mirror 30: 0xba24b931 (stale) Mirror 31: 0xba24b931 (stale) Mirror 32: 0xba24b931 (stale) Mirror 33: 0xba24b931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1789499129.571900 write 0x01a5fd thru 0x027deb (0xd7ef bytes) 000802 1789499129.584393 trunc from 0x022644 to 0x02d209 (0xabc6 bytes) 000803 1789499129.594157 trunc from 0x01d6e2 to 0x023dba (0x66d9 bytes) 000804 1789499129.597359 mapwrite 0x031ef2 thru 0x033776 (0x1885 bytes) 000805 1789499129.620966 write 0x017143 thru 0x025c82 (0xeb40 bytes) 000806 1789499129.641692 fallocate 0x03fd37 thru 0x03ffff (0x2c9 bytes) 000807 1789499129.645573 trunc from 0x040000 to 0x019449 (0x26bb6 bytes) 000808 1789499129.663460 read 0x011251 thru 0x01243e (0x11ee bytes) 000809 1789499129.668209 trunc from 0x01944a to 0x0251a6 (0xbd5d bytes) 000810 1789499129.683132 punch from 0x020546 to 0x0251a6 (0x4c61 bytes) 000811 1789499129.688146 trunc from 0x0251a7 to 0x025de2 (0xc3c bytes) 000814 1789499129.705874 mapread 0x019fee thru 0x025de2 (0xbdf5 bytes) 000815 1789499129.717494 trunc from 0x025de3 to 0x015098 (0x10d4a bytes) 000816 1789499129.738548 MIRROR_RESYNC 15 mirrors 000817 1789499131.782990 write 0x013aca thru 0x019f4c (0x6483 bytes) 000818 1789499131.819477 trunc from 0x019f4d to 0x0376fa (0x1d7ae bytes) 000819 1789499131.834316 punch from 0x0067f3 to 0x00c5af (0x5dbd bytes) 000820 1789499131.839566 write 0x0198c5 thru 0x01c515 (0x2c51 bytes) 000821 1789499131.844133 trunc from 0x021206 to 0x02750f (0x630a bytes) 000822 1789499131.848194 mapread 0x0091f0 thru 0x017321 (0xe132 bytes) 000824 1789499131.862584 write 0x004698 thru 0x006833 (0x219c bytes) 000825 1789499131.863523 read 0x0089fe thru 0x010902 (0x7f05 bytes) 000826 1789499131.864078 read 0x00bee9 thru 0x00f292 (0x33aa bytes) 000827 1789499131.872552 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: 0x3e1699a9 (stale) Mirror 11: 0x3e1699a9 (stale) Mirror 13: 0x3e1699a9 (stale) Mirror 20: 0x3e1699a9 (stale) Mirror 21: 0x3e1699a9 (stale) Mirror 22: 0x3e1699a9 (stale) Mirror 23: 0x3e1699a9 (stale) Mirror 26: 0x3e1699a9 (stale) Mirror 28: 0x3e1699a9 (stale) Mirror 29: 0x3e1699a9 (stale) Mirror 30: 0x3e1699a9 (stale) Mirror 31: 0x3e1699a9 (stale) Mirror 32: 0x3e1699a9 (stale) Mirror 33: 0x3e1699a9 (stale) CRC-32 checksum value for chunk [0, 0x376fb): Mirror 1: 0xd3b854c9 Mirror 34: 0xd3b854c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376fb: skipped 000829 1789499132.670314 fallocate 0x015ef4 thru 0x0249c4 (0xead1 bytes) 000830 1789499132.701377 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: 0x3e1699a9 (stale) Mirror 13: 0x3e1699a9 (stale) Mirror 20: 0x3e1699a9 (stale) Mirror 21: 0x3e1699a9 (stale) Mirror 22: 0x3e1699a9 (stale) Mirror 23: 0x3e1699a9 (stale) Mirror 26: 0x3e1699a9 (stale) Mirror 28: 0x3e1699a9 (stale) Mirror 29: 0x3e1699a9 (stale) Mirror 30: 0x3e1699a9 (stale) Mirror 31: 0x3e1699a9 (stale) Mirror 32: 0x3e1699a9 (stale) Mirror 33: 0x3e1699a9 (stale) Mirror 34: 0xd3b854c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: 0x3e1699a9 (stale) Mirror 13: 0x3e1699a9 (stale) Mirror 20: 0x3e1699a9 (stale) Mirror 21: 0x3e1699a9 (stale) Mirror 22: 0x3e1699a9 (stale) Mirror 23: 0x3e1699a9 (stale) Mirror 26: 0x3e1699a9 (stale) Mirror 28: 0x3e1699a9 (stale) Mirror 29: 0x3e1699a9 (stale) Mirror 30: 0x3e1699a9 (stale) Mirror 31: 0x3e1699a9 (stale) Mirror 32: 0x3e1699a9 (stale) Mirror 33: 0x3e1699a9 (stale) Mirror 34: 0xd3b854c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376fb: skipped 000831 1789499133.304894 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376fb): Mirror 7: 0x3e1699a9 (stale) Mirror 13: 0x3e1699a9 (stale) Mirror 20: 0x3e1699a9 (stale) Mirror 21: 0x3e1699a9 (stale) Mirror 22: 0x3e1699a9 (stale) Mirror 23: 0x3e1699a9 (stale) Mirror 26: 0x3e1699a9 (stale) Mirror 28: 0x3e1699a9 (stale) Mirror 29: 0x3e1699a9 (stale) Mirror 30: 0x3e1699a9 (stale) Mirror 31: 0x3e1699a9 (stale) Mirror 32: 0x3e1699a9 (stale) Mirror 33: 0x3e1699a9 (stale) Mirror 34: 0xd3b854c9 (stale) CRC-32 checksum value for chunk [0, 0x376fb): Mirror 1: 0xd3b854c9 Mirror 35: 0xd3b854c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376fb: skipped 000832 1789499134.140900 punch from 0x00f24a to 0x01b5bf (0xc376 bytes) 000833 1789499134.168952 trunc from 0x0376fb to 0x004262 (0x33498 bytes) 000834 1789499134.187344 read 0x00229d thru 0x004262 (0x1fc6 bytes) 000835 1789499134.193723 fallocate 0x0394e2 thru 0x03ffff (0x6b1e bytes) 000836 1789499134.196804 punch from 0x033098 to 0x03d5bb (0xa524 bytes) 000837 1789499134.202867 read 0x015522 thru 0x022659 (0xd138 bytes) 000839 1789499134.214431 read 0x0229af thru 0x02df5a (0xb5ac bytes) 000840 1789499134.215091 read 0x005a1c thru 0x00c15b (0x6740 bytes) 000841 1789499134.216070 fallocate 0x00e9f7 thru 0x016fc7 (0x85d1 bytes) 000842 1789499134.219821 trunc from 0x03c69c to 0x03ca03 (0x368 bytes) 000843 1789499134.224340 punch from 0x0261c5 to 0x0265b7 (0x3f3 bytes) 000844 1789499134.230131 mapread 0x015b9d thru 0x01fefe (0xa362 bytes) 000846 1789499134.231898 fallocate 0x02d3ae thru 0x037af6 (0xa749 bytes) 000847 1789499134.236079 read 0x00d6b6 thru 0x014cfe (0x7649 bytes) 000848 1789499134.236673 trunc from 0x040000 to 0x012591 (0x2da6e bytes) 000849 1789499134.254471 trunc from 0x012592 to 0x0173cc (0x4e3b bytes) 000850 1789499134.266729 write 0x00b12d thru 0x010a49 (0x591d bytes) 000851 1789499134.267926 punch from 0x006d12 to 0x00fee2 (0x91d1 bytes) 000852 1789499134.279205 fallocate 0x03fd96 thru 0x03ffff (0x26a bytes) 000853 1789499134.282755 trunc from 0x040000 to 0x020db0 (0x1f24f bytes) 000854 1789499134.292085 trunc from 0x0069f2 to 0x00ee32 (0x8441 bytes) 000855 1789499134.298126 write 0x005117 thru 0x008320 (0x320a bytes) 000856 1789499134.304022 write 0x004e76 thru 0x014279 (0xf404 bytes) 000858 1789499134.310893 write 0x0240d4 thru 0x02af6f (0x6e9c bytes) 000859 1789499134.312512 mapwrite 0x018078 thru 0x01fd89 (0x7d12 bytes) 000860 1789499134.346882 mapread 0x0201d0 thru 0x02af6f (0xada0 bytes) 000861 1789499134.347829 fallocate 0x032b8d thru 0x037eb9 (0x532d bytes) 000862 1789499134.351880 trunc from 0x037eba to 0x01c515 (0x1b9a4 bytes) 000864 1789499134.363966 punch from 0x0161a8 to 0x0164a6 (0x2ff bytes) 000865 1789499134.371338 write 0x024f28 thru 0x02ff03 (0xafdc bytes) 000866 1789499134.373588 fallocate 0x01457a thru 0x01ff2e (0xb9b5 bytes) 000867 1789499134.382144 mapwrite 0x01b4cd thru 0x02456a (0x909e bytes) 000869 1789499134.425132 write 0x005691 thru 0x00dce4 (0x8654 bytes) 000871 1789499134.430354 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff04): Mirror 7: 0xcc226b5f (stale) Mirror 13: 0xcc226b5f (stale) Mirror 20: 0xcc226b5f (stale) Mirror 21: 0xcc226b5f (stale) Mirror 22: 0xcc226b5f (stale) Mirror 23: 0xcc226b5f (stale) Mirror 26: 0xcc226b5f (stale) Mirror 29: 0xcc226b5f (stale) Mirror 30: 0xcc226b5f (stale) Mirror 31: 0xcc226b5f (stale) Mirror 32: 0xcc226b5f (stale) Mirror 33: 0xcc226b5f (stale) Mirror 34: 0xe3ffb25 (stale) Mirror 35: 0xe3ffb25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff04): Mirror 7: 0xcc226b5f (stale) Mirror 13: 0xcc226b5f (stale) Mirror 20: 0xcc226b5f (stale) Mirror 21: 0xcc226b5f (stale) Mirror 22: 0xcc226b5f (stale) Mirror 23: 0xcc226b5f (stale) Mirror 26: 0xcc226b5f (stale) Mirror 29: 0xcc226b5f (stale) Mirror 30: 0xcc226b5f (stale) Mirror 31: 0xcc226b5f (stale) Mirror 32: 0xcc226b5f (stale) Mirror 33: 0xcc226b5f (stale) Mirror 34: 0xe3ffb25 (stale) Mirror 35: 0xe3ffb25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff04: skipped 000872 1789499134.970306 write 0x02f7e4 thru 0x036138 (0x6955 bytes) 000873 1789499134.981085 read 0x01e650 thru 0x0212b1 (0x2c62 bytes) 000874 1789499134.992735 punch from 0x01e23e to 0x02027a (0x203d bytes) 000875 1789499134.998926 fallocate 0x029b68 thru 0x0343fe (0xa897 bytes) 000876 1789499135.004483 read 0x01e028 thru 0x01ebab (0xb84 bytes) 000877 1789499135.034476 write 0x019633 thru 0x01ceba (0x3888 bytes) 000878 1789499135.039440 trunc from 0x036139 to 0x01e892 (0x178a6 bytes) 000879 1789499135.052751 trunc from 0x01e893 to 0x010540 (0xe352 bytes) 000880 1789499135.070809 fallocate 0x01ed61 thru 0x0247fa (0x5a9a bytes) 000881 1789499135.081909 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10541): Mirror 13: 0xd23a9a2f (stale) Mirror 20: 0xd23a9a2f (stale) Mirror 21: 0xd23a9a2f (stale) Mirror 22: 0xd23a9a2f (stale) Mirror 23: 0xd23a9a2f (stale) Mirror 26: 0xd23a9a2f (stale) Mirror 29: 0xd23a9a2f (stale) Mirror 30: 0xd23a9a2f (stale) Mirror 31: 0xd23a9a2f (stale) Mirror 32: 0xd23a9a2f (stale) Mirror 33: 0xd23a9a2f (stale) Mirror 34: 0xd2915b98 (stale) Mirror 35: 0xd2915b98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10541): Mirror 13: 0xd23a9a2f (stale) Mirror 20: 0xd23a9a2f (stale) Mirror 21: 0xd23a9a2f (stale) Mirror 22: 0xd23a9a2f (stale) Mirror 23: 0xd23a9a2f (stale) Mirror 26: 0xd23a9a2f (stale) Mirror 29: 0xd23a9a2f (stale) Mirror 30: 0xd23a9a2f (stale) Mirror 31: 0xd23a9a2f (stale) Mirror 32: 0xd23a9a2f (stale) Mirror 33: 0xd23a9a2f (stale) Mirror 34: 0xd2915b98 (stale) Mirror 35: 0xd2915b98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10541: skipped 000882 1789499135.556124 write 0x027e37 thru 0x037018 (0xf1e2 bytes) 000884 1789499135.562327 read 0x00f49b thru 0x01aa0a (0xb570 bytes) 000885 1789499135.568882 punch from 0x027f90 to 0x02e672 (0x66e3 bytes) 000886 1789499135.578212 trunc from 0x031254 to 0x031e8f (0xc3c bytes) 000887 1789499135.584647 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x37019): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37019): Mirror 13: 0x4150addb (stale) Mirror 20: 0x4150addb (stale) Mirror 21: 0x4150addb (stale) Mirror 22: 0x4150addb (stale) Mirror 23: 0x4150addb (stale) Mirror 26: 0x4150addb (stale) Mirror 29: 0x4150addb (stale) Mirror 30: 0x4150addb (stale) Mirror 31: 0x4150addb (stale) Mirror 32: 0x4150addb (stale) Mirror 33: 0x4150addb (stale) Mirror 34: 0xc9de726c (stale) Mirror 35: 0xc9de726c (stale) CRC-32 checksum value for chunk [0, 0x37019): Mirror 1: 0xdf5ead14 Mirror 36: 0xdf5ead14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37019: skipped 000888 1789499136.351446 read 0x023709 thru 0x02eef2 (0xb7ea bytes) 000889 1789499136.366219 punch from 0x00f833 to 0x01b5da (0xbda8 bytes) 000890 1789499136.398627 trunc from 0x01ccf8 to 0x024510 (0x7819 bytes) 000891 1789499136.401726 fallocate 0x029f1d thru 0x033179 (0x925d bytes) 000892 1789499136.404305 write 0x00070a thru 0x00508e (0x4985 bytes) 000894 1789499136.416529 read 0x01fb32 thru 0x02efa3 (0xf472 bytes) 000895 1789499136.423028 read 0x031b9e thru 0x037018 (0x547b bytes) 000896 1789499136.423648 fallocate 0x02ac29 thru 0x032c19 (0x7ff1 bytes) 000897 1789499136.426956 punch from 0x02ea44 to 0x02f8e2 (0xe9f bytes) 000899 1789499136.431267 read 0x010cf5 thru 0x018f0c (0x8218 bytes) 000900 1789499136.432050 write 0x0123bd thru 0x01ae6c (0x8ab0 bytes) 000901 1789499136.437132 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37019): Mirror 13: 0x4150addb (stale) Mirror 20: 0x4150addb (stale) Mirror 21: 0x4150addb (stale) Mirror 22: 0x4150addb (stale) Mirror 23: 0x4150addb (stale) Mirror 26: 0x4150addb (stale) Mirror 29: 0x4150addb (stale) Mirror 30: 0x4150addb (stale) Mirror 32: 0x4150addb (stale) Mirror 33: 0x4150addb (stale) Mirror 34: 0xc9de726c (stale) Mirror 35: 0xc9de726c (stale) Mirror 36: 0xdf5ead14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37019): Mirror 13: 0x4150addb (stale) Mirror 20: 0x4150addb (stale) Mirror 21: 0x4150addb (stale) Mirror 22: 0x4150addb (stale) Mirror 23: 0x4150addb (stale) Mirror 26: 0x4150addb (stale) Mirror 29: 0x4150addb (stale) Mirror 30: 0x4150addb (stale) Mirror 32: 0x4150addb (stale) Mirror 33: 0x4150addb (stale) Mirror 34: 0xc9de726c (stale) Mirror 35: 0xc9de726c (stale) Mirror 36: 0xdf5ead14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37019: skipped 000902 1789499136.892424 fallocate 0x039cbb thru 0x03ffff (0x6345 bytes) 000903 1789499136.896337 trunc from 0x040000 to 0x00021b (0x3fde4 bytes) 000904 1789499136.922170 write 0x019112 thru 0x02613f (0xd02e bytes) 000905 1789499136.924933 punch from 0x01fc25 to 0x01fcc7 (0xa3 bytes) 000907 1789499136.941481 trunc from 0x026140 to 0x036f7d (0x10e3e bytes) 000908 1789499136.953294 trunc from 0x009f34 to 0x00f06c (0x5139 bytes) 000909 1789499136.956474 mapread 0x00eb6b thru 0x014b26 (0x5fbc bytes) 000910 1789499136.975334 trunc from 0x036f7e to 0x031663 (0x591a bytes) 000911 1789499136.992001 punch from 0x026198 to 0x02d33a (0x71a3 bytes) 000912 1789499137.002060 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31664): Mirror 13: 0xebdeba25 (stale) Mirror 20: 0xebdeba25 (stale) Mirror 21: 0xebdeba25 (stale) Mirror 22: 0xebdeba25 (stale) Mirror 23: 0xebdeba25 (stale) Mirror 26: 0xebdeba25 (stale) Mirror 29: 0xebdeba25 (stale) Mirror 30: 0xebdeba25 (stale) Mirror 33: 0xebdeba25 (stale) Mirror 34: 0xd2ab7d0a (stale) Mirror 35: 0xd2ab7d0a (stale) Mirror 36: 0x4de835f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31664): Mirror 13: 0xebdeba25 (stale) Mirror 20: 0xebdeba25 (stale) Mirror 21: 0xebdeba25 (stale) Mirror 22: 0xebdeba25 (stale) Mirror 23: 0xebdeba25 (stale) Mirror 26: 0xebdeba25 (stale) Mirror 29: 0xebdeba25 (stale) Mirror 30: 0xebdeba25 (stale) Mirror 33: 0xebdeba25 (stale) Mirror 34: 0xd2ab7d0a (stale) Mirror 35: 0xd2ab7d0a (stale) Mirror 36: 0x4de835f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31664: skipped 000913 1789499137.413933 read 0x01bd6f thru 0x02bd21 (0xffb3 bytes) 000914 1789499137.426396 write 0x006ec2 thru 0x0144c2 (0xd601 bytes) 000915 1789499137.448962 punch from 0x0304f3 to 0x031663 (0x1171 bytes) 000916 1789499137.459942 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x31664): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31664): Mirror 13: 0xebdeba25 (stale) Mirror 20: 0xebdeba25 (stale) Mirror 21: 0xebdeba25 (stale) Mirror 22: 0xebdeba25 (stale) Mirror 23: 0xebdeba25 (stale) Mirror 26: 0xebdeba25 (stale) Mirror 29: 0xebdeba25 (stale) Mirror 30: 0xebdeba25 (stale) Mirror 33: 0xebdeba25 (stale) Mirror 34: 0xd2ab7d0a (stale) Mirror 35: 0xd2ab7d0a (stale) Mirror 36: 0x4de835f0 (stale) CRC-32 checksum value for chunk [0, 0x31664): Mirror 1: 0x73a9ef2f Mirror 37: 0x73a9ef2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31664: skipped 000917 1789499138.180228 fallocate 0x02c920 thru 0x03664d (0x9d2e bytes) 000918 1789499138.225459 trunc from 0x03664e to 0x019a3e (0x1cc0f bytes) 000919 1789499138.264708 mapwrite 0x00a843 thru 0x00ce43 (0x2601 bytes) 000920 1789499138.320481 fallocate 0x00c3ee thru 0x00cb5a (0x76d bytes) 000921 1789499138.335422 write 0x01ad1b thru 0x025079 (0xa35f bytes) 000922 1789499138.337629 fallocate 0x008f5c thru 0x00b25c (0x2301 bytes) 000923 1789499138.346589 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2507a): Mirror 13: 0x771688c0 (stale) Mirror 20: 0x771688c0 (stale) Mirror 21: 0x771688c0 (stale) Mirror 22: 0x771688c0 (stale) Mirror 23: 0x771688c0 (stale) Mirror 26: 0x771688c0 (stale) Mirror 29: 0x771688c0 (stale) Mirror 33: 0x771688c0 (stale) Mirror 34: 0xb1d9faa2 (stale) Mirror 35: 0xb1d9faa2 (stale) Mirror 36: 0xfa34a76f (stale) Mirror 37: 0xe9baeb52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2507a): Mirror 13: 0x771688c0 (stale) Mirror 20: 0x771688c0 (stale) Mirror 21: 0x771688c0 (stale) Mirror 22: 0x771688c0 (stale) Mirror 23: 0x771688c0 (stale) Mirror 26: 0x771688c0 (stale) Mirror 29: 0x771688c0 (stale) Mirror 33: 0x771688c0 (stale) Mirror 34: 0xb1d9faa2 (stale) Mirror 35: 0xb1d9faa2 (stale) Mirror 36: 0xfa34a76f (stale) Mirror 37: 0xe9baeb52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2507a: skipped 000924 1789499138.887953 trunc from 0x01d1c6 to 0x025079 (0x7eb4 bytes) 000925 1789499138.896060 trunc from 0x013e3a to 0x02348f (0xf656 bytes) 000926 1789499138.905565 MIRROR_RESYNC 13 mirrors 000927 1789499140.598654 MIRROR_RESYNC 13 mirrors 000928 1789499141.045024 punch from 0x01cc35 to 0x01d5e7 (0x9b3 bytes) 000929 1789499141.071364 trunc from 0x019b68 to 0x01fbd9 (0x6072 bytes) 000930 1789499141.074923 read 0x00f165 thru 0x0181fb (0x9097 bytes) 000931 1789499141.081082 write 0x01f48b thru 0x02acd4 (0xb84a bytes) 000932 1789499141.088724 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2acd5): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acd5): Mirror 13: 0x8ef1b0ec (stale) Mirror 20: 0x8ef1b0ec (stale) Mirror 21: 0x8ef1b0ec (stale) Mirror 22: 0x8ef1b0ec (stale) Mirror 23: 0x8ef1b0ec (stale) Mirror 26: 0x8ef1b0ec (stale) Mirror 29: 0x8ef1b0ec (stale) Mirror 33: 0x8ef1b0ec (stale) Mirror 34: 0x8ef1b0ec (stale) Mirror 35: 0x8ef1b0ec (stale) Mirror 36: 0x8ef1b0ec (stale) Mirror 37: 0x8ef1b0ec (stale) CRC-32 checksum value for chunk [0, 0x2acd5): Mirror 1: 0x283fb7d4 Mirror 38: 0x283fb7d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acd5: skipped 000934 1789499141.871296 mapwrite 0x02aca3 thru 0x02e6fc (0x3a5a bytes) 000935 1789499141.986734 punch from 0x018a0b to 0x01b49b (0x2a91 bytes) 000936 1789499141.989548 write 0x02415a thru 0x02d1d4 (0x907b bytes) 000937 1789499141.993703 mapread 0x000e58 thru 0x00d4c6 (0xc66f bytes) 000938 1789499142.001872 fallocate 0x0246d7 thru 0x0258fb (0x1225 bytes) 000939 1789499142.004579 write 0x02648f thru 0x02794e (0x14c0 bytes) 000940 1789499142.005228 write 0x027d92 thru 0x029832 (0x1aa1 bytes) 000941 1789499142.005859 punch from 0x02cb1e to 0x02e6fc (0x1bdf bytes) 000942 1789499142.014529 mapwrite 0x00ee15 thru 0x018e16 (0xa002 bytes) 000943 1789499142.035863 mapread 0x028dd0 thru 0x02e6fc (0x592d bytes) 000944 1789499142.040427 fallocate 0x00e857 thru 0x01cb2f (0xe2d9 bytes) 000945 1789499142.043432 write 0x0316d0 thru 0x034bc0 (0x34f1 bytes) 000946 1789499142.044928 mapwrite 0x0015dc thru 0x008091 (0x6ab6 bytes) 000947 1789499142.066885 mapwrite 0x021231 thru 0x02a713 (0x94e3 bytes) 000948 1789499142.100859 MIRROR_RESYNC 14 mirrors 000949 1789499144.403497 trunc from 0x034bc1 to 0x02633e (0xe882 bytes) 000951 1789499144.437997 mapwrite 0x00b8fa thru 0x00cd6b (0x1472 bytes) 000952 1789499144.455986 mapwrite 0x0040c0 thru 0x013580 (0xf4c1 bytes) 000953 1789499144.480850 fallocate 0x0142d8 thru 0x01fd6a (0xba93 bytes) 000954 1789499144.483566 trunc from 0x00600e to 0x00619f (0x192 bytes) 000955 1789499144.487040 mapwrite 0x01b543 thru 0x01f062 (0x3b20 bytes) 000956 1789499144.511490 trunc from 0x02633f to 0x01640a (0xff34 bytes) 000957 1789499144.523440 trunc from 0x01640b to 0x00f9fb (0x6a0f bytes) 000960 1789499144.537381 punch from 0x00aecc to 0x00f9fb (0x4b30 bytes) 000961 1789499144.543411 mapread 0x008c55 thru 0x00f9fb (0x6da7 bytes) 000962 1789499144.552803 write 0x020d73 thru 0x02cc3f (0xbecd bytes) 000963 1789499144.554307 punch from 0x0241e4 to 0x02cc3f (0x8a5c bytes) 000964 1789499144.564815 fallocate 0x00076f thru 0x006af6 (0x6388 bytes) 000965 1789499144.568493 trunc from 0x025e71 to 0x02cc3f (0x6dcf bytes) 000966 1789499144.571674 write 0x0193e4 thru 0x01f44e (0x606b bytes) 000967 1789499144.579893 MIRROR_RESYNC 14 mirrors 000968 1789499146.698193 punch from 0x01eba9 to 0x029857 (0xacaf bytes) 000969 1789499146.722428 fallocate 0x03dbfd thru 0x03ffff (0x2403 bytes) 000970 1789499146.726634 trunc from 0x02cc40 to 0x026d3b (0x5f04 bytes) 000971 1789499146.744597 fallocate 0x02028b thru 0x02bfb3 (0xbd29 bytes) 000972 1789499146.752285 write 0x03efcf thru 0x03ffff (0x1031 bytes) 000975 1789499146.758782 trunc from 0x040000 to 0x0369af (0x9650 bytes) 000976 1789499146.772639 trunc from 0x0218e2 to 0x02d3d0 (0xbaef bytes) 000977 1789499146.777487 fallocate 0x00ad09 thru 0x00b668 (0x960 bytes) 000978 1789499146.781145 trunc from 0x0369b0 to 0x024741 (0x1226e bytes) 000979 1789499146.791549 write 0x003a4a thru 0x00d8bd (0x9e74 bytes) 000980 1789499146.797387 trunc from 0x005a56 to 0x00da49 (0x7ff4 bytes) 000981 1789499146.811604 fallocate 0x03bb54 thru 0x03ffff (0x44ac bytes) 000982 1789499146.816676 mapwrite 0x036ce5 thru 0x03b689 (0x49a5 bytes) 000983 1789499146.847504 trunc from 0x040000 to 0x0004c1 (0x3fb3e bytes) 000984 1789499146.857349 trunc from 0x0004c2 to 0x039fea (0x39b29 bytes) 000985 1789499146.869296 mapwrite 0x024287 thru 0x03135f (0xd0d9 bytes) 000986 1789499146.912666 trunc from 0x039feb to 0x001490 (0x38b5a bytes) 000987 1789499146.931907 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2e8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1491): Mirror 13: 0x6a181361 (stale) Mirror 20: 0x6a181361 (stale) Mirror 21: 0x6a181361 (stale) Mirror 22: 0x6a181361 (stale) Mirror 26: 0x6a181361 (stale) Mirror 29: 0x6a181361 (stale) Mirror 33: 0x6a181361 (stale) Mirror 34: 0x6a181361 (stale) Mirror 35: 0x6a181361 (stale) Mirror 36: 0x6a181361 (stale) Mirror 37: 0x6a181361 (stale) Mirror 38: 0x6a181361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1491): Mirror 13: 0x6a181361 (stale) Mirror 20: 0x6a181361 (stale) Mirror 21: 0x6a181361 (stale) Mirror 22: 0x6a181361 (stale) Mirror 26: 0x6a181361 (stale) Mirror 29: 0x6a181361 (stale) Mirror 33: 0x6a181361 (stale) Mirror 34: 0x6a181361 (stale) Mirror 35: 0x6a181361 (stale) Mirror 36: 0x6a181361 (stale) Mirror 37: 0x6a181361 (stale) Mirror 38: 0x6a181361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1491: skipped 000988 1789499147.203791 write 0x01d48a thru 0x02c24e (0xedc5 bytes) 000989 1789499147.207667 fallocate 0x017df1 thru 0x01c06b (0x427b bytes) 000990 1789499147.210185 mapread 0x002e4a thru 0x00b976 (0x8b2d bytes) 000991 1789499147.221653 write 0x034044 thru 0x03bdec (0x7da9 bytes) 000993 1789499147.223503 fallocate 0x00b33c thru 0x01a925 (0xf5ea bytes) 000994 1789499147.226731 fallocate 0x01a68c thru 0x01e562 (0x3ed7 bytes) 000996 1789499147.230238 fallocate 0x035435 thru 0x038bd9 (0x37a5 bytes) 000997 1789499147.233035 trunc from 0x03bded to 0x03b488 (0x964 bytes) 000998 1789499147.240735 fallocate 0x005db2 thru 0x01088d (0xaadc bytes) 000999 1789499147.243545 read 0x03054c thru 0x03a430 (0x9ee5 bytes) 001000 1789499147.246529 trunc from 0x016c6b to 0x01d5ba (0x6950 bytes) All operations completed A-OK! PASS 70a (85s) == sanity-flr test 71: check mirror extend parent fid ==== 15:05:50 (1789499150) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 15:05:55 (1789499155) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112972 s, 18.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684061 s, 15.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 15:06:00 (1789499160) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4051616 .. Extending file size to 6621824 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7281408 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8050816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8268128 .. Extending file size to 9220992 .. Extending file size to 8942976 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9117664 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62202 62203 62205 62206 62209 done 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) 192.168.206.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:07:13 (1789499233) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5596288 .. Extending file size to 522048 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6183584 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8113344 .. Extending file size to 6214656 .. failed Extending file size to 6581376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8208480 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8228192 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8619936 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9347744 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8802720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64185 64186 64188 64190 64191 done 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) 192.168.206.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:08:27 (1789499307) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1268924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2564 1268740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26068 3580952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44136 7169904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1268924 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2564 1268740 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26068 3580952 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44136 7169904 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521148 s, 40.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119938 s, 17.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xad4737a0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:08:36 (1789499316) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0934991 s, 22.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 15:08:39 (1789499319) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:08:43 (1789499323) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0598877 s, 35.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x41a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x41b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x41c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:08:47 (1789499327) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475372 s, 44.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:08:51 (1789499331) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0602446 s, 34.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:09:18 (1789499358) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298818 s, 35.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } PASS 204d (37s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:09:56 (1789499396) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475645 s, 44.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:10:33 (1789499433) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg603-server: Pool lustre.test_204f created Adding targets to pool oleg603-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg603-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.056987 s, 36.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg603-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg603-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg603-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:11:24 (1789499484) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:11:28 (1789499488) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:11:32 (1789499492) oleg603-server: Pool lustre.test_206 created Adding targets to pool oleg603-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg603-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg603-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg603-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg603-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:11:52 (1789499512) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486929 s, 21.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181229 s, 57.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x28ee890 Mirror 3: 0x28ee890 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:11:57 (1789499517) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:11:59 (1789499519) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:12:02 (1789499522) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0302338 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x680000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xeb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xeb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xeb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2020000 Read this (4 bytes): BBBB 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) PASS 209a (27s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:12:29 (1789499549) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0115704 s, 708 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:12:33 (1789499553) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270705 s, 38.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x121:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611271 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 15:12:34.000000000 -0400 Modify: 2026-09-15 15:12:34.000000000 -0400 Change: 2026-09-15 15:12:35.000000000 -0400 Birth: 2026-09-15 15:12:34.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:12:38 (1789499558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408983 s, 25.6 MB/s oleg603-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg603-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg603-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:12:57 (1789499577) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.142039 s, 288 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0245591 s, 167 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 15 15:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 15 15:12 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:13:02 (1789499582) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:13:06 (1789499586) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1152 sec =========== 15:13:10 (1789499590) === sanity-flr: start cleanup 15:13:11 (1789499591) === === sanity-flr: finish cleanup 15:13:16 (1789499596) ===