-----============= acceptance-small: sanity-flr ============----- Tue Sep 22 14:06:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg228-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:06:56 (1790100416) === oleg228-client.virtnet: executing check_config_client /mnt/lustre oleg228-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg228-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff994605282800.idle_timeout=debug osc.lustre-OST0001-osc-ffff994605282800.idle_timeout=debug disable quota as required oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:07:44 (1790100464) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 14:07:47 (1790100467) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:08:06 (1790100486) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:08:10 (1790100490) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:08:13 (1790100493) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:08:38 (1790100518) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:08:41 (1790100521) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:08:45 (1790100525) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff994605282800.stats=clear osc.lustre-OST0001-osc-ffff994605282800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:08:52 (1790100532) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:09:03 (1790100543) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:09:11 (1790100551) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:09:14 (1790100554) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:09:22 (1790100562) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 14:09:30 (1790100570) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:09:39 (1790100579) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:09:48 (1790100588) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:09:58 (1790100598) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:10:10 (1790100610) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:10:18 (1790100618) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.17149 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484377 s, 21.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:10:27 (1790100627) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.238631 s, 22.0 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:10:37 (1790100637) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:10:47 (1790100647) Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:12:04 (1790100724) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.583575 s, 18.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:12:51 (1790100771) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:14:11 (1790100851) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.72597 s, 19.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 22 14:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.800s user 0m0.001s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.362s user 0m0.001s sys 0m0.220s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.314s user 0m0.000s sys 0m0.197s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994605282800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:15:56 (1790100956) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:16:00 (1790100960) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:16:03 (1790100963) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 14:16:06 (1790100966) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 14:16:11 (1790100971) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.376727 s, 33.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:16:23 (1790100983) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060722 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348347 s, 30.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:16:31 (1790100991) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:16:40 (1790101000) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0027181 s, 377 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00690483 s, 148 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000799847 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107082 s, 956 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:16:50 (1790101010) create test files with size 15747634 .. ebc1627bdfc14f1cf4c1523e6a5b6d0f - 2c501ecdc8d63e6f9d831f5c9b9c3e13 - 7d3188134a62b8811669b008efce4218 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 14:17:14 (1790101034) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:17:25 (1790101045) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:17:33 (1790101053) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638173 s, 16.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069451 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 14:17:42 (1790101062) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112076 s, 18.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.351171 s, 11.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195385 s, 10.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 14:17:59 (1790101079) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:18:03 (1790101083) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (6s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:18:09 (1790101089) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289777 s, 36.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:18:18 (1790101098) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.281197 s, 11.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:18:27 (1790101107) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:18:37 (1790101117) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.525359 s, 20.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:18:50 (1790101130) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9946092d5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.408807 s, 25.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9946181df800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 14:19:06 (1790101146) error lfs split: create victim file failed: File exists PASS 44f (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:19:25 (1790101165) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:20:08 (1790101208) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:20:43 (1790101243) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 14:20:47 (1790101247) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256242 s, 12.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205438 s, 15.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb6bae720 (nosync,stale) Mirror 2: 0xb6bae720 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5c43e354f6e6db62f01d9a1ca25ab9ba - mirror 1 checksum: 5a8404b4fbfc4978d90f0a22381d86ff - mirror 2 checksum: 5c43e354f6e6db62f01d9a1ca25ab9ba - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-22T14:20:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-22T14:20:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5c43e354f6e6db62f01d9a1ca25ab9ba - mirror 1 checksum: 5a8404b4fbfc4978d90f0a22381d86ff - mirror 2 checksum: 5a8404b4fbfc4978d90f0a22381d86ff - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-22T14:20:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-22T14:20:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:21:00 (1790101260) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633743 s, 16.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:21:10 (1790101270) Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 14:21:18 (1790101278) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 14:21:42 (1790101302) targets are mounted 14:21:42 (1790101302) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x9:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360941 s, 29.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x9 lmm_fid: [0x240000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:22:00 (1790101320) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0472552 s, 433 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0211956 s, 966 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:22:12 (1790101332) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.109377 s, 187 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00959023 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:22:24 (1790101344) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00912457 s, 1.8 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:22:34 (1790101354) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0606348 s, 338 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:22:44 (1790101364) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199751 s, 205 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0189143 s, 217 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:22:52 (1790101372) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:23:15 (1790101395) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:23:44 (1790101424) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:24:08 (1790101448) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:24:33 (1790101473) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:24:42 (1790101482) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58236 mirror split pid 58238 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58236 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58238 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:25:49 (1790101549) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1551 truncating to largest ever: 0x1fe78 000001 1790101552.228850 trunc from 00000000 to 0x01fe77 (0x1fe78 bytes) 000002 1790101552.262893 write 0x004e8e thru 0x00cc06 (0x7d79 bytes) 000003 1790101552.283309 punch from 0x00eeb0 to 0x01022b (0x137c bytes) 000004 1790101552.289919 fallocate 0x00d663 thru 0x01abff (0xd59d bytes) 000005 1790101552.301194 write 0x01da45 thru 0x01db12 (0xce bytes) 000007 1790101552.313070 punch from 0x01f8a7 to 0x01fe77 (0x5d1 bytes) 000008 1790101552.320762 punch from 0x010633 to 0x01e191 (0xdb5f bytes) 000009 1790101552.365562 write 0x032c85 thru 0x033fce (0x134a bytes) 000011 1790101552.366979 mapread 0x0015bd thru 0x011291 (0xfcd5 bytes) 000012 1790101552.406045 mapwrite 0x031dcf thru 0x0338b2 (0x1ae4 bytes) 000013 1790101552.455149 punch from 0x01cf40 to 0x02c324 (0xf3e5 bytes) 000015 1790101552.462289 read 0x00be1c thru 0x00ec04 (0x2de9 bytes) truncating to largest ever: 0x22a07 000016 1790101552.463890 trunc from 0x033fcf to 0x022a06 (0x115c8 bytes) 000017 1790101552.485501 mapread 0x0163ba thru 0x020e4a (0xaa91 bytes) 000018 1790101552.513363 write 0x00674e thru 0x0161e3 (0xfa96 bytes) 000019 1790101552.516733 mapread 0x01ba8f thru 0x022a06 (0x6f78 bytes) 000020 1790101552.518429 punch from 0x00d586 to 0x014732 (0x71ad bytes) 000021 1790101552.568755 trunc from 0x01d059 to 0x0226a6 (0x564e bytes) 000023 1790101552.585520 mapread 0x016b34 thru 0x0196c4 (0x2b91 bytes) 000024 1790101552.589921 fallocate 0x007685 thru 0x016e1a (0xf796 bytes) 000025 1790101552.620752 fallocate 0x00a161 thru 0x00dadd (0x397d bytes) 000026 1790101552.627671 mapread 0x019c7f thru 0x022a06 (0x8d88 bytes) 000029 1790101552.661782 read 0x003b0b thru 0x012a2f (0xef25 bytes) 000030 1790101552.683524 punch from 0x01cb33 to 0x01d3fc (0x8ca bytes) 000031 1790101552.699899 trunc from 0x00504c to 0x013fcc (0xef81 bytes) 000032 1790101552.730999 punch from 0x0097de to 0x01211d (0x8940 bytes) 000033 1790101552.754897 trunc from 0x022a07 to 0x01ba69 (0x6f9d bytes) fallocating to largest ever: 0x2b26b 000036 1790101552.852640 fallocate 0x01e081 thru 0x02b26a (0xd1ea bytes) 000037 1790101552.891392 MIRROR_EXTEND to 2 mirrors 000038 1790101554.050369 write 0x02a380 thru 0x037f91 (0xdc12 bytes) 000040 1790101554.100056 read 0x00b5ce thru 0x0182ff (0xcd32 bytes) 000042 1790101554.122474 mapread 0x024aeb thru 0x02b774 (0x6c8a bytes) truncating to largest ever: 0x33f38 000043 1790101554.165908 trunc from 0x037f92 to 0x033f37 (0x405a bytes) 000044 1790101554.206559 trunc from 0x02b525 to 0x032d7b (0x7857 bytes) 000045 1790101554.219509 mapwrite 0x01b70c thru 0x02892f (0xd224 bytes) truncating to largest ever: 0x35d23 000046 1790101554.278242 trunc from 0x033f38 to 0x035d22 (0x1deb bytes) 000048 1790101554.298540 mapread 0x033cdd thru 0x035d22 (0x2046 bytes) 000049 1790101554.334313 trunc from 0x0154a2 to 0x01999d (0x44fc bytes) 000051 1790101554.350455 punch from 0x013fd2 to 0x0160bf (0x20ee bytes) 000052 1790101554.359974 trunc from 0x035d23 to 0x013c7e (0x220a4 bytes) 000053 1790101554.416025 MIRROR_RESYNC 2 mirrors 000054 1790101555.263914 fallocate 0x00f660 thru 0x01afeb (0xb98c bytes) 000055 1790101555.345201 mapread 0x00b24a thru 0x00b46b (0x222 bytes) 000056 1790101555.373068 trunc from 0x00d734 to 0x010631 (0x2efe bytes) truncating to largest ever: 0x35fea 000057 1790101555.385860 trunc from 0x013c7f to 0x035fe9 (0x2236b bytes) 000059 1790101555.455560 mapwrite 0x02ed21 thru 0x0316f1 (0x29d1 bytes) 000060 1790101555.514866 write 0x02f8f1 thru 0x035a54 (0x6164 bytes) 000061 1790101555.517767 trunc from 0x035fea to 0x004e6e (0x3117b bytes) 000062 1790101555.567532 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) 000063 1790101555.816761 mapwrite 0x017fdd thru 0x025abf (0xdae3 bytes) 000066 1790101555.922532 read 0x01746d thru 0x018d9c (0x1930 bytes) 000067 1790101555.923525 trunc from 0x025ac0 to 0x01419f (0x11920 bytes) 000068 1790101555.945351 write 0x01396d thru 0x01de5c (0xa4f0 bytes) 000069 1790101555.946894 mapread 0x014e49 thru 0x01de5c (0x9014 bytes) 000070 1790101555.949760 write 0x0138f3 thru 0x02208d (0xe79b bytes) 000071 1790101555.952907 write 0x03e4c8 thru 0x03ffff (0x1b38 bytes) 000074 1790101555.956918 read 0x03d195 thru 0x03ffff (0x2e6b bytes) 000075 1790101555.973269 mapread 0x01cfc9 thru 0x028af9 (0xbb31 bytes) 000076 1790101555.974603 punch from 0x025be6 to 0x030a2c (0xae47 bytes) 000078 1790101555.981312 mapread 0x002ef7 thru 0x00b83f (0x8949 bytes) 000079 1790101555.983517 mapread 0x0350cd thru 0x03ffff (0xaf33 bytes) 000080 1790101555.988695 MIRROR_EXTEND to 2 mirrors 000081 1790101556.815559 fallocate 0x022342 thru 0x027d4d (0x5a0c bytes) 000082 1790101556.851736 read 0x0100c1 thru 0x01891b (0x885b bytes) 000083 1790101556.877507 mapwrite 0x01f78b thru 0x026454 (0x6cca bytes) 000084 1790101556.934467 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x752c2cb5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe56af59a Mirror 3: 0xe56af59a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000085 1790101557.879227 write 0x00f1d1 thru 0x01b15e (0xbf8e bytes) 000086 1790101557.935980 trunc from 0x040000 to 0x001978 (0x3e687 bytes) 000087 1790101557.968422 trunc from 0x001475 to 0x001978 (0x504 bytes) 000088 1790101557.979202 MIRROR_RESYNC 3 mirrors 000089 1790101558.549531 write 0x01944d thru 0x021348 (0x7efc bytes) 000090 1790101558.586706 punch from 0x004675 to 0x007e53 (0x37df bytes) 000091 1790101558.593153 mapwrite 0x01b116 thru 0x0263d5 (0xb2c0 bytes) 000093 1790101558.638217 fallocate 0x01c560 thru 0x02554e (0x8fef bytes) 000094 1790101558.646533 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x263d6): Mirror 2: 0x60b19da1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x263d6): Mirror 2: 0x60b19da1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x263d6: skipped 000095 1790101558.858135 read 0x015154 thru 0x01f5ad (0xa45a bytes) 000096 1790101558.867004 write 0x03c10f thru 0x03d7a6 (0x1698 bytes) truncating to largest ever: 0x36a1c 000097 1790101558.870947 trunc from 0x03d7a7 to 0x036a1b (0x6d8b bytes) 000098 1790101558.882483 trunc from 0x02d061 to 0x036a1b (0x99bb bytes) 000099 1790101558.888008 mapwrite 0x00044b thru 0x000512 (0xc8 bytes) 000100 1790101558.901886 write 0x02a3c4 thru 0x02f99c (0x55d9 bytes) 000102 1790101558.916841 MIRROR_RESYNC 2 mirrors 000103 1790101559.602706 trunc from 0x027dfd to 0x02b7ef (0x39f3 bytes) 000104 1790101559.634170 mapread 0x023325 thru 0x02bd5c (0x8a38 bytes) 000105 1790101559.653406 fallocate 0x023d31 thru 0x02483b (0xb0b bytes) 000106 1790101559.657744 fallocate 0x0251a8 thru 0x027853 (0x26ac bytes) 000107 1790101559.661878 mapwrite 0x0043fa thru 0x00a74b (0x6352 bytes) 000108 1790101559.684994 mapread 0x00e562 thru 0x01b58b (0xd02a bytes) 000110 1790101559.701833 MIRROR_RESYNC 2 mirrors 000111 1790101560.415762 trunc from 0x01f933 to 0x022796 (0x2e64 bytes) 000112 1790101560.467889 mapwrite 0x005c5c thru 0x015052 (0xf3f7 bytes) 000113 1790101560.517069 trunc from 0x0347ac to 0x036a1b (0x2270 bytes) 000114 1790101560.520951 trunc from 0x0347f2 to 0x035ea8 (0x16b7 bytes) 000116 1790101560.524818 mapread 0x012b31 thru 0x015e84 (0x3354 bytes) 000117 1790101560.529027 MIRROR_RESYNC 2 mirrors 000118 1790101561.248098 fallocate 0x00a44c thru 0x018798 (0xe34d bytes) 000119 1790101561.313178 mapread 0x0328d3 thru 0x036a1b (0x4149 bytes) 000120 1790101561.346871 write 0x02dab4 thru 0x035961 (0x7eae bytes) 000121 1790101561.349382 mapread 0x01697f thru 0x0255aa (0xec2c bytes) 000122 1790101561.367692 trunc from 0x036a1c to 0x01ea6d (0x17fae bytes) 000123 1790101561.439205 trunc from 0x010912 to 0x016818 (0x5f07 bytes) 000124 1790101561.446706 write 0x030861 thru 0x033ea6 (0x3646 bytes) 000125 1790101561.447787 mapwrite 0x0157dc thru 0x01c328 (0x6b4d bytes) 000126 1790101561.491396 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33ea7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ea7): Mirror 2: 0x10591c18 (stale) CRC-32 checksum value for chunk [0, 0x33ea7): Mirror 1: 0xe735b2b4 Mirror 3: 0xe735b2b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ea7: skipped 000127 1790101562.284950 trunc from 0x033ea7 to 0x02c3ba (0x7aec bytes) 000128 1790101562.383296 read 0x02b9b8 thru 0x02c3ba (0xa03 bytes) 000129 1790101562.390970 trunc from 0x01c743 to 0x023d03 (0x75c1 bytes) 000130 1790101562.397574 fallocate 0x0243c9 thru 0x02c0ae (0x7ce6 bytes) 000131 1790101562.415212 mapwrite 0x00c0e2 thru 0x019e4c (0xdd6b bytes) 000132 1790101562.467050 fallocate 0x021b7c thru 0x03028d (0xe712 bytes) 000133 1790101562.476416 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 2: 0xa59e852a (stale) Mirror 3: 0xc764f7a8 (stale) CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 1: 0x46806b11 Mirror 4: 0x46806b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3bb: skipped 000134 1790101563.485404 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 2: 0xa59e852a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 2: 0xa59e852a (stale) CRC-32 checksum value for chunk [0, 0x2c3bb): Mirror 1: 0x46806b11 Mirror 4: 0x46806b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3bb: skipped 000135 1790101563.884053 mapwrite 0x009e56 thru 0x00ce0a (0x2fb5 bytes) 000136 1790101563.972934 write 0x03d206 thru 0x03f9bd (0x27b8 bytes) 000137 1790101563.978470 trunc from 0x03f9be to 0x016be1 (0x28ddc bytes) 000139 1790101564.014633 write 0x00df59 thru 0x00e43a (0x4e2 bytes) 000140 1790101564.015566 mapwrite 0x0230a5 thru 0x032111 (0xf06d bytes) 000141 1790101564.069103 fallocate 0x013b70 thru 0x01de5d (0xa2ee bytes) 000142 1790101564.073707 trunc from 0x032112 to 0x001cb7 (0x3045a bytes) 000143 1790101564.086923 mapread 0x00184d thru 0x001cb7 (0x46b bytes) 000144 1790101564.092296 punch from 0x000a80 to 0x001cb7 (0x1238 bytes) 000145 1790101564.097342 write 0x0284dc thru 0x037c72 (0xf797 bytes) 000146 1790101564.099547 trunc from 0x0175c3 to 0x026d55 (0xf793 bytes) 000147 1790101564.104830 mapread 0x02b7a5 thru 0x02e04c (0x28a8 bytes) 000148 1790101564.105926 fallocate 0x006137 thru 0x014827 (0xe6f1 bytes) 000149 1790101564.109929 mapread 0x0127f9 thru 0x0171f3 (0x49fb bytes) 000150 1790101564.117250 mapwrite 0x02dc51 thru 0x03adf1 (0xd1a1 bytes) 000151 1790101564.153895 mapwrite 0x03c9f2 thru 0x03ee59 (0x2468 bytes) 000152 1790101564.184295 fallocate 0x0221b7 thru 0x02d374 (0xb1be bytes) 000153 1790101564.188832 mapread 0x03cd20 thru 0x03e9e1 (0x1cc2 bytes) 000154 1790101564.190504 punch from 0x019c04 to 0x01ed09 (0x5106 bytes) 000156 1790101564.195244 trunc from 0x03ee5a to 0x009301 (0x35b58 bytes) 000157 1790101564.212462 write 0x000ab1 thru 0x007374 (0x68c4 bytes) 000159 1790101564.226259 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x9302): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9302): Mirror 2: 0x71cad3ba (stale) Mirror 4: 0x71cad3ba (stale) CRC-32 checksum value for chunk [0, 0x9302): Mirror 1: 0xad421a3a Mirror 5: 0xad421a3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9302: skipped 000160 1790101564.806477 trunc from 0x009302 to 0x02db7c (0x2487b bytes) 000161 1790101564.856788 mapread 0x0229fb thru 0x02db7c (0xb182 bytes) 000163 1790101564.876948 fallocate 0x0069c2 thru 0x00e430 (0x7a6f bytes) 000164 1790101564.881460 trunc from 0x02db7d to 0x021866 (0xc316 bytes) 000165 1790101564.897169 write 0x02bb96 thru 0x02c8cb (0xd36 bytes) 000166 1790101564.898271 trunc from 0x0272ec to 0x02c8cb (0x55e0 bytes) 000167 1790101564.907551 fallocate 0x00b85c thru 0x00c3ea (0xb8f bytes) 000168 1790101564.911216 mapread 0x021623 thru 0x025d00 (0x46de bytes) 000169 1790101564.922401 fallocate 0x005b4c thru 0x006e41 (0x12f6 bytes) 000170 1790101564.926397 trunc from 0x0210a7 to 0x021829 (0x783 bytes) 000172 1790101564.933839 fallocate 0x0080c6 thru 0x00bd84 (0x3cbf bytes) 000174 1790101564.939127 fallocate 0x019c35 thru 0x029073 (0xf43f bytes) 000175 1790101564.953808 read 0x024f5f thru 0x02a169 (0x520b bytes) 000176 1790101564.956428 trunc from 0x02c8cc to 0x00c88c (0x2003f bytes) 000178 1790101564.976800 MIRROR_RESYNC 4 mirrors 000179 1790101565.884474 MIRROR_RESYNC 4 mirrors 000180 1790101566.281817 mapwrite 0x01fd30 thru 0x02eee4 (0xf1b5 bytes) 000181 1790101566.453712 write 0x0109e6 thru 0x0176a3 (0x6cbe bytes) 000182 1790101566.466246 read 0x020d9a thru 0x027393 (0x65fa bytes) 000183 1790101566.467115 write 0x01eabd thru 0x02be7a (0xd3be bytes) 000184 1790101566.469422 trunc from 0x02eee5 to 0x027eb9 (0x702b bytes) 000185 1790101566.491922 trunc from 0x01b015 to 0x025960 (0xa94c bytes) 000186 1790101566.523613 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27eba): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27eba): Mirror 2: 0x8d771a90 (stale) Mirror 4: 0x8d771a90 (stale) Mirror 5: 0x8d771a90 (stale) CRC-32 checksum value for chunk [0, 0x27eba): Mirror 1: 0x4688ded5 Mirror 6: 0x4688ded5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27eba: skipped 000187 1790101567.750416 write 0x0235e3 thru 0x031c05 (0xe623 bytes) 000188 1790101567.834586 punch from 0x014dd7 to 0x01e5b9 (0x97e3 bytes) 000189 1790101567.850408 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31c06): Mirror 2: 0x5e4d61f6 (stale) Mirror 4: 0x5e4d61f6 (stale) Mirror 5: 0x5e4d61f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c06): Mirror 2: 0x5e4d61f6 (stale) Mirror 4: 0x5e4d61f6 (stale) Mirror 5: 0x5e4d61f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c06: skipped 000190 1790101568.480463 mapwrite 0x004ace thru 0x00e184 (0x96b7 bytes) 000191 1790101568.523203 trunc from 0x031c06 to 0x001741 (0x304c4 bytes) fallocating to largest ever: 0x40000 000192 1790101568.545801 fallocate 0x03e2af thru 0x03ffff (0x1d51 bytes) 000193 1790101568.551038 read 0x000d1a thru 0x001741 (0xa28 bytes) 000194 1790101568.555956 fallocate 0x0327f4 thru 0x03ffff (0xd80c bytes) 000195 1790101568.561865 mapwrite 0x03f6ad thru 0x03ffff (0x953 bytes) 000196 1790101568.600051 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x911ec9f6 (stale) Mirror 4: 0x911ec9f6 (stale) Mirror 5: 0x911ec9f6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde79e2af Mirror 6: 0xde79e2af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000197 1790101569.961524 trunc from 0x040000 to 0x03422e (0xbdd1 bytes) 000198 1790101570.090744 fallocate 0x03a652 thru 0x03ffff (0x59ae bytes) 000199 1790101570.097166 read 0x016289 thru 0x01beee (0x5c66 bytes) 000200 1790101570.131881 mapwrite 0x015dc4 thru 0x0175b9 (0x17f6 bytes) 000201 1790101570.180613 read 0x0120e4 thru 0x0166d6 (0x45f3 bytes) 000202 1790101570.223371 write 0x0236f1 thru 0x02fa2a (0xc33a bytes) 000203 1790101570.226908 punch from 0x000d57 to 0x003903 (0x2bad bytes) 000204 1790101570.242818 fallocate 0x00f7e0 thru 0x01cd99 (0xd5ba bytes) 000205 1790101570.260302 punch from 0x028871 to 0x032e0f (0xa59f bytes) 000206 1790101570.297752 read 0x019414 thru 0x021c8f (0x887c bytes) 000207 1790101570.298441 mapwrite 0x01d68e thru 0x02d4bf (0xfe32 bytes) 000208 1790101570.357082 mapread 0x02ec29 thru 0x02f416 (0x7ee bytes) 000209 1790101570.357945 mapread 0x006131 thru 0x013707 (0xd5d7 bytes) 000212 1790101570.359694 fallocate 0x036abe thru 0x039503 (0x2a46 bytes) 000214 1790101570.371777 punch from 0x003f0d to 0x007ed6 (0x3fca bytes) 000215 1790101570.377253 write 0x01605c thru 0x01b377 (0x531c bytes) 000216 1790101570.379700 mapread 0x0171a3 thru 0x018850 (0x16ae bytes) 000217 1790101570.383307 trunc from 0x01675d to 0x01f859 (0x90fd bytes) 000218 1790101570.440171 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39504): Mirror 2: 0x9e1af12b (stale) Mirror 5: 0x9e1af12b (stale) Mirror 6: 0xb5a51f28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39504): Mirror 2: 0x9e1af12b (stale) Mirror 5: 0x9e1af12b (stale) Mirror 6: 0xb5a51f28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39504: skipped 000219 1790101570.925255 trunc from 0x020bd8 to 0x0283e6 (0x780f bytes) 000220 1790101570.946847 fallocate 0x00a1c5 thru 0x017b94 (0xd9d0 bytes) 000221 1790101570.956372 trunc from 0x039504 to 0x02e719 (0xadea bytes) 000222 1790101570.995597 write 0x01aef8 thru 0x024a55 (0x9b5e bytes) 000223 1790101571.020582 read 0x01abf3 thru 0x02a2a9 (0xf6b7 bytes) 000224 1790101571.031362 punch from 0x01afb6 to 0x026bbf (0xbc0a bytes) 000225 1790101571.060212 trunc from 0x019616 to 0x028895 (0xf280 bytes) 000227 1790101571.071821 write 0x01845b thru 0x0225e7 (0xa18d bytes) 000229 1790101571.084289 mapread 0x02643f thru 0x02779c (0x135e bytes) 000230 1790101571.104737 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e71a): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e71a): Mirror 2: 0x67593c9a (stale) Mirror 5: 0x67593c9a (stale) Mirror 6: 0x5f0dd3cb (stale) CRC-32 checksum value for chunk [0, 0x2e71a): Mirror 1: 0xef0b1518 Mirror 7: 0xef0b1518 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e71a: skipped 000231 1790101572.364168 mapwrite 0x001157 thru 0x00ef44 (0xddee bytes) 000233 1790101572.519143 read 0x005aa9 thru 0x00a0c2 (0x461a bytes) 000235 1790101572.526171 punch from 0x0013cf to 0x006c10 (0x5842 bytes) 000236 1790101572.533736 punch from 0x020094 to 0x02225d (0x21ca bytes) 000237 1790101572.539809 mapread 0x025d16 thru 0x02a079 (0x4364 bytes) 000238 1790101572.552713 punch from 0x003e05 to 0x009423 (0x561f bytes) 000239 1790101572.563715 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e71a): Mirror 2: 0x67593c9a (stale) Mirror 5: 0x67593c9a (stale) Mirror 6: 0x5f0dd3cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e71a): Mirror 2: 0x67593c9a (stale) Mirror 5: 0x67593c9a (stale) Mirror 6: 0x5f0dd3cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e71a: skipped 000242 1790101573.039249 trunc from 0x02e71a to 0x00f3b5 (0x1f364 bytes) 000243 1790101573.084683 trunc from 0x00f3b6 to 0x02ac09 (0x1b854 bytes) 000244 1790101573.100694 trunc from 0x01e69f to 0x02151c (0x2e7e bytes) 000245 1790101573.104878 write 0x0306c4 thru 0x036501 (0x5e3e bytes) 000246 1790101573.106767 mapwrite 0x020abc thru 0x02a692 (0x9bd7 bytes) 000247 1790101573.143210 mapread 0x020e74 thru 0x021a47 (0xbd4 bytes) 000248 1790101573.144580 trunc from 0x036502 to 0x0312c3 (0x523e bytes) 000249 1790101573.163044 fallocate 0x00a794 thru 0x00f7ed (0x505a bytes) 000250 1790101573.172242 read 0x01341d thru 0x0208ce (0xd4b2 bytes) 000251 1790101573.180646 mapread 0x0127c8 thru 0x01c89c (0xa0d5 bytes) 000252 1790101573.189841 write 0x03be3c thru 0x03ffff (0x41c4 bytes) 000254 1790101573.191765 mapread 0x022822 thru 0x031eed (0xf6cc bytes) 000255 1790101573.205162 MIRROR_RESYNC 4 mirrors 000256 1790101575.120163 read 0x017e5e thru 0x01e641 (0x67e4 bytes) 000257 1790101575.139030 trunc from 0x040000 to 0x00c811 (0x337ee bytes) 000258 1790101575.185146 trunc from 0x00b84e to 0x00c811 (0xfc4 bytes) 000259 1790101575.191263 write 0x01803f thru 0x01eeab (0x6e6d bytes) 000260 1790101575.193138 read 0x00a0ce thru 0x00c75e (0x2691 bytes) 000261 1790101575.212225 punch from 0x00249e to 0x009c5e (0x77c1 bytes) 000262 1790101575.219511 fallocate 0x02ea54 thru 0x031cf0 (0x329d bytes) 000263 1790101575.227043 trunc from 0x01eeac to 0x00502d (0x19e7e bytes) 000265 1790101575.269698 write 0x029c1c thru 0x031236 (0x761b bytes) 000266 1790101575.277207 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31237): Mirror 5: 0x378eceb1 (stale) Mirror 6: 0x378eceb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31237): Mirror 5: 0x378eceb1 (stale) Mirror 6: 0x378eceb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31237: skipped 000267 1790101575.750604 write 0x016902 thru 0x024eda (0xe5d9 bytes) 000268 1790101575.767102 write 0x006336 thru 0x012867 (0xc532 bytes) 000269 1790101575.781806 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31237): Mirror 6: 0x378eceb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31237): Mirror 6: 0x378eceb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31237: skipped 000270 1790101576.145091 mapwrite 0x00dcf1 thru 0x0138e6 (0x5bf6 bytes) 000271 1790101576.174272 trunc from 0x023fa9 to 0x02876f (0x47c7 bytes) 000272 1790101576.180234 mapread 0x02dcd2 thru 0x031236 (0x3565 bytes) 000273 1790101576.190206 fallocate 0x0257d9 thru 0x0265b7 (0xddf bytes) 000274 1790101576.203918 write 0x005292 thru 0x012be3 (0xd952 bytes) 000275 1790101576.218278 punch from 0x030a3c to 0x031236 (0x7fb bytes) 000277 1790101576.226919 mapread 0x01220c thru 0x01c0b0 (0x9ea5 bytes) 000278 1790101576.246568 write 0x01aff9 thru 0x026a52 (0xba5a bytes) 000279 1790101576.249706 mapwrite 0x023897 thru 0x02db1c (0xa286 bytes) 000280 1790101576.270698 trunc from 0x031237 to 0x00c96a (0x248cc bytes) 000281 1790101576.295381 read 0x004ef7 thru 0x00814f (0x3259 bytes) 000283 1790101576.303673 mapread 0x001853 thru 0x00749f (0x5c4d bytes) 000284 1790101576.314032 mapread 0x0092fe thru 0x00bc19 (0x291c bytes) 000285 1790101576.316684 read 0x00ac48 thru 0x00c96a (0x1d23 bytes) 000286 1790101576.317479 trunc from 0x00c96b to 0x00287e (0xa0ec bytes) 000287 1790101576.337146 punch from 0x0025ec to 0x00287e (0x293 bytes) 000288 1790101576.344092 read 0x00190f thru 0x00287e (0xf70 bytes) 000289 1790101576.353745 fallocate 0x03ad99 thru 0x03ffff (0x5267 bytes) 000290 1790101576.359066 trunc from 0x00287f to 0x01db9c (0x1b31e bytes) 000291 1790101576.375548 fallocate 0x027a40 thru 0x0299c8 (0x1f89 bytes) 000292 1790101576.386400 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) 000293 1790101576.543568 read 0x00e1f7 thru 0x019891 (0xb69b bytes) 000294 1790101576.562361 trunc from 0x0299c9 to 0x001a88 (0x27f40 bytes) 000295 1790101576.625096 read 0x00102e thru 0x001a88 (0xa5b bytes) 000296 1790101576.641130 mapwrite 0x0124c4 thru 0x01bb47 (0x9684 bytes) 000297 1790101576.692379 trunc from 0x01bb48 to 0x0190bc (0x2a8b bytes) 000298 1790101576.710049 mapwrite 0x01c874 thru 0x026f34 (0xa6c1 bytes) 000299 1790101576.752821 punch from 0x0015b0 to 0x004319 (0x2d6a bytes) 000300 1790101576.758794 write 0x016653 thru 0x01ebc6 (0x8574 bytes) 000301 1790101576.760842 punch from 0x007d94 to 0x00c9dd (0x4c4a bytes) 000302 1790101576.766056 mapread 0x0009f2 thru 0x00a701 (0x9d10 bytes) 000303 1790101576.802101 punch from 0x00e2cf to 0x010662 (0x2394 bytes) 000305 1790101576.808979 trunc from 0x01da38 to 0x026f34 (0x94fd bytes) 000306 1790101576.823752 trunc from 0x00a33e to 0x011a62 (0x7725 bytes) 000308 1790101576.829667 mapwrite 0x008caa thru 0x008e6a (0x1c1 bytes) 000309 1790101576.841539 trunc from 0x00c97e to 0x01c3e4 (0xfa67 bytes) 000310 1790101576.846682 punch from 0x00a1dc to 0x013440 (0x9265 bytes) 000311 1790101576.856021 MIRROR_EXTEND to 2 mirrors 000312 1790101577.559965 MIRROR_EXTEND to 3 mirrors 000313 1790101578.114612 mapread 0x0215f0 thru 0x026f34 (0x5945 bytes) 000314 1790101578.138154 trunc from 0x007ad6 to 0x013c7e (0xc1a9 bytes) 000315 1790101578.174597 mapwrite 0x01d797 thru 0x027c34 (0xa49e bytes) 000317 1790101578.229026 write 0x019170 thru 0x01a216 (0x10a7 bytes) 000319 1790101578.231790 trunc from 0x027c35 to 0x01482d (0x13407 bytes) 000320 1790101578.257924 fallocate 0x010efc thru 0x018f71 (0x8076 bytes) 000321 1790101578.264460 punch from 0x00f054 to 0x012ab6 (0x3a63 bytes) 000322 1790101578.269587 read 0x003182 thru 0x009322 (0x61a1 bytes) 000323 1790101578.278465 trunc from 0x0002da to 0x004e62 (0x4b89 bytes) 000324 1790101578.283525 mapread 0x008fd9 thru 0x00dffe (0x5026 bytes) 000326 1790101578.293714 punch from 0x000356 to 0x00a29a (0x9f45 bytes) 000327 1790101578.304740 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1482e): Mirror 2: 0xf086fb45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1482e): Mirror 2: 0xf086fb45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1482e: skipped 000328 1790101578.566292 fallocate 0x02e801 thru 0x034df9 (0x65f9 bytes) 000329 1790101578.570790 write 0x002a43 thru 0x00bf7c (0x953a bytes) 000330 1790101578.589247 mapwrite 0x034911 thru 0x03d173 (0x8863 bytes) 000331 1790101578.660483 write 0x003de2 thru 0x009ed1 (0x60f0 bytes) 000332 1790101578.661584 mapread 0x00ec8d thru 0x0197ab (0xab1f bytes) 000333 1790101578.684541 MIRROR_RESYNC 2 mirrors 000335 1790101579.345442 MIRROR_EXTEND to 3 mirrors 000336 1790101579.895880 read 0x037d10 thru 0x03d173 (0x5464 bytes) 000337 1790101579.906755 read 0x032206 thru 0x038444 (0x623f bytes) 000339 1790101579.918014 mapread 0x00c480 thru 0x01ad77 (0xe8f8 bytes) 000340 1790101579.918856 mapwrite 0x006e08 thru 0x00e222 (0x741b bytes) 000342 1790101580.027463 write 0x02fc19 thru 0x032c90 (0x3078 bytes) 000343 1790101580.063899 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d174): Mirror 3: 0xfc933008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d174): Mirror 3: 0xfc933008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d174: skipped 000344 1790101580.369266 mapread 0x01958c thru 0x01d2c5 (0x3d3a bytes) 000345 1790101580.396515 punch from 0x01988c to 0x022c27 (0x939c bytes) 000346 1790101580.401728 mapwrite 0x03ff38 thru 0x03ffff (0xc8 bytes) 000347 1790101580.439063 mapread 0x021ba3 thru 0x03054a (0xe9a8 bytes) 000348 1790101580.463711 mapwrite 0x018655 thru 0x02793f (0xf2eb bytes) 000349 1790101580.489313 mapread 0x021441 thru 0x022352 (0xf12 bytes) 000350 1790101580.493491 MIRROR_RESYNC 2 mirrors 000351 1790101581.386548 read 0x008532 thru 0x01389c (0xb36b bytes) 000352 1790101581.431234 MIRROR_RESYNC 2 mirrors 000353 1790101582.093785 MIRROR_RESYNC 2 mirrors 000354 1790101582.550059 write 0x01cb24 thru 0x01ff81 (0x345e bytes) 000355 1790101582.646736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfdde5c28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85644e1d Mirror 4: 0x85644e1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000356 1790101584.048418 trunc from 0x0107fa to 0x016cbd (0x64c4 bytes) 000357 1790101584.102221 read 0x03c796 thru 0x03ffff (0x386a bytes) 000358 1790101584.111277 mapread 0x025646 thru 0x0320c3 (0xca7e bytes) 000359 1790101584.148930 mapwrite 0x012a75 thru 0x0174d6 (0x4a62 bytes) 000360 1790101584.176281 trunc from 0x01cc2b to 0x02104c (0x4422 bytes) 000361 1790101584.180425 trunc from 0x02444a to 0x0310a5 (0xcc5c bytes) 000362 1790101584.185287 trunc from 0x004f63 to 0x007652 (0x26f0 bytes) 000365 1790101584.190792 write 0x03bf9d thru 0x03ffff (0x4063 bytes) 000366 1790101584.209032 mapwrite 0x00c0db thru 0x00d2f4 (0x121a bytes) 000367 1790101584.232251 mapwrite 0x00a22e thru 0x013956 (0x9729 bytes) 000368 1790101584.275510 fallocate 0x02fe15 thru 0x03fd81 (0xff6d bytes) 000369 1790101584.292690 mapread 0x02705c thru 0x030403 (0x93a8 bytes) 000370 1790101584.324967 punch from 0x03457d to 0x039dc7 (0x584b bytes) 000371 1790101584.338499 punch from 0x006574 to 0x0080b1 (0x1b3e bytes) 000372 1790101584.346587 trunc from 0x040000 to 0x00e240 (0x31dbf bytes) 000373 1790101584.377526 mapwrite 0x011d9d thru 0x01a805 (0x8a69 bytes) 000374 1790101584.494676 fallocate 0x01ae14 thru 0x0222ae (0x749b bytes) 000375 1790101584.515226 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a806): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a806): Mirror 3: 0xe720274e (stale) Mirror 4: 0xe720274e (stale) CRC-32 checksum value for chunk [0, 0x1a806): Mirror 1: 0x52d5087d Mirror 5: 0x52d5087d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a806: skipped 000376 1790101586.236244 mapwrite 0x031805 thru 0x0343b2 (0x2bae bytes) 000377 1790101586.450650 mapwrite 0x03918b thru 0x03ffff (0x6e75 bytes) 000378 1790101586.521027 mapwrite 0x02c918 thru 0x02d340 (0xa29 bytes) 000379 1790101586.533135 trunc from 0x0187b9 to 0x027808 (0xf050 bytes) 000381 1790101586.539235 mapwrite 0x0208be thru 0x024409 (0x3b4c bytes) 000382 1790101586.587028 fallocate 0x02a811 thru 0x02f5dd (0x4dcd bytes) 000383 1790101586.592246 punch from 0x000b4b to 0x00cb98 (0xc04e bytes) 000384 1790101586.597472 write 0x002948 thru 0x00e5ab (0xbc64 bytes) 000385 1790101586.607204 trunc from 0x01a44a to 0x023764 (0x931b bytes) 000386 1790101586.613202 trunc from 0x040000 to 0x02d772 (0x1288d bytes) 000387 1790101586.633556 mapread 0x01d1f5 thru 0x02ae7a (0xdc86 bytes) 000388 1790101586.646895 trunc from 0x025ad2 to 0x02d772 (0x7ca1 bytes) 000389 1790101586.652948 trunc from 0x0071b3 to 0x0170d3 (0xff21 bytes) 000390 1790101586.676728 read 0x01c589 thru 0x02ab12 (0xe58a bytes) 000391 1790101586.690652 MIRROR_RESYNC 4 mirrors 000392 1790101588.630261 read 0x02b45b thru 0x02d772 (0x2318 bytes) 000393 1790101588.640800 mapread 0x00a31d thru 0x00cec5 (0x2ba9 bytes) 000394 1790101588.659247 punch from 0x011676 to 0x01e564 (0xceef bytes) 000395 1790101588.692177 write 0x008e1b thru 0x01490b (0xbaf1 bytes) 000396 1790101588.702907 trunc from 0x02ab9f to 0x02d772 (0x2bd4 bytes) 000397 1790101588.708090 punch from 0x02d2d7 to 0x02d772 (0x49c bytes) 000398 1790101588.712291 read 0x000852 thru 0x00459c (0x3d4b bytes) 000399 1790101588.723191 mapwrite 0x037b4b thru 0x03ffff (0x84b5 bytes) 000400 1790101588.773108 trunc from 0x028488 to 0x037952 (0xf4cb bytes) 000401 1790101588.779100 mapread 0x0148b4 thru 0x018bdb (0x4328 bytes) 000402 1790101588.784130 MIRROR_RESYNC 4 mirrors 000403 1790101590.066518 mapread 0x00e4b9 thru 0x014bf9 (0x6741 bytes) 000404 1790101590.078687 trunc from 0x03bf85 to 0x03ed96 (0x2e12 bytes) 000405 1790101590.104144 fallocate 0x010132 thru 0x0109a3 (0x872 bytes) 000406 1790101590.108386 mapread 0x003371 thru 0x010cb4 (0xd944 bytes) 000407 1790101590.118117 fallocate 0x01e982 thru 0x021f7a (0x35f9 bytes) 000408 1790101590.126990 MIRROR_RESYNC 4 mirrors 000409 1790101591.428676 fallocate 0x009628 thru 0x009e47 (0x820 bytes) 000410 1790101591.459826 mapwrite 0x0263f5 thru 0x0266cd (0x2d9 bytes) 000411 1790101591.487635 read 0x00fcfc thru 0x01873d (0x8a42 bytes) 000412 1790101591.498412 mapwrite 0x01bc0a thru 0x02b06f (0xf466 bytes) 000413 1790101591.560011 read 0x000eab thru 0x008c5c (0x7db2 bytes) 000414 1790101591.578794 MIRROR_RESYNC 4 mirrors 000415 1790101593.072824 punch from 0x017ef2 to 0x01d7bb (0x58ca bytes) 000416 1790101593.128213 MIRROR_RESYNC 4 mirrors 000418 1790101595.250905 read 0x00e1bb thru 0x01b583 (0xd3c9 bytes) 000419 1790101595.260908 write 0x0292e9 thru 0x033466 (0xa17e bytes) 000420 1790101595.365221 fallocate 0x0115e0 thru 0x015fa6 (0x49c7 bytes) 000421 1790101595.385063 fallocate 0x02512f thru 0x026afb (0x19cd bytes) 000422 1790101595.404700 trunc from 0x03f2ee to 0x03ffff (0xd12 bytes) 000423 1790101595.433500 trunc from 0x031afc to 0x031b6a (0x6f bytes) 000424 1790101595.464561 mapread 0x004afa thru 0x005492 (0x999 bytes) 000425 1790101595.477656 punch from 0x00520c to 0x00a2f5 (0x50ea bytes) 000426 1790101595.490156 punch from 0x0124a4 to 0x01c55c (0xa0b9 bytes) 000427 1790101595.509614 write 0x015306 thru 0x0182a5 (0x2fa0 bytes) 000428 1790101595.538932 mapwrite 0x01c2e4 thru 0x01e2d1 (0x1fee bytes) 000429 1790101595.643595 trunc from 0x031d7c to 0x0331a1 (0x1426 bytes) 000430 1790101595.669244 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39307c7 (stale) Mirror 4: 0x39307c7 (stale) Mirror 5: 0x39307c7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x442e3749 Mirror 6: 0x442e3749 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000431 1790101597.160420 read 0x02f8ba thru 0x03da79 (0xe1c0 bytes) 000434 1790101597.196704 punch from 0x012a1b to 0x02097f (0xdf65 bytes) 000436 1790101597.254426 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39307c7 (stale) Mirror 4: 0x39307c7 (stale) Mirror 5: 0x39307c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39307c7 (stale) Mirror 4: 0x39307c7 (stale) Mirror 5: 0x39307c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000437 1790101597.752843 trunc from 0x01df7e to 0x0217b4 (0x3837 bytes) 000438 1790101597.756783 fallocate 0x01d6fd thru 0x02d62b (0xff2f bytes) 000439 1790101597.762010 fallocate 0x014ae4 thru 0x01acd0 (0x61ed bytes) 000440 1790101597.765809 mapwrite 0x026610 thru 0x030e4a (0xa83b bytes) 000441 1790101597.826692 trunc from 0x040000 to 0x0286ed (0x17912 bytes) 000442 1790101597.861070 trunc from 0x0286ee to 0x03a993 (0x122a6 bytes) 000444 1790101597.905147 fallocate 0x0274ad thru 0x02b927 (0x447b bytes) 000445 1790101597.916263 mapread 0x02877f thru 0x03339c (0xac1e bytes) 000446 1790101597.963598 trunc from 0x03728a to 0x03a993 (0x370a bytes) 000448 1790101597.969357 punch from 0x023e8f to 0x029c97 (0x5e09 bytes) 000449 1790101597.977149 mapwrite 0x00cde5 thru 0x017880 (0xaa9c bytes) 000450 1790101598.073022 trunc from 0x03a994 to 0x013f2d (0x26a66 bytes) 000452 1790101598.106979 trunc from 0x005fad to 0x0096b6 (0x370a bytes) 000453 1790101598.120197 punch from 0x000da9 to 0x00f3fc (0xe654 bytes) 000454 1790101598.130161 trunc from 0x0020bd to 0x006d66 (0x4caa bytes) 000456 1790101598.163530 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13f2e): Mirror 4: 0x401620e (stale) Mirror 5: 0x401620e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13f2e): Mirror 4: 0x401620e (stale) Mirror 5: 0x401620e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13f2e: skipped 000457 1790101598.748503 trunc from 0x013f2e to 0x02066a (0xc73d bytes) 000458 1790101598.789879 write 0x007235 thru 0x00c6d6 (0x54a2 bytes) 000459 1790101598.823572 trunc from 0x02066b to 0x03480e (0x141a4 bytes) 000460 1790101598.889929 read 0x00b246 thru 0x01262f (0x73ea bytes) 000461 1790101598.901917 mapwrite 0x022013 thru 0x023a11 (0x19ff bytes) 000462 1790101598.974626 mapwrite 0x030208 thru 0x03dddc (0xdbd5 bytes) 000463 1790101599.088420 punch from 0x009312 to 0x00f559 (0x6248 bytes) 000464 1790101599.096511 mapread 0x03b381 thru 0x03dddc (0x2a5c bytes) 000465 1790101599.100721 mapwrite 0x039eaf thru 0x03ffff (0x6151 bytes) 000466 1790101599.161143 fallocate 0x019cbc thru 0x01a108 (0x44d bytes) 000467 1790101599.169163 read 0x0250df thru 0x032c89 (0xdbab bytes) 000468 1790101599.183391 punch from 0x026d88 to 0x029ff9 (0x3272 bytes) 000469 1790101599.190529 fallocate 0x03ff73 thru 0x03ffff (0x8d bytes) 000470 1790101599.196344 trunc from 0x040000 to 0x02ba11 (0x145ee bytes) 000471 1790101599.222787 read 0x021950 thru 0x02acf9 (0x93aa bytes) 000472 1790101599.229701 mapwrite 0x019422 thru 0x022cd8 (0x98b7 bytes) 000473 1790101599.280401 mapwrite 0x01abbb thru 0x025ffe (0xb444 bytes) 000474 1790101599.325534 punch from 0x016dfa to 0x016e60 (0x67 bytes) 000475 1790101599.338086 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba12): Mirror 4: 0x9e016e76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba12): Mirror 4: 0x9e016e76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba12: skipped 000476 1790101599.804180 fallocate 0x027677 thru 0x028191 (0xb1b bytes) 000477 1790101599.837397 punch from 0x009374 to 0x015a6e (0xc6fb bytes) 000478 1790101599.848217 mapread 0x00c0f6 thru 0x019d6c (0xdc77 bytes) 000479 1790101599.910215 write 0x03cd14 thru 0x03d15d (0x44a bytes) 000480 1790101599.919017 trunc from 0x03d15e to 0x031d88 (0xb3d5 bytes) 000481 1790101599.963257 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31d89): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d89): Mirror 4: 0x241d05af (stale) CRC-32 checksum value for chunk [0, 0x31d89): Mirror 1: 0xe937488b Mirror 5: 0xe937488b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d89: skipped 000482 1790101601.679042 trunc from 0x031d89 to 0x03a85d (0x8ad5 bytes) 000483 1790101601.777571 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a85e): Mirror 5: 0xdf409e8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a85e): Mirror 5: 0xdf409e8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a85e: skipped 000484 1790101602.244253 fallocate 0x01be87 thru 0x02a271 (0xe3eb bytes) 000485 1790101602.255002 mapread 0x028173 thru 0x036a6b (0xe8f9 bytes) 000486 1790101602.306914 fallocate 0x0209d3 thru 0x0212a2 (0x8d0 bytes) 000487 1790101602.315695 trunc from 0x00cdf4 to 0x00e353 (0x1560 bytes) 000488 1790101602.333991 write 0x03715f thru 0x03ffff (0x8ea1 bytes) 000489 1790101602.346671 trunc from 0x031489 to 0x038adf (0x7657 bytes) 000490 1790101602.374582 mapread 0x025ed9 thru 0x0355e3 (0xf70b bytes) 000491 1790101602.425986 trunc from 0x024641 to 0x025a4e (0x140e bytes) 000492 1790101602.430867 fallocate 0x021936 thru 0x02a385 (0x8a50 bytes) 000493 1790101602.474964 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xba973809 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe276ea9c Mirror 6: 0xe276ea9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1790101603.940307 punch from 0x0244b2 to 0x02a927 (0x6476 bytes) 000496 1790101604.015266 read 0x034b42 thru 0x03ff17 (0xb3d6 bytes) 000497 1790101604.049611 mapread 0x0147c6 thru 0x01ba42 (0x727d bytes) 000498 1790101604.072756 trunc from 0x01150e to 0x012dda (0x18cd bytes) 000499 1790101604.076193 trunc from 0x040000 to 0x0031da (0x3ce25 bytes) 000501 1790101604.109055 read 0x002540 thru 0x0031da (0xc9b bytes) 000502 1790101604.129019 mapwrite 0x0341e1 thru 0x0350bc (0xedc bytes) 000503 1790101604.233265 read 0x01e782 thru 0x02181b (0x309a bytes) 000504 1790101604.235941 write 0x00a411 thru 0x012716 (0x8306 bytes) 000505 1790101604.245733 mapread 0x02a173 thru 0x0330d6 (0x8f64 bytes) 000507 1790101604.256486 trunc from 0x002524 to 0x00660c (0x40e9 bytes) 000508 1790101604.271086 read 0x010c3b thru 0x019c0f (0x8fd5 bytes) 000509 1790101604.271927 fallocate 0x01b74f thru 0x027562 (0xbe14 bytes) 000510 1790101604.285939 trunc from 0x0350bd to 0x03a236 (0x517a bytes) 000511 1790101604.314751 fallocate 0x00917b thru 0x010e4b (0x7cd1 bytes) 000512 1790101604.329118 trunc from 0x03a237 to 0x03e232 (0x3ffc bytes) 000513 1790101604.372187 mapwrite 0x00a6ae thru 0x013001 (0x8954 bytes) 000514 1790101604.408496 trunc from 0x025c31 to 0x02849e (0x286e bytes) 000515 1790101604.417213 trunc from 0x03e233 to 0x03b83f (0x29f3 bytes) 000516 1790101604.476030 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: 0x81bbe252 (stale) Mirror 6: 0xb89c3133 (stale) CRC-32 checksum value for chunk [0, 0x3b840): Mirror 1: 0x7c81d3be Mirror 7: 0x7c81d3be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b840: skipped 000517 1790101606.335431 read 0x0041dd thru 0x01383a (0xf65e bytes) 000518 1790101606.377935 trunc from 0x02e017 to 0x03181b (0x3805 bytes) 000519 1790101606.475891 read 0x000f77 thru 0x009c47 (0x8cd1 bytes) 000520 1790101606.498814 trunc from 0x01ed78 to 0x02dc22 (0xeeab bytes) 000521 1790101606.520966 mapread 0x0347f0 thru 0x035e4b (0x165c bytes) 000522 1790101606.536901 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: 0x81bbe252 (stale) Mirror 6: 0xb89c3133 (stale) Mirror 7: 0x7c81d3be (stale) CRC-32 checksum value for chunk [0, 0x3b840): Mirror 1: 0x7c81d3be Mirror 8: 0x7c81d3be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b840: skipped 000523 1790101608.261087 fallocate 0x01f64b thru 0x024b0d (0x54c3 bytes) 000524 1790101608.328068 write 0x03056e thru 0x035801 (0x5294 bytes) 000525 1790101608.359420 write 0x00ba3a thru 0x0140b2 (0x8679 bytes) 000526 1790101608.382853 write 0x00004e thru 0x0061e5 (0x6198 bytes) 000527 1790101608.401424 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b840): Mirror 5: 0x81bbe252 (stale) Mirror 6: 0xb89c3133 (stale) Mirror 7: 0x7c81d3be (stale) Mirror 8: 0x7c81d3be (stale) CRC-32 checksum value for chunk [0, 0x3b840): Mirror 1: 0xe67bf650 Mirror 9: 0xe67bf650 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b840: skipped 000528 1790101609.478498 fallocate 0x005426 thru 0x0129bb (0xd596 bytes) 000529 1790101609.523560 punch from 0x00cb31 to 0x01853c (0xba0c bytes) 000530 1790101609.537538 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b840): Mirror 6: 0xb89c3133 (stale) Mirror 7: 0x7c81d3be (stale) Mirror 8: 0x7c81d3be (stale) Mirror 9: 0xe67bf650 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b840): Mirror 6: 0xb89c3133 (stale) Mirror 7: 0x7c81d3be (stale) Mirror 8: 0x7c81d3be (stale) Mirror 9: 0xe67bf650 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b840: skipped 000531 1790101609.941021 fallocate 0x028e26 thru 0x02fc18 (0x6df3 bytes) 000532 1790101609.967834 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b840): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b840): Mirror 6: 0xb89c3133 (stale) Mirror 7: 0x7c81d3be (stale) Mirror 8: 0x7c81d3be (stale) Mirror 9: 0xe67bf650 (stale) CRC-32 checksum value for chunk [0, 0x3b840): Mirror 1: 0x5ce5e493 Mirror 10: 0x5ce5e493 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b840: skipped 000533 1790101611.508870 read 0x004c85 thru 0x009b40 (0x4ebc bytes) 000534 1790101611.534444 fallocate 0x03d443 thru 0x03f32e (0x1eec bytes) 000535 1790101611.630239 mapread 0x0091e5 thru 0x012e15 (0x9c31 bytes) 000536 1790101611.682903 MIRROR_RESYNC 6 mirrors 000537 1790101613.465248 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) 000538 1790101613.857677 fallocate 0x025a93 thru 0x033cfb (0xe269 bytes) 000539 1790101613.907091 trunc from 0x032149 to 0x038e2d (0x6ce5 bytes) 000540 1790101613.912309 punch from 0x028f5f to 0x031b16 (0x8bb8 bytes) 000541 1790101613.918906 trunc from 0x003da1 to 0x005275 (0x14d5 bytes) 000542 1790101613.924211 punch from 0x023ec4 to 0x02714f (0x328c bytes) 000543 1790101613.928935 trunc from 0x03b840 to 0x036ce1 (0x4b5e bytes) 000544 1790101613.951296 write 0x00d44a thru 0x011e85 (0x4a3c bytes) 000545 1790101613.960188 write 0x03e9f3 thru 0x03ffff (0x160d bytes) 000546 1790101613.961584 mapread 0x01fb68 thru 0x025571 (0x5a0a bytes) 000547 1790101613.982493 fallocate 0x0359eb thru 0x03c3c2 (0x69d8 bytes) 000548 1790101613.986723 trunc from 0x03788a to 0x03ffff (0x8776 bytes) 000549 1790101614.003307 fallocate 0x020720 thru 0x02b295 (0xab76 bytes) 000550 1790101614.007794 mapread 0x004f64 thru 0x00fa3c (0xaad9 bytes) 000551 1790101614.019014 trunc from 0x01facf to 0x02a953 (0xae85 bytes) 000552 1790101614.027177 MIRROR_RESYNC 5 mirrors 000553 1790101615.779520 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) 000554 1790101616.211905 trunc from 0x029147 to 0x0304e3 (0x739d bytes) 000555 1790101616.275649 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7eeca0c4 (stale) Mirror 9: 0x7eeca0c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7eeca0c4 (stale) Mirror 9: 0x7eeca0c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000556 1790101616.803781 mapwrite 0x01ad8f thru 0x02a1e9 (0xf45b bytes) 000557 1790101616.892160 trunc from 0x01541b to 0x023404 (0xdfea bytes) 000558 1790101616.906777 punch from 0x035881 to 0x03d02d (0x77ad bytes) 000559 1790101616.914366 mapread 0x016ad1 thru 0x024710 (0xdc40 bytes) 000560 1790101616.937671 read 0x03ec24 thru 0x03ffff (0x13dc bytes) 000562 1790101616.944199 trunc from 0x040000 to 0x03e225 (0x1dda bytes) 000563 1790101617.014537 fallocate 0x0000c5 thru 0x00b013 (0xaf4f bytes) 000564 1790101617.023627 read 0x0244d4 thru 0x030843 (0xc370 bytes) 000565 1790101617.048155 punch from 0x007ed4 to 0x00eb2a (0x6c57 bytes) 000566 1790101617.057816 mapwrite 0x035761 thru 0x036490 (0xd30 bytes) 000567 1790101617.098434 fallocate 0x0249ae thru 0x02d840 (0x8e93 bytes) 000568 1790101617.112692 fallocate 0x013854 thru 0x01513b (0x18e8 bytes) 000569 1790101617.125853 mapread 0x032090 thru 0x033fcb (0x1f3c bytes) 000570 1790101617.131963 mapwrite 0x019461 thru 0x026972 (0xd512 bytes) 000571 1790101617.184021 fallocate 0x0350d7 thru 0x03f89a (0xa7c4 bytes) 000572 1790101617.191857 fallocate 0x016bd6 thru 0x0234c7 (0xc8f2 bytes) 000574 1790101617.199163 write 0x00cc5d thru 0x010d6a (0x410e bytes) 000576 1790101617.209733 mapread 0x015022 thru 0x016585 (0x1564 bytes) 000578 1790101617.212685 read 0x001676 thru 0x00f145 (0xdad0 bytes) 000580 1790101617.232671 fallocate 0x005143 thru 0x0140ed (0xefab bytes) 000581 1790101617.243297 write 0x00b504 thru 0x00f889 (0x4386 bytes) 000582 1790101617.244801 trunc from 0x02212b to 0x031e8c (0xfd62 bytes) 000583 1790101617.252095 fallocate 0x030197 thru 0x03cfa2 (0xce0c bytes) 000584 1790101617.258945 mapread 0x018e75 thru 0x026339 (0xd4c5 bytes) 000585 1790101617.325817 MIRROR_RESYNC 3 mirrors 000586 1790101618.800482 MIRROR_EXTEND to 4 mirrors 000587 1790101619.727163 fallocate 0x0034d9 thru 0x01314f (0xfc77 bytes) 000588 1790101619.747678 fallocate 0x035837 thru 0x03ffff (0xa7c9 bytes) 000589 1790101619.752658 trunc from 0x03f89b to 0x00e056 (0x31844 bytes) 000590 1790101619.769531 trunc from 0x0021e6 to 0x00e056 (0xbe71 bytes) 000591 1790101619.778668 MIRROR_RESYNC 4 mirrors 000592 1790101620.645477 write 0x03e7aa thru 0x03f4c2 (0xd19 bytes) 000594 1790101620.676598 mapwrite 0x004d51 thru 0x00e97f (0x9c2f bytes) 000595 1790101620.733689 trunc from 0x03f4c3 to 0x02379d (0x1bd25 bytes) 000596 1790101620.748155 read 0x02051c thru 0x02379d (0x3282 bytes) 000598 1790101620.752213 fallocate 0x03bee1 thru 0x03ffff (0x411f bytes) 000599 1790101620.755516 mapwrite 0x030576 thru 0x036ae3 (0x656e bytes) 000600 1790101620.773085 trunc from 0x040000 to 0x007dcc (0x38233 bytes) 000601 1790101620.798957 write 0x00a4d2 thru 0x00f66c (0x519b bytes) 000602 1790101620.809295 MIRROR_RESYNC 4 mirrors 000603 1790101622.105803 write 0x00ed47 thru 0x019bb7 (0xae71 bytes) 000605 1790101622.140688 punch from 0x0155eb to 0x016d9a (0x17b0 bytes) 000607 1790101622.164005 write 0x02fea4 thru 0x03753d (0x769a bytes) 000608 1790101622.179797 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3753e): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3753e): Mirror 6: 0x5a71b91e (stale) Mirror 9: 0x5a71b91e (stale) Mirror 10: 0x5a71b91e (stale) CRC-32 checksum value for chunk [0, 0x3753e): Mirror 1: 0x3bf94b14 Mirror 11: 0x3bf94b14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3753e: skipped 000610 1790101623.300983 trunc from 0x01879c to 0x01f1c6 (0x6a2b bytes) 000612 1790101623.404032 trunc from 0x03711e to 0x03753d (0x420 bytes) 000613 1790101623.412784 mapread 0x025d92 thru 0x03533e (0xf5ad bytes) 000614 1790101623.473834 mapread 0x012b6b thru 0x01e351 (0xb7e7 bytes) 000615 1790101623.525928 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: 0x5a71b91e (stale) Mirror 10: 0x5a71b91e (stale) Mirror 11: 0x3bf94b14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: 0x5a71b91e (stale) Mirror 10: 0x5a71b91e (stale) Mirror 11: 0x3bf94b14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3753e: skipped 000616 1790101624.260391 write 0x01cea4 thru 0x024e47 (0x7fa4 bytes) 000617 1790101624.328959 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: 0x5a71b91e (stale) Mirror 10: 0x5a71b91e (stale) Mirror 11: 0x3bf94b14 (stale) CRC-32 checksum value for chunk [0, 0x3753e): Mirror 1: 0xd2640bfe Mirror 12: 0xd2640bfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3753e: skipped 000618 1790101625.722159 mapread 0x02f8a8 thru 0x03753d (0x7c96 bytes) 000619 1790101625.741096 mapread 0x0351ad thru 0x03753d (0x2391 bytes) 000620 1790101625.742216 fallocate 0x03822f thru 0x03ffff (0x7dd1 bytes) 000621 1790101625.776208 read 0x005ecb thru 0x007c6a (0x1da0 bytes) 000622 1790101625.788931 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: 0x5a71b91e (stale) Mirror 11: 0x3bf94b14 (stale) Mirror 12: 0xd2640bfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3753e): Mirror 9: 0x5a71b91e (stale) Mirror 11: 0x3bf94b14 (stale) Mirror 12: 0xd2640bfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3753e: skipped 000623 1790101626.110849 mapwrite 0x02765a thru 0x02e485 (0x6e2c bytes) 000625 1790101626.147281 mapwrite 0x039ebd thru 0x03ffff (0x6143 bytes) 000626 1790101626.196124 trunc from 0x0358eb to 0x03ffff (0xa715 bytes) 000627 1790101626.201958 mapwrite 0x005fb5 thru 0x00bc54 (0x5ca0 bytes) 000628 1790101626.235033 read 0x03d5ee thru 0x03f06e (0x1a81 bytes) 000629 1790101626.244649 mapread 0x02b5ce thru 0x036384 (0xadb7 bytes) 000630 1790101626.255576 write 0x03cce4 thru 0x03f07d (0x239a bytes) 000631 1790101626.257074 read 0x01eae9 thru 0x025089 (0x65a1 bytes) 000633 1790101626.265180 mapread 0x000341 thru 0x0093ce (0x908e bytes) 000634 1790101626.267070 read 0x004a74 thru 0x00a0c1 (0x564e bytes) 000635 1790101626.271702 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1790101626.602271 trunc from 0x02cc06 to 0x033cb0 (0x70ab bytes) 000638 1790101626.625185 trunc from 0x012932 to 0x02071c (0xddeb bytes) 000639 1790101626.639751 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x845f8896 Mirror 13: 0x845f8896 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000641 1790101627.985225 read 0x011f86 thru 0x01b277 (0x92f2 bytes) 000642 1790101628.028631 read 0x01fd84 thru 0x02905d (0x92da bytes) 000643 1790101628.055593 fallocate 0x01e571 thru 0x02c245 (0xdcd5 bytes) 000644 1790101628.096056 punch from 0x03a686 to 0x03ffff (0x597a bytes) 000645 1790101628.102827 read 0x0291a0 thru 0x02a16e (0xfcf bytes) 000646 1790101628.125576 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) Mirror 13: 0x845f8896 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc06954c4 Mirror 14: 0xc06954c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000648 1790101629.311601 fallocate 0x0236ed thru 0x0272ae (0x3bc2 bytes) 000649 1790101629.383073 write 0x0071c4 thru 0x00aa1e (0x385b bytes) 000650 1790101629.416179 fallocate 0x029e6a thru 0x0390da (0xf271 bytes) 000651 1790101629.430197 trunc from 0x021920 to 0x02e080 (0xc761 bytes) 000652 1790101629.442138 mapwrite 0x03b862 thru 0x03ffff (0x479e bytes) 000653 1790101629.527804 trunc from 0x006aa7 to 0x01255f (0xbab9 bytes) 000654 1790101629.541014 fallocate 0x02e741 thru 0x02ec72 (0x532 bytes) 000655 1790101629.549073 write 0x032776 thru 0x03fcc7 (0xd552 bytes) 000656 1790101629.565183 write 0x0036ef thru 0x01134c (0xdc5e bytes) 000657 1790101629.591555 read 0x02482a thru 0x02dc0d (0x93e4 bytes) 000658 1790101629.613565 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) Mirror 13: 0x845f8896 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafe03ded (stale) Mirror 12: 0x9642cad9 (stale) Mirror 13: 0x845f8896 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000659 1790101630.195684 punch from 0x01cce2 to 0x027e93 (0xb1b2 bytes) 000660 1790101630.201756 write 0x014b03 thru 0x02026f (0xb76d bytes) 000661 1790101630.227069 write 0x009f86 thru 0x0145af (0xa62a bytes) 000662 1790101630.239281 trunc from 0x040000 to 0x001747 (0x3e8b8 bytes) 000663 1790101630.274293 punch from 0x000690 to 0x00131c (0xc8d bytes) 000665 1790101630.287251 mapwrite 0x0377c1 thru 0x03ed63 (0x75a3 bytes) 000666 1790101630.429761 MIRROR_RESYNC 4 mirrors 000668 1790101631.568575 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) 000669 1790101632.081249 fallocate 0x0284ae thru 0x02ae15 (0x2968 bytes) 000670 1790101632.147795 mapwrite 0x00bf52 thru 0x0122c4 (0x6373 bytes) 000671 1790101632.193902 fallocate 0x037dee thru 0x0390ed (0x1300 bytes) 000672 1790101632.202855 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 11: 0xe054f99b (stale) Mirror 13: 0xe054f99b (stale) CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 1: 0xdf95f38f Mirror 14: 0xdf95f38f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed64: skipped 000673 1790101633.254123 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 11: 0xe054f99b (stale) Mirror 13: 0xe054f99b (stale) CRC-32 checksum value for chunk [0, 0x3ed64): Mirror 1: 0xdf95f38f Mirror 14: 0xdf95f38f Mirror 15: 0xdf95f38f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed64: skipped 000674 1790101634.411684 trunc from 0x03ed64 to 0x01db9f (0x211c4 bytes) 000675 1790101634.497022 fallocate 0x0208c0 thru 0x022bd7 (0x2318 bytes) 000676 1790101634.507035 trunc from 0x01dba0 to 0x031945 (0x13da6 bytes) 000677 1790101634.536438 mapwrite 0x022d1b thru 0x023f75 (0x125b bytes) 000678 1790101634.574780 trunc from 0x031946 to 0x02ad16 (0x6c2f bytes) 000679 1790101634.597847 write 0x01e9d2 thru 0x026152 (0x7781 bytes) 000680 1790101634.615202 mapwrite 0x018896 thru 0x021cfb (0x9466 bytes) 000681 1790101634.677946 MIRROR_RESYNC 5 mirrors 000682 1790101636.576821 mapread 0x029cf6 thru 0x02ad16 (0x1021 bytes) 000683 1790101636.597234 read 0x0014c8 thru 0x00cc61 (0xb79a bytes) 000684 1790101636.645490 mapwrite 0x00d93f thru 0x01acab (0xd36d bytes) 000685 1790101636.949711 write 0x015427 thru 0x0174dc (0x20b6 bytes) 000686 1790101636.952212 mapwrite 0x011659 thru 0x020625 (0xefcd bytes) 000687 1790101637.058303 write 0x025c43 thru 0x0302b3 (0xa671 bytes) 000688 1790101637.064611 punch from 0x002827 to 0x009db7 (0x7591 bytes) 000689 1790101637.079659 write 0x025ab2 thru 0x0329a0 (0xceef bytes) 000690 1790101637.087280 trunc from 0x00dd12 to 0x010afa (0x2de9 bytes) 000691 1790101637.098947 mapread 0x007a67 thru 0x017352 (0xf8ec bytes) 000692 1790101637.124458 fallocate 0x033c83 thru 0x035acb (0x1e49 bytes) 000694 1790101637.143052 trunc from 0x035acc to 0x02ee75 (0x6c56 bytes) 000695 1790101637.229715 MIRROR_RESYNC 5 mirrors 000696 1790101639.068883 trunc from 0x026fcc to 0x027d4b (0xd80 bytes) 000697 1790101639.115919 punch from 0x020616 to 0x02ee75 (0xe860 bytes) 000698 1790101639.123181 punch from 0x00714b to 0x008936 (0x17ec bytes) 000699 1790101639.128078 punch from 0x01e806 to 0x025020 (0x681b bytes) 000700 1790101639.134247 mapread 0x01acfc thru 0x024c54 (0x9f59 bytes) 000702 1790101639.158141 trunc from 0x015355 to 0x02525c (0xff08 bytes) 000703 1790101639.166689 punch from 0x0221e4 to 0x0256ef (0x350c bytes) 000704 1790101639.174072 trunc from 0x001e4e to 0x0111db (0xf38e bytes) 000705 1790101639.181925 trunc from 0x02ee76 to 0x01d490 (0x119e5 bytes) 000706 1790101639.221539 fallocate 0x004649 thru 0x00fbca (0xb582 bytes) 000707 1790101639.244831 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d491): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d491): Mirror 11: 0x6d8448ff (stale) Mirror 13: 0x6d8448ff (stale) Mirror 14: 0x6d8448ff (stale) Mirror 15: 0x6d8448ff (stale) CRC-32 checksum value for chunk [0, 0x1d491): Mirror 1: 0xecd1649 Mirror 16: 0xecd1649 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d491: skipped 000708 1790101640.681107 write 0x035c22 thru 0x03ffff (0xa3de bytes) 000709 1790101640.745778 trunc from 0x029e67 to 0x0334cb (0x9665 bytes) 000710 1790101640.754272 mapread 0x039baa thru 0x03e725 (0x4b7c bytes) 000711 1790101640.755598 write 0x01c446 thru 0x024b59 (0x8714 bytes) 000712 1790101640.787596 trunc from 0x040000 to 0x00d016 (0x32fe9 bytes) 000714 1790101640.855720 fallocate 0x0385ea thru 0x03ffff (0x7a16 bytes) 000715 1790101640.870552 fallocate 0x01bfc6 thru 0x01dfe5 (0x2020 bytes) 000716 1790101640.880921 fallocate 0x02a5f0 thru 0x02a615 (0x26 bytes) 000717 1790101640.890105 trunc from 0x039f1d to 0x03a02b (0x10f bytes) 000718 1790101640.919690 fallocate 0x02ca83 thru 0x03222d (0x57ab bytes) 000719 1790101640.956061 mapwrite 0x00d454 thru 0x0183a3 (0xaf50 bytes) 000721 1790101641.142232 mapread 0x00e953 thru 0x0138dc (0x4f8a bytes) 000722 1790101641.147104 mapwrite 0x03dec4 thru 0x03ffff (0x213c bytes) 000723 1790101641.217202 punch from 0x032c7e to 0x03c361 (0x96e4 bytes) 000724 1790101641.225016 fallocate 0x02cfc3 thru 0x0341e0 (0x721e bytes) 000725 1790101641.238967 mapread 0x017213 thru 0x01d7c0 (0x65ae bytes) 000726 1790101641.243151 fallocate 0x01c23c thru 0x028f79 (0xcd3e bytes) 000727 1790101641.250383 write 0x00da88 thru 0x016173 (0x86ec bytes) 000729 1790101641.252745 mapwrite 0x00a577 thru 0x00f2b9 (0x4d43 bytes) 000730 1790101641.318422 MIRROR_RESYNC 6 mirrors 000731 1790101643.941900 fallocate 0x030a41 thru 0x0360e8 (0x56a8 bytes) 000732 1790101644.013106 mapwrite 0x0196f3 thru 0x024328 (0xac36 bytes) 000733 1790101644.119673 punch from 0x024a2b to 0x02e6a8 (0x9c7e bytes) 000734 1790101644.136951 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x60a7d91b (stale) Mirror 13: 0x60a7d91b (stale) Mirror 14: 0x60a7d91b (stale) Mirror 15: 0x60a7d91b (stale) Mirror 16: 0x60a7d91b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad615772 Mirror 17: 0xad615772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000735 1790101645.549722 fallocate 0x03547c thru 0x03ffff (0xab84 bytes) 000736 1790101645.646415 trunc from 0x040000 to 0x00b991 (0x3466e bytes) 000737 1790101645.728877 trunc from 0x004c26 to 0x0061b4 (0x158f bytes) 000738 1790101645.748401 trunc from 0x000909 to 0x002cce (0x23c6 bytes) 000739 1790101645.764302 mapread 0x009a2d thru 0x00b991 (0x1f65 bytes) 000740 1790101645.827088 mapread 0x00694d thru 0x00b991 (0x5045 bytes) 000741 1790101645.843473 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb992): Mirror 11: 0x8c248593 (stale) Mirror 13: 0x8c248593 (stale) Mirror 14: 0x8c248593 (stale) Mirror 16: 0x8c248593 (stale) Mirror 17: 0x8c248593 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb992): Mirror 11: 0x8c248593 (stale) Mirror 13: 0x8c248593 (stale) Mirror 14: 0x8c248593 (stale) Mirror 16: 0x8c248593 (stale) Mirror 17: 0x8c248593 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb992: skipped 000743 1790101646.652637 trunc from 0x00b992 to 0x01cca4 (0x11313 bytes) 000744 1790101646.717486 fallocate 0x01ffec thru 0x0226d8 (0x26ed bytes) 000745 1790101646.725541 fallocate 0x03790d thru 0x038fc5 (0x16b9 bytes) 000746 1790101646.750639 mapwrite 0x016b0c thru 0x0228b2 (0xbda7 bytes) 000747 1790101646.860438 write 0x0009b3 thru 0x00d71d (0xcd6b bytes) 000748 1790101646.878620 trunc from 0x00cdf8 to 0x01ae51 (0xe05a bytes) 000749 1790101646.904019 mapread 0x0379dc thru 0x038fc5 (0x15ea bytes) 000750 1790101646.922685 trunc from 0x018da3 to 0x01ab95 (0x1df3 bytes) 000751 1790101646.936672 mapwrite 0x012840 thru 0x01871e (0x5edf bytes) 000753 1790101647.015129 mapread 0x002845 thru 0x0119a8 (0xf164 bytes) 000754 1790101647.016117 mapwrite 0x035ab2 thru 0x03ffff (0xa54e bytes) 000755 1790101647.105061 read 0x03433c thru 0x03c41c (0x80e1 bytes) 000757 1790101647.108596 trunc from 0x01d0f0 to 0x028732 (0xb643 bytes) 000758 1790101647.137619 fallocate 0x014191 thru 0x016524 (0x2394 bytes) 000759 1790101647.157989 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x60a7d91b (stale) Mirror 13: 0x60a7d91b (stale) Mirror 14: 0x60a7d91b (stale) Mirror 16: 0x60a7d91b (stale) Mirror 17: 0xad615772 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99db571f Mirror 18: 0x99db571f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1790101648.397193 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x60a7d91b (stale) Mirror 13: 0x60a7d91b (stale) Mirror 14: 0x60a7d91b (stale) Mirror 16: 0x60a7d91b (stale) Mirror 17: 0xad615772 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99db571f Mirror 18: 0x99db571f Mirror 19: 0x99db571f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000762 1790101649.766979 trunc from 0x040000 to 0x039382 (0x6c7d bytes) 000763 1790101649.969982 fallocate 0x01739b thru 0x02362e (0xc294 bytes) 000764 1790101649.980436 mapread 0x034713 thru 0x0374ba (0x2da8 bytes) 000765 1790101650.039833 read 0x0040e0 thru 0x0111f2 (0xd113 bytes) 000766 1790101650.059740 mapread 0x021c79 thru 0x030613 (0xe99b bytes) 000767 1790101650.127683 mapread 0x003574 thru 0x007c5f (0x46ec bytes) 000768 1790101650.140522 read 0x01cd9d thru 0x01dec8 (0x112c bytes) 000770 1790101650.145470 mapread 0x018cc6 thru 0x02522f (0xc56a bytes) 000771 1790101650.167985 read 0x00daf7 thru 0x00f9d4 (0x1ede bytes) 000772 1790101650.177665 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39383): Mirror 11: 0xed19be33 (stale) Mirror 13: 0xed19be33 (stale) Mirror 16: 0xed19be33 (stale) Mirror 17: 0xd59fabf3 (stale) Mirror 18: 0x38bbb7a9 (stale) Mirror 19: 0x38bbb7a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39383): Mirror 11: 0xed19be33 (stale) Mirror 13: 0xed19be33 (stale) Mirror 16: 0xed19be33 (stale) Mirror 17: 0xd59fabf3 (stale) Mirror 18: 0x38bbb7a9 (stale) Mirror 19: 0x38bbb7a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39383: skipped 000773 1790101650.934107 punch from 0x011df1 to 0x017ef4 (0x6104 bytes) 000774 1790101650.949057 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39383): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39383): Mirror 11: 0xed19be33 (stale) Mirror 13: 0xed19be33 (stale) Mirror 16: 0xed19be33 (stale) Mirror 17: 0xd59fabf3 (stale) Mirror 18: 0x38bbb7a9 (stale) Mirror 19: 0x38bbb7a9 (stale) CRC-32 checksum value for chunk [0, 0x39383): Mirror 1: 0xeff9ffb2 Mirror 20: 0xeff9ffb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39383: skipped 000775 1790101652.314891 MIRROR_RESYNC 8 mirrors 000776 1790101654.946347 MIRROR_EXTEND to 9 mirrors 000777 1790101655.630358 fallocate 0x03bbec thru 0x03da87 (0x1e9c bytes) 000778 1790101655.682488 trunc from 0x039383 to 0x00a40a (0x2ef78 bytes) 000779 1790101655.704592 read 0x001045 thru 0x00a40a (0x93c6 bytes) 000780 1790101655.713539 punch from 0x002637 to 0x007284 (0x4c4e bytes) 000781 1790101655.724038 MIRROR_RESYNC 9 mirrors 000783 1790101657.729751 punch from 0x006ef8 to 0x00a40a (0x3513 bytes) 000784 1790101657.799619 trunc from 0x00a40b to 0x0121cb (0x7dc1 bytes) 000786 1790101657.828865 mapwrite 0x009829 thru 0x00fb9d (0x6375 bytes) 000787 1790101657.876482 punch from 0x00571b to 0x012104 (0xc9ea bytes) 000788 1790101657.881062 trunc from 0x00ee1b to 0x010373 (0x1559 bytes) 000789 1790101657.887618 read 0x00d723 thru 0x0121cb (0x4aa9 bytes) 000791 1790101657.897458 read 0x00762f thru 0x00b31a (0x3cec bytes) 000792 1790101657.908856 mapread 0x007285 thru 0x009af2 (0x286e bytes) 000793 1790101657.910273 read 0x000e35 thru 0x004ca3 (0x3e6f bytes) 000794 1790101657.910923 trunc from 0x0121cc to 0x03053a (0x1e36f bytes) 000795 1790101657.938118 mapread 0x02b726 thru 0x03053a (0x4e15 bytes) 000796 1790101657.947357 mapread 0x02a304 thru 0x02cd9d (0x2a9a bytes) 000797 1790101657.953313 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3053b): Mirror 11: 0x2fbec33 (stale) Mirror 13: 0x2fbec33 (stale) Mirror 16: 0x2fbec33 (stale) Mirror 17: 0x2fbec33 (stale) Mirror 18: 0x2fbec33 (stale) Mirror 19: 0x2fbec33 (stale) Mirror 20: 0x2fbec33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3053b): Mirror 11: 0x2fbec33 (stale) Mirror 13: 0x2fbec33 (stale) Mirror 16: 0x2fbec33 (stale) Mirror 17: 0x2fbec33 (stale) Mirror 18: 0x2fbec33 (stale) Mirror 19: 0x2fbec33 (stale) Mirror 20: 0x2fbec33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3053b: skipped 000798 1790101658.717501 trunc from 0x0126b7 to 0x01424a (0x1b94 bytes) 000799 1790101658.742839 punch from 0x00cb20 to 0x00f608 (0x2ae9 bytes) 000800 1790101658.750530 write 0x027669 thru 0x02a4f6 (0x2e8e bytes) 000801 1790101658.806445 punch from 0x02c1b9 to 0x03053a (0x4382 bytes) 000802 1790101658.814199 trunc from 0x03053b to 0x03b28d (0xad53 bytes) 000803 1790101658.879804 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b28e): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b28e): Mirror 11: 0xce6e8bb5 (stale) Mirror 13: 0xce6e8bb5 (stale) Mirror 16: 0xce6e8bb5 (stale) Mirror 17: 0xce6e8bb5 (stale) Mirror 18: 0xce6e8bb5 (stale) Mirror 19: 0xce6e8bb5 (stale) Mirror 20: 0xce6e8bb5 (stale) CRC-32 checksum value for chunk [0, 0x3b28e): Mirror 1: 0xd9f8f26d Mirror 21: 0xd9f8f26d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b28e: skipped 000804 1790101660.661726 mapread 0x03a55d thru 0x03b28d (0xd31 bytes) 000805 1790101660.676540 mapread 0x039fde thru 0x03aad0 (0xaf3 bytes) 000806 1790101660.688311 read 0x02ab87 thru 0x038b45 (0xdfbf bytes) 000807 1790101660.696036 trunc from 0x03b28e to 0x0019bf (0x398ce bytes) 000808 1790101660.759222 punch from 0x001348 to 0x0019bf (0x678 bytes) 000809 1790101660.765416 fallocate 0x002c86 thru 0x006ffc (0x4377 bytes) 000810 1790101660.776487 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x19c0): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c0): Mirror 11: 0x7fe5c080 (stale) Mirror 13: 0x7fe5c080 (stale) Mirror 16: 0x7fe5c080 (stale) Mirror 17: 0x7fe5c080 (stale) Mirror 18: 0x7fe5c080 (stale) Mirror 19: 0x7fe5c080 (stale) Mirror 20: 0x7fe5c080 (stale) Mirror 21: 0x7fe5c080 (stale) CRC-32 checksum value for chunk [0, 0x19c0): Mirror 1: 0x90b75355 Mirror 22: 0x90b75355 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c0: skipped 000811 1790101661.817641 mapread 0x000ebf thru 0x0019bf (0xb01 bytes) 000813 1790101661.856165 fallocate 0x0071eb thru 0x016a67 (0xf87d bytes) 000814 1790101661.903512 punch from 0x00fb2a to 0x016a67 (0x6f3e bytes) 000815 1790101661.908532 mapread 0x00eac6 thru 0x014ad0 (0x600b bytes) 000816 1790101661.922269 trunc from 0x00cbd6 to 0x016a67 (0x9e92 bytes) 000817 1790101661.926966 mapread 0x00eda2 thru 0x0120df (0x333e bytes) 000818 1790101661.938771 mapwrite 0x02e983 thru 0x03aa32 (0xc0b0 bytes) 000819 1790101661.987382 mapwrite 0x033967 thru 0x03df0d (0xa5a7 bytes) 000820 1790101662.044509 mapwrite 0x0181fc thru 0x01ce5c (0x4c61 bytes) 000821 1790101662.100180 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3df0e): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df0e): Mirror 11: 0xda1c2403 (stale) Mirror 13: 0xda1c2403 (stale) Mirror 16: 0xda1c2403 (stale) Mirror 17: 0xda1c2403 (stale) Mirror 18: 0xda1c2403 (stale) Mirror 19: 0xda1c2403 (stale) Mirror 20: 0xda1c2403 (stale) Mirror 21: 0x4ae8f29 (stale) Mirror 22: 0x63bb952 (stale) CRC-32 checksum value for chunk [0, 0x3df0e): Mirror 1: 0xd1bc4276 Mirror 23: 0xd1bc4276 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df0e: skipped 000822 1790101663.061096 mapwrite 0x004b5a thru 0x00517a (0x621 bytes) 000823 1790101663.236122 fallocate 0x038d16 thru 0x03ba17 (0x2d02 bytes) 000824 1790101663.240914 mapwrite 0x007914 thru 0x0128ea (0xafd7 bytes) 000826 1790101663.275014 fallocate 0x034d79 thru 0x03828b (0x3513 bytes) 000827 1790101663.285272 trunc from 0x03df0e to 0x010793 (0x2d77a bytes) 000828 1790101663.363571 write 0x015fa0 thru 0x02008b (0xa0ec bytes) 000829 1790101663.367340 mapwrite 0x0089e4 thru 0x010388 (0x79a5 bytes) 000830 1790101663.395847 trunc from 0x0000f0 to 0x0099b7 (0x98c8 bytes) 000831 1790101663.405831 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2008c): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2008c): Mirror 11: 0x5906014e (stale) Mirror 13: 0x5906014e (stale) Mirror 16: 0x5906014e (stale) Mirror 17: 0x5906014e (stale) Mirror 18: 0x5906014e (stale) Mirror 19: 0x5906014e (stale) Mirror 20: 0x5906014e (stale) Mirror 21: 0x259a58cb (stale) Mirror 22: 0xde5bec39 (stale) Mirror 23: 0x99f3eb57 (stale) CRC-32 checksum value for chunk [0, 0x2008c): Mirror 1: 0x3e398f41 Mirror 24: 0x3e398f41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2008c: skipped 000832 1790101664.323950 MIRROR_RESYNC 12 mirrors 000833 1790101666.335666 write 0x0072a1 thru 0x0085fc (0x135c bytes) 000834 1790101666.394230 fallocate 0x036d3a thru 0x03ffff (0x92c6 bytes) 000835 1790101666.399809 mapread 0x02febc thru 0x030191 (0x2d6 bytes) 000836 1790101666.410114 punch from 0x023e99 to 0x0258bc (0x1a24 bytes) 000837 1790101666.414221 mapwrite 0x021d27 thru 0x030d14 (0xefee bytes) 000838 1790101666.456501 mapwrite 0x01f78c thru 0x020504 (0xd79 bytes) 000839 1790101666.473135 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 13: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 17: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0833322 Mirror 25: 0xc0833322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000840 1790101668.275078 trunc from 0x0139fa to 0x015eb0 (0x24b7 bytes) 000841 1790101668.395671 write 0x02dd3f thru 0x030ff0 (0x32b2 bytes) 000842 1790101668.438921 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 13: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) Mirror 25: 0xc0833322 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 13: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) Mirror 25: 0xc0833322 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1790101669.511080 write 0x020e7e thru 0x02ea3c (0xdbbf bytes) 000844 1790101669.552913 trunc from 0x02ab69 to 0x03824c (0xd6e4 bytes) 000845 1790101669.588935 trunc from 0x03f084 to 0x03ffff (0xf7c bytes) 000846 1790101669.606867 mapread 0x009629 thru 0x0103f1 (0x6dc9 bytes) 000847 1790101669.677394 read 0x02d530 thru 0x03bccc (0xe79d bytes) 000848 1790101669.696595 read 0x02a7b6 thru 0x035707 (0xaf52 bytes) 000849 1790101669.713145 write 0x025f7e thru 0x032e2b (0xceae bytes) 000851 1790101669.716107 punch from 0x028e2a to 0x033e88 (0xb05f bytes) 000852 1790101669.777698 mapwrite 0x039767 thru 0x03ffff (0x6899 bytes) 000853 1790101669.816499 punch from 0x0016e4 to 0x00ccd2 (0xb5ef bytes) 000854 1790101669.840220 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) Mirror 25: 0xc0833322 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) Mirror 25: 0xc0833322 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1790101671.083989 fallocate 0x02cf1d thru 0x0375cc (0xa6b0 bytes) 000856 1790101671.103948 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1b61072e (stale) Mirror 16: 0x1b61072e (stale) Mirror 18: 0x1b61072e (stale) Mirror 19: 0x1b61072e (stale) Mirror 20: 0x1b61072e (stale) Mirror 21: 0x1b61072e (stale) Mirror 22: 0x1b61072e (stale) Mirror 23: 0x1b61072e (stale) Mirror 24: 0x1b61072e (stale) Mirror 25: 0xc0833322 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa0a775ca Mirror 26: 0xa0a775ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1790101673.073294 MIRROR_RESYNC 12 mirrors 000858 1790101676.588739 fallocate 0x0163e9 thru 0x020821 (0xa439 bytes) 000859 1790101676.653738 MIRROR_RESYNC 12 mirrors 000860 1790101681.006449 read 0x0222a6 thru 0x030e10 (0xeb6b bytes) 000861 1790101681.023664 fallocate 0x036733 thru 0x03baf8 (0x53c6 bytes) 000862 1790101681.064795 read 0x015ca2 thru 0x01845b (0x27ba bytes) 000863 1790101681.077763 write 0x013779 thru 0x02053d (0xcdc5 bytes) 000864 1790101681.079770 trunc from 0x02a5fd to 0x038e8d (0xe891 bytes) 000865 1790101681.085488 trunc from 0x040000 to 0x00d5c6 (0x32a39 bytes) 000866 1790101681.113845 trunc from 0x00cad3 to 0x00d5c6 (0xaf4 bytes) 000867 1790101681.120394 punch from 0x002b2d to 0x00d5c6 (0xaa9a bytes) 000868 1790101681.130947 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xd5c7): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd5c7): Mirror 11: 0xa49ada98 (stale) Mirror 16: 0xa49ada98 (stale) Mirror 18: 0xa49ada98 (stale) Mirror 19: 0xa49ada98 (stale) Mirror 20: 0xa49ada98 (stale) Mirror 21: 0xa49ada98 (stale) Mirror 22: 0xa49ada98 (stale) Mirror 23: 0xa49ada98 (stale) Mirror 24: 0xa49ada98 (stale) Mirror 25: 0xa49ada98 (stale) Mirror 26: 0xa49ada98 (stale) CRC-32 checksum value for chunk [0, 0xd5c7): Mirror 1: 0x96abc44c Mirror 27: 0x96abc44c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd5c7: skipped 000869 1790101682.701578 mapread 0x0008fd thru 0x00762c (0x6d30 bytes) 000870 1790101682.747482 mapwrite 0x021732 thru 0x0301e0 (0xeaaf bytes) 000871 1790101682.975684 read 0x019920 thru 0x026230 (0xc911 bytes) 000873 1790101682.989041 MIRROR_RESYNC 13 mirrors 000875 1790101685.419019 read 0x0248a2 thru 0x02b893 (0x6ff2 bytes) 000876 1790101685.427100 trunc from 0x0301e1 to 0x03b207 (0xb027 bytes) 000877 1790101685.494913 trunc from 0x03b208 to 0x007b04 (0x33703 bytes) 000878 1790101685.535938 mapwrite 0x03ea0d thru 0x03ffff (0x15f3 bytes) 000879 1790101685.616787 write 0x01d0a8 thru 0x028d03 (0xbc5c bytes) 000880 1790101685.630548 fallocate 0x0337de thru 0x03a3ce (0x6bf1 bytes) 000881 1790101685.661543 MIRROR_RESYNC 13 mirrors 000882 1790101688.293994 trunc from 0x012279 to 0x017db4 (0x5b3c bytes) 000883 1790101688.349257 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4d49f049 (stale) Mirror 18: 0x4d49f049 (stale) Mirror 19: 0x4d49f049 (stale) Mirror 20: 0x4d49f049 (stale) Mirror 21: 0x4d49f049 (stale) Mirror 22: 0x4d49f049 (stale) Mirror 23: 0x4d49f049 (stale) Mirror 24: 0x4d49f049 (stale) Mirror 25: 0x4d49f049 (stale) Mirror 26: 0x4d49f049 (stale) Mirror 27: 0x4d49f049 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4d49f049 (stale) Mirror 18: 0x4d49f049 (stale) Mirror 19: 0x4d49f049 (stale) Mirror 20: 0x4d49f049 (stale) Mirror 21: 0x4d49f049 (stale) Mirror 22: 0x4d49f049 (stale) Mirror 23: 0x4d49f049 (stale) Mirror 24: 0x4d49f049 (stale) Mirror 25: 0x4d49f049 (stale) Mirror 26: 0x4d49f049 (stale) Mirror 27: 0x4d49f049 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1790101689.054189 write 0x021d70 thru 0x02d64e (0xb8df bytes) 000885 1790101689.076477 MIRROR_RESYNC 12 mirrors 000887 1790101691.060777 trunc from 0x040000 to 0x00c80c (0x337f3 bytes) 000888 1790101691.102687 write 0x01dec0 thru 0x01fc91 (0x1dd2 bytes) 000889 1790101691.114735 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fc92): Mirror 16: 0x3a9a68ac (stale) Mirror 19: 0x3a9a68ac (stale) Mirror 20: 0x3a9a68ac (stale) Mirror 21: 0x3a9a68ac (stale) Mirror 22: 0x3a9a68ac (stale) Mirror 23: 0x3a9a68ac (stale) Mirror 24: 0x3a9a68ac (stale) Mirror 25: 0x3a9a68ac (stale) Mirror 26: 0x3a9a68ac (stale) Mirror 27: 0x3a9a68ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc92): Mirror 16: 0x3a9a68ac (stale) Mirror 19: 0x3a9a68ac (stale) Mirror 20: 0x3a9a68ac (stale) Mirror 21: 0x3a9a68ac (stale) Mirror 22: 0x3a9a68ac (stale) Mirror 23: 0x3a9a68ac (stale) Mirror 24: 0x3a9a68ac (stale) Mirror 25: 0x3a9a68ac (stale) Mirror 26: 0x3a9a68ac (stale) Mirror 27: 0x3a9a68ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc92: skipped 000891 1790101691.601925 MIRROR_RESYNC 11 mirrors 000892 1790101692.921880 read 0x015a20 thru 0x01fc91 (0xa272 bytes) 000893 1790101692.930859 mapwrite 0x02911a thru 0x038be2 (0xfac9 bytes) 000894 1790101693.024861 mapwrite 0x00f75b thru 0x01c26e (0xcb14 bytes) 000896 1790101693.064269 punch from 0x015463 to 0x0193cd (0x3f6b bytes) 000897 1790101693.070238 punch from 0x02328a to 0x0302c3 (0xd03a bytes) 000898 1790101693.075868 mapread 0x007ed0 thru 0x00e9be (0x6aef bytes) 000899 1790101693.086054 trunc from 0x02c485 to 0x036f4c (0xaac8 bytes) 000900 1790101693.093944 write 0x00297c thru 0x004a2b (0x20b0 bytes) 000901 1790101693.095173 fallocate 0x01baf7 thru 0x01d393 (0x189d bytes) 000902 1790101693.100968 punch from 0x023eb9 to 0x03274a (0xe892 bytes) 000903 1790101693.107650 mapwrite 0x034600 thru 0x03ffff (0xba00 bytes) 000904 1790101693.160936 punch from 0x02803a to 0x02b659 (0x3620 bytes) 000905 1790101693.165265 trunc from 0x01c0ce to 0x0233be (0x72f1 bytes) 000906 1790101693.174258 trunc from 0x040000 to 0x030d9b (0xf264 bytes) 000907 1790101693.199860 trunc from 0x030d9c to 0x023095 (0xdd06 bytes) 000908 1790101693.250202 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x23096): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23096): Mirror 16: 0x4c1b839c (stale) Mirror 19: 0x4c1b839c (stale) Mirror 20: 0x4c1b839c (stale) Mirror 21: 0x4c1b839c (stale) Mirror 22: 0x4c1b839c (stale) Mirror 23: 0x4c1b839c (stale) Mirror 24: 0x4c1b839c (stale) Mirror 25: 0x4c1b839c (stale) Mirror 26: 0x4c1b839c (stale) Mirror 27: 0x4c1b839c (stale) CRC-32 checksum value for chunk [0, 0x23096): Mirror 1: 0xf0b488ea Mirror 28: 0xf0b488ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23096: skipped 000909 1790101694.874286 fallocate 0x0099dc thru 0x009a04 (0x29 bytes) 000910 1790101694.974007 trunc from 0x023096 to 0x01f2bb (0x3dda bytes) 000911 1790101695.028951 fallocate 0x011678 thru 0x019f3e (0x88c7 bytes) 000912 1790101695.038604 fallocate 0x02ce5d thru 0x039fee (0xd192 bytes) 000913 1790101695.055291 read 0x0215b8 thru 0x03088c (0xf2d5 bytes) 000914 1790101695.106547 fallocate 0x020fa1 thru 0x0257a1 (0x4801 bytes) 000915 1790101695.121530 read 0x018fd4 thru 0x01e677 (0x56a4 bytes) 000916 1790101695.137196 write 0x011828 thru 0x012427 (0xc00 bytes) 000917 1790101695.138206 fallocate 0x02a2c5 thru 0x030daf (0x6aeb bytes) 000918 1790101695.164603 trunc from 0x039fef to 0x014320 (0x25cce bytes) 000919 1790101695.206750 read 0x00bea3 thru 0x013515 (0x7673 bytes) 000920 1790101695.207453 punch from 0x001303 to 0x00e5fc (0xd2fa bytes) 000921 1790101695.215806 mapread 0x00534d thru 0x0110fb (0xbdaf bytes) 000922 1790101695.240173 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x14321): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14321): Mirror 16: 0x8ee54ea5 (stale) Mirror 19: 0x8ee54ea5 (stale) Mirror 20: 0x8ee54ea5 (stale) Mirror 21: 0x8ee54ea5 (stale) Mirror 22: 0x8ee54ea5 (stale) Mirror 23: 0x8ee54ea5 (stale) Mirror 24: 0x8ee54ea5 (stale) Mirror 25: 0x8ee54ea5 (stale) Mirror 26: 0x8ee54ea5 (stale) Mirror 27: 0x8ee54ea5 (stale) Mirror 28: 0x13a1fa63 (stale) CRC-32 checksum value for chunk [0, 0x14321): Mirror 1: 0x51bfdace Mirror 29: 0x51bfdace lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14321: skipped 000923 1790101696.312421 trunc from 0x010b1a to 0x014320 (0x3807 bytes) 000926 1790101696.381981 read 0x0125fe thru 0x012867 (0x26a bytes) 000927 1790101696.389436 mapread 0x003ade thru 0x00aa64 (0x6f87 bytes) 000928 1790101696.457473 fallocate 0x0017db thru 0x001c66 (0x48c bytes) 000929 1790101696.472497 mapread 0x011864 thru 0x014320 (0x2abd bytes) 000930 1790101696.486904 fallocate 0x0098ad thru 0x0189a2 (0xf0f6 bytes) 000931 1790101696.496117 punch from 0x008786 to 0x00d04a (0x48c5 bytes) 000932 1790101696.526868 fallocate 0x024b21 thru 0x02ae5f (0x633f bytes) 000933 1790101696.542625 trunc from 0x0189a3 to 0x03ffa7 (0x27605 bytes) 000934 1790101696.674863 mapread 0x034711 thru 0x03dabf (0x93af bytes) 000935 1790101696.716153 read 0x03a491 thru 0x03ffa7 (0x5b17 bytes) 000936 1790101696.716774 mapread 0x001552 thru 0x00703f (0x5aee bytes) 000937 1790101696.741139 read 0x01c520 thru 0x020a41 (0x4522 bytes) 000938 1790101696.767066 punch from 0x02b952 to 0x02e6a5 (0x2d54 bytes) 000939 1790101696.780008 trunc from 0x032c44 to 0x03c823 (0x9be0 bytes) 000940 1790101696.792924 mapread 0x02bfdc thru 0x03492b (0x8950 bytes) 000941 1790101696.877637 read 0x01930e thru 0x01f229 (0x5f1c bytes) 000942 1790101696.880252 read 0x010d9f thru 0x01ae97 (0xa0f9 bytes) 000943 1790101696.883183 fallocate 0x01f16a thru 0x028107 (0x8f9e bytes) 000944 1790101696.896271 read 0x007e5e thru 0x01721a (0xf3bd bytes) 000945 1790101696.897867 trunc from 0x03ffa8 to 0x021535 (0x1ea72 bytes) 000946 1790101696.920797 mapwrite 0x02c67b thru 0x03259f (0x5f25 bytes) 000947 1790101696.983586 write 0x01d21e thru 0x026f11 (0x9cf4 bytes) 000948 1790101696.986169 read 0x01819f thru 0x027eaa (0xfd0c bytes) 000949 1790101696.987154 mapwrite 0x037449 thru 0x03ffff (0x8bb7 bytes) 000950 1790101697.040151 read 0x035e40 thru 0x0373b4 (0x1575 bytes) 000951 1790101697.041499 mapwrite 0x01ce16 thru 0x027649 (0xa834 bytes) 000952 1790101697.065036 trunc from 0x040000 to 0x025c82 (0x1a37d bytes) 000953 1790101697.083969 fallocate 0x02d06e thru 0x02f3f1 (0x2384 bytes) 000954 1790101697.091535 mapwrite 0x00cb2f thru 0x01c22b (0xf6fd bytes) 000955 1790101697.127033 mapwrite 0x004276 thru 0x008b6b (0x48f6 bytes) 000956 1790101697.152735 mapwrite 0x01ecb1 thru 0x02e436 (0xf786 bytes) 000957 1790101697.226103 trunc from 0x02cec3 to 0x02df6e (0x10ac bytes) 000958 1790101697.234077 trunc from 0x01f372 to 0x02228f (0x2f1e bytes) 000959 1790101697.247775 MIRROR_RESYNC 13 mirrors 000960 1790101701.501959 trunc from 0x02e437 to 0x00883f (0x25bf7 bytes) 000961 1790101701.568429 write 0x031ab9 thru 0x03ffff (0xe547 bytes) 000962 1790101701.570243 write 0x037deb thru 0x03fd07 (0x7f1d bytes) 000963 1790101701.573477 trunc from 0x03bec5 to 0x03f053 (0x318f bytes) 000964 1790101701.601119 punch from 0x0181db to 0x021bda (0x9a00 bytes) 000965 1790101701.613459 write 0x02fc23 thru 0x03159e (0x197c bytes) 000967 1790101701.625514 fallocate 0x01c0fb thru 0x020d0d (0x4c13 bytes) 000968 1790101701.646420 read 0x0116dc thru 0x02168c (0xffb1 bytes) 000969 1790101701.667677 read 0x02eb09 thru 0x032834 (0x3d2c bytes) 000970 1790101701.686877 mapwrite 0x02bac7 thru 0x02de8b (0x23c5 bytes) 000971 1790101701.706504 write 0x02f08a thru 0x033ad4 (0x4a4b bytes) 000973 1790101701.709477 trunc from 0x0345da to 0x03549c (0xec3 bytes) 000974 1790101701.716141 write 0x02d901 thru 0x037947 (0xa047 bytes) 000975 1790101701.718904 write 0x00661f thru 0x00dcb4 (0x7696 bytes) 000976 1790101701.729471 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x51578844 (stale) Mirror 20: 0x51578844 (stale) Mirror 21: 0x51578844 (stale) Mirror 22: 0x51578844 (stale) Mirror 23: 0x51578844 (stale) Mirror 24: 0x51578844 (stale) Mirror 25: 0x51578844 (stale) Mirror 26: 0x51578844 (stale) Mirror 27: 0x51578844 (stale) Mirror 28: 0x51578844 (stale) Mirror 29: 0x51578844 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x51578844 (stale) Mirror 20: 0x51578844 (stale) Mirror 21: 0x51578844 (stale) Mirror 22: 0x51578844 (stale) Mirror 23: 0x51578844 (stale) Mirror 24: 0x51578844 (stale) Mirror 25: 0x51578844 (stale) Mirror 26: 0x51578844 (stale) Mirror 27: 0x51578844 (stale) Mirror 28: 0x51578844 (stale) Mirror 29: 0x51578844 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000977 1790101702.331067 fallocate 0x02120e thru 0x029541 (0x8334 bytes) 000978 1790101702.340529 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x51578844 (stale) Mirror 20: 0x51578844 (stale) Mirror 21: 0x51578844 (stale) Mirror 22: 0x51578844 (stale) Mirror 23: 0x51578844 (stale) Mirror 24: 0x51578844 (stale) Mirror 25: 0x51578844 (stale) Mirror 26: 0x51578844 (stale) Mirror 27: 0x51578844 (stale) Mirror 28: 0x51578844 (stale) Mirror 29: 0x51578844 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9c723406 Mirror 30: 0x9c723406 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000979 1790101703.787120 punch from 0x0156a7 to 0x0175f4 (0x1f4e bytes) 000980 1790101703.892591 trunc from 0x02c624 to 0x03696d (0xa34a bytes) 000981 1790101703.900548 trunc from 0x040000 to 0x03b5c2 (0x4a3d bytes) 000982 1790101703.959485 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b5c3): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5c3): Mirror 16: 0xf342f926 (stale) Mirror 20: 0xf342f926 (stale) Mirror 21: 0xf342f926 (stale) Mirror 22: 0xf342f926 (stale) Mirror 23: 0xf342f926 (stale) Mirror 24: 0xf342f926 (stale) Mirror 25: 0xf342f926 (stale) Mirror 26: 0xf342f926 (stale) Mirror 27: 0xf342f926 (stale) Mirror 28: 0xf342f926 (stale) Mirror 29: 0xf342f926 (stale) Mirror 30: 0xda0194a4 (stale) CRC-32 checksum value for chunk [0, 0x3b5c3): Mirror 1: 0xe16233f1 Mirror 31: 0xe16233f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5c3: skipped 000983 1790101705.616387 read 0x02c430 thru 0x03266a (0x623b bytes) 000984 1790101705.635300 mapread 0x02178f thru 0x022f50 (0x17c2 bytes) 000985 1790101705.660610 mapread 0x01536e thru 0x016eaa (0x1b3d bytes) 000986 1790101705.685100 write 0x0287bd thru 0x02cec2 (0x4706 bytes) 000987 1790101705.747606 trunc from 0x02e6ef to 0x03b5c2 (0xced4 bytes) 000988 1790101705.755982 punch from 0x01ce0d to 0x025464 (0x8658 bytes) 000990 1790101705.762180 punch from 0x012778 to 0x0203fe (0xdc87 bytes) 000991 1790101705.767736 mapwrite 0x01efa3 thru 0x02e9f4 (0xfa52 bytes) 000993 1790101705.809976 mapwrite 0x036646 thru 0x03ffff (0x99ba bytes) 000994 1790101705.859984 fallocate 0x022eda thru 0x0291f3 (0x631a bytes) 000995 1790101705.865160 trunc from 0x040000 to 0x037e87 (0x8178 bytes) 000996 1790101705.881583 trunc from 0x024007 to 0x02d9e2 (0x99dc bytes) 000997 1790101705.894038 MIRROR_RESYNC 14 mirrors 000998 1790101710.220880 trunc from 0x037e88 to 0x002033 (0x35e54 bytes) 000999 1790101710.304392 trunc from 0x001b39 to 0x002033 (0x4fb bytes) 001000 1790101710.311805 write 0x0141b0 thru 0x0168a9 (0x26fa bytes) All operations completed A-OK! PASS 70a (167s) == sanity-flr test 71: check mirror extend parent fid ==== 14:28:36 (1790101716) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x212:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 14:28:45 (1790101725) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203113 s, 10.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132446 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 14:28:55 (1790101735) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5636960 .. Extending file size to 5626240 .. Extending file size to 6342784 .. Extending file size to 6893440 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8643328 .. failed Extending file size to 6933216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8254368 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8258304 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8662784 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8564064 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62121 62122 62123 62124 62128 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:30:15 (1790101815) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8319936 .. done Extending file size to 4330688 .. Extending file size to 7536224 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8038656 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9296960 .. failed Extending file size to 8329056 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8378048 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8477184 .. Waiting 63599 63600 63602 63603 63606 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:31:33 (1790101893) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2304 1269000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24760 3582260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18848 3588172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43608 7170432 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2304 1269000 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24760 3582260 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18848 3588172 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43608 7170432 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125156 s, 16.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199416 s, 10.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc2544b46 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:31:52 (1790101912) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146107 s, 14.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 14:31:59 (1790101919) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x143:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x143:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x144:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:32:07 (1790101927) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123807 s, 16.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x243:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x244:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x245:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:32:16 (1790101936) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0787019 s, 26.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x244:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x244:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:32:24 (1790101944) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174477 s, 12.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x145:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x145:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:33:01 (1790101981) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126011 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x246:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x246:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x247:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:33:53 (1790102033) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138725 s, 15.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:34:50 (1790102090) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg228-server: Pool lustre.test_204f created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12602 s, 16.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg228-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:36:10 (1790102170) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:36:17 (1790102177) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:36:25 (1790102185) oleg228-server: Pool lustre.test_206 created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg228-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:36:54 (1790102214) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104615 s, 10.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749151 s, 14.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd584cf70 Mirror 3: 0xd584cf70 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:37:05 (1790102225) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:37:09 (1790102229) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:37:12 (1790102232) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0579463 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:37:46 (1790102266) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0135713 s, 604 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:37:53 (1790102273) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728176 s, 14.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa5 lmm_fid: [0x200000405:0xa5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa5 lmm_fid: [0x200000405:0xa5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611301 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 14:37:55.000000000 -0400 Modify: 2026-09-22 14:37:55.000000000 -0400 Change: 2026-09-22 14:37:56.000000000 -0400 Birth: 2026-09-22 14:37:55.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:38:00 (1790102280) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457515 s, 22.9 MB/s oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg228-server: O/280000401/d15/14f: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:38:22 (1790102302) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.309417 s, 132 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0275316 s, 149 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 22 14:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xaa:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 22 14:38 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:38:31 (1790102311) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:38:41 (1790102321) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1928 sec =========== 14:38:49 (1790102329) === sanity-flr: start cleanup 14:38:51 (1790102331) === === sanity-flr: finish cleanup 14:38:59 (1790102339) ===