-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 15:57:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:57:54 (1787860674) === oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94b904bab800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94b904bab800.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:58:38 (1787860718) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8212] == sanity-flr test 0a: lfs mirror create with -N option == 15:58:41 (1787860721) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:59:00 (1787860740) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:59:03 (1787860743) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:59:06 (1787860746) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:59:32 (1787860772) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:59:35 (1787860775) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:59:39 (1787860779) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94b904bab800.stats=clear osc.lustre-OST0001-osc-ffff94b904bab800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:59:48 (1787860788) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:59:59 (1787860799) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:00:09 (1787860809) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:00:13 (1787860813) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:00:20 (1787860820) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 16:00:31 (1787860831) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:00:41 (1787860841) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:00:51 (1787860851) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:01:00 (1787860860) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:01:10 (1787860870) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:01:18 (1787860878) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.73091 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450083 s, 23.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:01:28 (1787860888) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.207178 s, 20.2 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:01:39 (1787860899) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:01:52 (1787860912) Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:03:07 (1787860987) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.405656 s, 15.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:03:53 (1787861033) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:05:13 (1787861113) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.29674 s, 15.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 16:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.168s user 0m0.002s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.171s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.402s user 0m0.001s sys 0m0.193s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94b904bab800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:06:58 (1787861218) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:07:02 (1787861222) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:07:05 (1787861225) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 16:07:09 (1787861229) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 16:07:18 (1787861238) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.698775 s, 18.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:07:32 (1787861252) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076062 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037957 s, 27.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:07:40 (1787861260) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:07:49 (1787861269) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000974964 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000981985 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155771 s, 657 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00177475 s, 577 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 16:08:00 (1787861280) create test files with size 15738873 .. dbf05e80b3b83711eef23aa68a53e51d - fa8a1251f53c9af21d47ac0f004c49c8 - 1c296c6f2e4039748dacc7e7df56a578 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 16:08:27 (1787861307) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:08:43 (1787861323) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:08:52 (1787861332) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860904 s, 12.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240743 s, 43.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 16:09:02 (1787861342) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10831 s, 19.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.307482 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16189 s, 13.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 16:09:16 (1787861356) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:09:20 (1787861360) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:09:23 (1787861363) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266022 s, 39.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 16:09:31 (1787861371) llite.lustre-ffff94b924479000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 16:09:39 (1787861379) llite.lustre-ffff94b924479000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058017 s, 18.1 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241073 s, 43.5 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 16:09:48 (1787861388) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.273378 s, 11.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:09:58 (1787861398) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:10:08 (1787861408) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.418611 s, 25.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:10:20 (1787861420) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff94b91871c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.475942 s, 22.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff94b91a168800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:10:39 (1787861439) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:11:22 (1787861482) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:12:08 (1787861528) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:12:11 (1787861531) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.307086 s, 10.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.189407 s, 16.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5fe57a8c (nosync,stale) Mirror 2: 0x5fe57a8c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 96bb5a35482aa3e2f821f91e104cd325 - mirror 1 checksum: 5438e48300f835bd5146dd18e04df038 - mirror 2 checksum: 96bb5a35482aa3e2f821f91e104cd325 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:12:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:12:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 96bb5a35482aa3e2f821f91e104cd325 - mirror 1 checksum: 5438e48300f835bd5146dd18e04df038 - mirror 2 checksum: 5438e48300f835bd5146dd18e04df038 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:12:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:12:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:12:23 (1787861543) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336479 s, 31.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:12:32 (1787861552) Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 16:12:45 (1787861565) shut down facet: mds1 facet_host: oleg138-server facet_failover_host: oleg138-server Failover mds1 to oleg138-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 16:13:18 (1787861598) targets are mounted 16:13:18 (1787861598) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393568 s, 26.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:13:39 (1787861619) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0429852 s, 476 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0279972 s, 732 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:13:51 (1787861631) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0591432 s, 346 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0187493 s, 1.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:14:03 (1787861643) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0047781 s, 3.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:14:12 (1787861652) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0508893 s, 402 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:14:21 (1787861661) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0102838 s, 398 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0520826 s, 78.6 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:14:30 (1787861670) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:14:54 (1787861694) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:15:25 (1787861725) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:15:51 (1787861751) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:16:18 (1787861778) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 16:16:28 (1787861788) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58920 mirror split pid 58921 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58920 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58921 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 16:17:36 (1787861856) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 1858 000001 1787861859.117741 mapwrite 0x01c297 thru 0x020123 (0x3e8d bytes) 000002 1787861859.209670 write 0x01cc4e thru 0x026af1 (0x9ea4 bytes) 000004 1787861859.216763 write 0x02c5fc thru 0x035962 (0x9367 bytes) 000007 1787861859.222249 write 0x03e1ce thru 0x03ffff (0x1e32 bytes) 000008 1787861859.224320 read 0x034975 thru 0x03a449 (0x5ad5 bytes) 000009 1787861859.232737 read 0x02c7e1 thru 0x037d19 (0xb539 bytes) 000010 1787861859.233369 punch from 0x02536c to 0x034f84 (0xfc19 bytes) 000011 1787861859.254799 mapwrite 0x002b8f thru 0x00c432 (0x98a4 bytes) 000012 1787861859.310953 trunc from 0x034e75 to 0x03ffff (0xb18b bytes) 000013 1787861859.318745 mapread 0x034578 thru 0x03c3ab (0x7e34 bytes) 000014 1787861859.374347 punch from 0x001078 to 0x00a9a4 (0x992d bytes) 000015 1787861859.393487 punch from 0x02c8b6 to 0x037b17 (0xb262 bytes) 000016 1787861859.404016 punch from 0x00aae4 to 0x013607 (0x8b24 bytes) 000018 1787861859.422902 write 0x0013ad thru 0x003784 (0x23d8 bytes) 000021 1787861859.445934 write 0x002ef0 thru 0x00abd8 (0x7ce9 bytes) 000022 1787861859.452806 trunc from 0x02fba2 to 0x035460 (0x58bf bytes) 000023 1787861859.458656 mapread 0x021489 thru 0x02baad (0xa625 bytes) 000024 1787861859.459656 mapwrite 0x02141b thru 0x0214bf (0xa5 bytes) 000025 1787861859.484103 mapread 0x0063c7 thru 0x009a57 (0x3691 bytes) 000027 1787861859.486424 trunc from 0x023b6d to 0x0275d0 (0x3a64 bytes) fallocating to largest ever: 0x251e1 000029 1787861859.498541 fallocate 0x01bd21 thru 0x0251e0 (0x94c0 bytes) 000030 1787861859.504025 write 0x038ff9 thru 0x03ffff (0x7007 bytes) 000031 1787861859.508977 read 0x039e11 thru 0x03e020 (0x4210 bytes) 000032 1787861859.510383 punch from 0x01d49d to 0x02d294 (0xfdf8 bytes) 000033 1787861859.517905 mapwrite 0x036233 thru 0x0389ee (0x27bc bytes) 000034 1787861859.549847 mapread 0x03c200 thru 0x03ffff (0x3e00 bytes) 000035 1787861859.550725 trunc from 0x0202c7 to 0x0230db (0x2e15 bytes) 000038 1787861859.561811 trunc from 0x011ea7 to 0x014150 (0x22aa bytes) 000039 1787861859.566628 trunc from 0x040000 to 0x01bba5 (0x2445a bytes) 000040 1787861859.595776 mapwrite 0x008736 thru 0x00b9a2 (0x326d bytes) 000041 1787861859.618478 mapread 0x000309 thru 0x00823b (0x7f33 bytes) 000042 1787861859.621095 punch from 0x011398 to 0x015f41 (0x4baa bytes) 000043 1787861859.627072 mapread 0x014488 thru 0x015b3f (0x16b8 bytes) 000044 1787861859.640031 punch from 0x011c5c to 0x0150d8 (0x347d bytes) 000045 1787861859.645672 punch from 0x01674d to 0x01bba5 (0x5459 bytes) fallocating to largest ever: 0x2ec8f 000046 1787861859.651073 fallocate 0x02644f thru 0x02ec8e (0x8840 bytes) 000048 1787861859.658035 trunc from 0x01bba6 to 0x01c2ad (0x708 bytes) 000049 1787861859.676960 fallocate 0x01ef87 thru 0x02d649 (0xe6c3 bytes) 000050 1787861859.683073 mapwrite 0x038605 thru 0x03ffff (0x79fb bytes) 000052 1787861859.826444 read 0x003c73 thru 0x012002 (0xe390 bytes) 000053 1787861859.875933 punch from 0x023cf5 to 0x0316b9 (0xd9c5 bytes) 000054 1787861859.880098 mapwrite 0x033b86 thru 0x03ffff (0xc47a bytes) 000055 1787861859.900868 write 0x002179 thru 0x002eb3 (0xd3b bytes) 000056 1787861859.902004 fallocate 0x0239e9 thru 0x02671d (0x2d35 bytes) 000057 1787861859.907303 trunc from 0x040000 to 0x01277a (0x2d885 bytes) 000058 1787861859.922024 punch from 0x001694 to 0x001e02 (0x76f bytes) 000060 1787861859.932640 mapwrite 0x00e0ec thru 0x01974d (0xb662 bytes) 000061 1787861860.009186 trunc from 0x0160ec to 0x01974d (0x3662 bytes) 000062 1787861860.015240 punch from 0x00ae1e to 0x00b48d (0x670 bytes) 000063 1787861860.022543 mapread 0x012fac thru 0x01974d (0x67a2 bytes) 000064 1787861860.043375 fallocate 0x020255 thru 0x02523c (0x4fe8 bytes) fallocating to largest ever: 0x37d90 000065 1787861860.049993 fallocate 0x02b80d thru 0x037d8f (0xc583 bytes) 000066 1787861860.055858 trunc from 0x037d90 to 0x0206d5 (0x176ba bytes) 000067 1787861860.073974 trunc from 0x0206d6 to 0x000ffe (0x1f6d7 bytes) 000068 1787861860.090449 mapread 0x00076b thru 0x000ffe (0x894 bytes) 000069 1787861860.092450 mapread 0x000364 thru 0x000ffe (0xc9b bytes) 000070 1787861860.094056 fallocate 0x00f6a4 thru 0x01edbe (0xf71b bytes) 000071 1787861860.100026 punch from 0x01dc2f to 0x01edbe (0x1190 bytes) 000073 1787861860.105798 fallocate 0x0044b6 thru 0x00848d (0x3fd8 bytes) 000074 1787861860.113867 MIRROR_EXTEND to 2 mirrors 000075 1787861860.794739 mapwrite 0x01f196 thru 0x02bb24 (0xc98f bytes) 000076 1787861860.905902 trunc from 0x02bb25 to 0x0194fe (0x12626 bytes) 000077 1787861860.936773 MIRROR_RESYNC 2 mirrors 000078 1787861861.661066 fallocate 0x011ce4 thru 0x01224b (0x568 bytes) 000079 1787861861.707449 trunc from 0x00ed0a to 0x010bae (0x1ea5 bytes) fallocating to largest ever: 0x3e7ea 000080 1787861861.713466 fallocate 0x02f871 thru 0x03e7e9 (0xef79 bytes) 000081 1787861861.721497 write 0x0068ad thru 0x010be4 (0xa338 bytes) 000083 1787861861.735928 read 0x030b28 thru 0x036485 (0x595e bytes) 000084 1787861861.757539 trunc from 0x01d7b8 to 0x020e59 (0x36a2 bytes) 000087 1787861861.773429 fallocate 0x00e5fc thru 0x0156ed (0x70f2 bytes) 000088 1787861861.783027 write 0x01f4bd thru 0x0284af (0x8ff3 bytes) 000089 1787861861.797650 write 0x00f239 thru 0x01e177 (0xef3f bytes) 000090 1787861861.805558 write 0x0217c4 thru 0x030a54 (0xf291 bytes) 000091 1787861861.808494 trunc from 0x03e7ea to 0x01e8b5 (0x1ff34 bytes) 000092 1787861861.837373 fallocate 0x016d72 thru 0x017678 (0x907 bytes) 000093 1787861861.847080 trunc from 0x01e8b6 to 0x03e7a7 (0x1fef2 bytes) 000094 1787861861.869465 trunc from 0x0051cd to 0x014eab (0xfcdf bytes) 000095 1787861861.902637 punch from 0x001855 to 0x00fb76 (0xe322 bytes) 000096 1787861861.917082 punch from 0x020038 to 0x0202be (0x287 bytes) 000097 1787861861.925223 write 0x00973c thru 0x01862c (0xeef1 bytes) 000099 1787861861.942085 fallocate 0x033592 thru 0x0362f2 (0x2d61 bytes) 000100 1787861861.952910 mapwrite 0x016e71 thru 0x01fc74 (0x8e04 bytes) 000101 1787861862.003251 write 0x009fb3 thru 0x00f802 (0x5850 bytes) 000102 1787861862.004570 read 0x02be5e thru 0x02e3e7 (0x258a bytes) 000103 1787861862.054730 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e7a8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7a8): Mirror 2: 0xfa0c4887 (stale) CRC-32 checksum value for chunk [0, 0x3e7a8): Mirror 1: 0x88696603 Mirror 3: 0x88696603 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7a8: skipped 000104 1787861862.831730 mapwrite 0x01a6e6 thru 0x028286 (0xdba1 bytes) 000105 1787861862.967185 fallocate 0x026ba7 thru 0x02948a (0x28e4 bytes) 000106 1787861862.975421 trunc from 0x03637d to 0x03c7e4 (0x6468 bytes) 000107 1787861862.981965 write 0x02ce3b thru 0x031af3 (0x4cb9 bytes) 000108 1787861862.990003 punch from 0x03b8fd to 0x03bb3f (0x243 bytes) 000110 1787861862.995004 fallocate 0x006faf thru 0x00e5ad (0x75ff bytes) 000111 1787861863.000285 punch from 0x00f1d9 to 0x011038 (0x1e60 bytes) 000113 1787861863.010461 read 0x039d4d thru 0x03e7a7 (0x4a5b bytes) fallocating to largest ever: 0x40000 000114 1787861863.021051 fallocate 0x03e36b thru 0x03ffff (0x1c95 bytes) 000115 1787861863.026165 fallocate 0x014af5 thru 0x017320 (0x282c bytes) 000116 1787861863.032860 mapwrite 0x021363 thru 0x030f2d (0xfbcb bytes) 000117 1787861863.051241 mapread 0x02eebb thru 0x03d302 (0xe448 bytes) 000118 1787861863.056052 read 0x017ed5 thru 0x01cba7 (0x4cd3 bytes) 000119 1787861863.056446 punch from 0x01c493 to 0x0229d8 (0x6546 bytes) 000120 1787861863.060680 trunc from 0x040000 to 0x010001 (0x2fffe bytes) 000121 1787861863.099417 mapread 0x00e8fb thru 0x010001 (0x1707 bytes) 000122 1787861863.111938 punch from 0x007f01 to 0x008772 (0x872 bytes) 000123 1787861863.116382 mapwrite 0x03c50b thru 0x03ffff (0x3af5 bytes) 000124 1787861863.173209 trunc from 0x008f97 to 0x0184cd (0xf537 bytes) 000125 1787861863.180902 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x4503010d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x28075749 Mirror 4: 0x28075749 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000126 1787861864.191476 write 0x028722 thru 0x0290ad (0x98c bytes) 000127 1787861864.263614 punch from 0x0194dc to 0x022c81 (0x97a6 bytes) 000129 1787861864.268276 write 0x03df43 thru 0x03ffff (0x20bd bytes) 000130 1787861864.291970 mapwrite 0x0358e9 thru 0x03ffff (0xa717 bytes) 000131 1787861864.339424 trunc from 0x040000 to 0x03ce78 (0x3187 bytes) 000132 1787861864.386839 read 0x021802 thru 0x02ab7e (0x937d bytes) 000133 1787861864.395721 fallocate 0x00c3f8 thru 0x00d9a9 (0x15b2 bytes) 000134 1787861864.400201 write 0x01c69c thru 0x01ef45 (0x28aa bytes) 000135 1787861864.408351 trunc from 0x03bdb0 to 0x03ce78 (0x10c9 bytes) 000137 1787861864.414216 trunc from 0x03ce79 to 0x03b722 (0x1756 bytes) 000138 1787861864.429299 punch from 0x038208 to 0x03b722 (0x351b bytes) 000139 1787861864.433467 fallocate 0x032e1e thru 0x033ab9 (0xc9c bytes) 000141 1787861864.440036 read 0x031d1d thru 0x03b722 (0x9a06 bytes) 000143 1787861864.456597 trunc from 0x00b57e to 0x00d54b (0x1fce bytes) 000144 1787861864.460968 fallocate 0x016a5a thru 0x01e9f5 (0x7f9c bytes) 000145 1787861864.471494 trunc from 0x03b723 to 0x01393c (0x27de6 bytes) 000146 1787861864.484403 trunc from 0x00274a to 0x00a1ca (0x7a81 bytes) 000147 1787861864.488801 punch from 0x013108 to 0x01393c (0x835 bytes) 000149 1787861864.496021 trunc from 0x01393d to 0x009997 (0x9fa5 bytes) 000150 1787861864.510417 read 0x007892 thru 0x009997 (0x2106 bytes) 000151 1787861864.525885 mapread 0x004ec1 thru 0x009997 (0x4ad7 bytes) 000152 1787861864.527265 mapread 0x000823 thru 0x008a71 (0x824f bytes) 000153 1787861864.528227 punch from 0x006724 to 0x009997 (0x3274 bytes) 000154 1787861864.534271 fallocate 0x00c10d thru 0x00eaf2 (0x29e6 bytes) 000155 1787861864.539435 fallocate 0x037dc7 thru 0x03ffff (0x8239 bytes) 000156 1787861864.545462 read 0x001c3d thru 0x006d8b (0x514f bytes) 000157 1787861864.552392 punch from 0x00632f to 0x009997 (0x3669 bytes) 000158 1787861864.557617 mapread 0x001fc6 thru 0x009997 (0x79d2 bytes) 000159 1787861864.580067 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x9998): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9998): Mirror 2: 0x8fd52d0c (stale) Mirror 3: 0x61d39157 (stale) Mirror 4: 0x8fd52d0c (stale) CRC-32 checksum value for chunk [0, 0x9998): Mirror 1: 0x8fd52d0c Mirror 5: 0x8fd52d0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9998: skipped 000160 1787861865.478909 trunc from 0x009998 to 0x018cd9 (0xf342 bytes) 000161 1787861865.581469 trunc from 0x018cda to 0x02ddd8 (0x150ff bytes) 000162 1787861865.603585 trunc from 0x0279eb to 0x02a6e6 (0x2cfc bytes) 000163 1787861865.611893 read 0x027ad8 thru 0x02ddd8 (0x6301 bytes) 000164 1787861865.627796 fallocate 0x030cce thru 0x03ffff (0xf332 bytes) 000166 1787861865.636487 trunc from 0x040000 to 0x019448 (0x26bb7 bytes) 000167 1787861865.672091 mapwrite 0x02fac8 thru 0x03bf5b (0xc494 bytes) 000168 1787861865.718215 trunc from 0x01d3b6 to 0x02c40a (0xf055 bytes) 000169 1787861865.721632 punch from 0x01b9d1 to 0x026fb9 (0xb5e9 bytes) 000170 1787861865.724928 punch from 0x01ef2f to 0x02d029 (0xe0fb bytes) 000171 1787861865.728313 mapread 0x0235da thru 0x032152 (0xeb79 bytes) 000172 1787861865.750419 mapread 0x02dbf6 thru 0x032217 (0x4622 bytes) 000175 1787861865.758831 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf5c): Mirror 2: 0xe6a7bc35 (stale) Mirror 4: 0xe6a7bc35 (stale) Mirror 5: 0xe6a7bc35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf5c): Mirror 2: 0xe6a7bc35 (stale) Mirror 4: 0xe6a7bc35 (stale) Mirror 5: 0xe6a7bc35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf5c: skipped 000176 1787861866.162904 punch from 0x016370 to 0x01dc96 (0x7927 bytes) 000177 1787861866.171843 trunc from 0x03bf5c to 0x031e37 (0xa124 bytes) 000178 1787861866.190464 write 0x0332d0 thru 0x039a2e (0x675f bytes) 000180 1787861866.194158 punch from 0x033bfc to 0x039a2e (0x5e33 bytes) 000181 1787861866.223645 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a2f): Mirror 4: 0x6421de9d (stale) Mirror 5: 0x6421de9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a2f): Mirror 4: 0x6421de9d (stale) Mirror 5: 0x6421de9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a2f: skipped 000183 1787861866.603771 write 0x00006b thru 0x00bc99 (0xbc2f bytes) 000184 1787861866.626165 mapread 0x0172cf thru 0x01a102 (0x2e34 bytes) 000185 1787861866.648026 read 0x031b9d thru 0x032bab (0x100f bytes) 000187 1787861866.655324 punch from 0x036739 to 0x0399d3 (0x329b bytes) 000188 1787861866.661465 trunc from 0x039a2f to 0x03dec1 (0x4493 bytes) 000189 1787861866.676777 read 0x0273f4 thru 0x02fc42 (0x884f bytes) 000190 1787861866.706748 mapwrite 0x01da61 thru 0x029474 (0xba14 bytes) 000191 1787861866.747879 fallocate 0x03f94b thru 0x03ffff (0x6b5 bytes) 000192 1787861866.759051 mapread 0x0262c3 thru 0x02e8a3 (0x85e1 bytes) 000193 1787861866.767418 trunc from 0x020a3f to 0x02dc52 (0xd214 bytes) 000194 1787861866.778916 mapwrite 0x02a519 thru 0x03a455 (0xff3d bytes) 000195 1787861866.873430 mapwrite 0x0053f8 thru 0x0084cd (0x30d6 bytes) 000196 1787861866.923323 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 4: 0x9e42c795 (stale) Mirror 5: 0x4a928a51 (stale) CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 1: 0x87bb2b6a Mirror 6: 0x87bb2b6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dec2: skipped 000197 1787861868.195894 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 4: 0x9e42c795 (stale) Mirror 5: 0x4a928a51 (stale) CRC-32 checksum value for chunk [0, 0x3dec2): Mirror 1: 0x87bb2b6a Mirror 6: 0x87bb2b6a Mirror 7: 0x87bb2b6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dec2: skipped 000198 1787861869.510153 trunc from 0x03dec2 to 0x017768 (0x26759 bytes) 000199 1787861869.593908 trunc from 0x017769 to 0x00dfe4 (0x9784 bytes) 000200 1787861869.616035 punch from 0x0096f6 to 0x00dfe4 (0x48ef bytes) 000201 1787861869.624835 write 0x004550 thru 0x00bb9d (0x764e bytes) 000202 1787861869.648054 read 0x008ee8 thru 0x00dfe4 (0x50fd bytes) 000203 1787861869.654836 write 0x026693 thru 0x027583 (0xef1 bytes) 000204 1787861869.657284 punch from 0x01df99 to 0x0243dc (0x6444 bytes) 000205 1787861869.662199 mapwrite 0x000f3e thru 0x0047ab (0x386e bytes) 000206 1787861869.703803 trunc from 0x017367 to 0x01b2f2 (0x3f8c bytes) 000208 1787861869.714710 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27584): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27584): Mirror 4: 0x7ec3b5f7 (stale) Mirror 5: 0x7ec3b5f7 (stale) Mirror 6: 0xbd53211f (stale) Mirror 7: 0xbd53211f (stale) CRC-32 checksum value for chunk [0, 0x27584): Mirror 1: 0x6bcb4908 Mirror 8: 0x6bcb4908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27584: skipped 000209 1787861870.600311 read 0x016ff1 thru 0x01865e (0x166e bytes) 000210 1787861870.607451 write 0x03d242 thru 0x03ffff (0x2dbe bytes) 000211 1787861870.646450 trunc from 0x040000 to 0x017f74 (0x2808b bytes) 000212 1787861870.665278 fallocate 0x015ffb thru 0x022460 (0xc466 bytes) 000213 1787861870.692404 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17f75): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f75): Mirror 4: 0x2d9430a6 (stale) Mirror 5: 0x2d9430a6 (stale) Mirror 6: 0xcee2c483 (stale) Mirror 7: 0xcee2c483 (stale) Mirror 8: 0xc35b0c92 (stale) CRC-32 checksum value for chunk [0, 0x17f75): Mirror 1: 0xc35b0c92 Mirror 9: 0xc35b0c92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f75: skipped 000214 1787861871.489113 mapwrite 0x025116 thru 0x034a5c (0xf947 bytes) 000215 1787861871.593655 mapread 0x008e1b thru 0x00e6f7 (0x58dd bytes) 000216 1787861871.603303 mapwrite 0x00901d thru 0x0119aa (0x898e bytes) 000217 1787861871.627406 write 0x029369 thru 0x03217c (0x8e14 bytes) 000218 1787861871.629291 mapwrite 0x00d448 thru 0x01cd1c (0xf8d5 bytes) 000219 1787861871.659353 trunc from 0x025d4a to 0x02ba4f (0x5d06 bytes) 000220 1787861871.664410 mapwrite 0x0359fc thru 0x03ffff (0xa604 bytes) 000221 1787861871.716170 trunc from 0x040000 to 0x002352 (0x3dcad bytes) 000222 1787861871.737821 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2353): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2353): Mirror 4: 0x86a140af (stale) Mirror 5: 0x86a140af (stale) Mirror 6: 0x7b146450 (stale) Mirror 7: 0x7b146450 (stale) Mirror 8: 0x5796c19 (stale) Mirror 9: 0x5796c19 (stale) CRC-32 checksum value for chunk [0, 0x2353): Mirror 1: 0x5796c19 Mirror 10: 0x5796c19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2353: skipped 000223 1787861872.783993 write 0x03ce64 thru 0x03ffff (0x319c bytes) 000224 1787861872.834173 mapread 0x03e339 thru 0x03ffff (0x1cc7 bytes) 000225 1787861872.835325 read 0x0297e2 thru 0x0377f8 (0xe017 bytes) 000226 1787861872.848432 write 0x036489 thru 0x037eb7 (0x1a2f bytes) 000227 1787861872.849750 trunc from 0x02f458 to 0x036ffa (0x7ba3 bytes) 000228 1787861872.867274 read 0x028822 thru 0x03783d (0xf01c bytes) 000229 1787861872.905889 fallocate 0x01222a thru 0x01e376 (0xc14d bytes) 000230 1787861872.910639 read 0x026157 thru 0x02eb87 (0x8a31 bytes) 000231 1787861872.911318 trunc from 0x040000 to 0x01a8db (0x25724 bytes) 000232 1787861872.935352 mapwrite 0x03277d thru 0x035884 (0x3108 bytes) 000233 1787861873.014836 mapread 0x00646a thru 0x009f54 (0x3aeb bytes) 000234 1787861873.017351 mapread 0x0138f5 thru 0x015e70 (0x257c bytes) 000235 1787861873.020397 fallocate 0x00ce5a thru 0x0112bc (0x4463 bytes) 000237 1787861873.038776 write 0x03a68d thru 0x03ffff (0x5973 bytes) 000238 1787861873.041316 write 0x02e694 thru 0x038e5e (0xa7cb bytes) 000239 1787861873.053646 read 0x022cf6 thru 0x022de7 (0xf2 bytes) 000240 1787861873.062360 mapwrite 0x028f47 thru 0x02f215 (0x62cf bytes) 000241 1787861873.093497 MIRROR_RESYNC 8 mirrors 000243 1787861875.050839 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000245 1787861875.325594 trunc from 0x02b5f2 to 0x032ffc (0x7a0b bytes) 000246 1787861875.366186 fallocate 0x035b50 thru 0x039ba2 (0x4053 bytes) 000247 1787861875.372025 trunc from 0x040000 to 0x01a704 (0x258fb bytes) 000248 1787861875.394228 mapwrite 0x028db3 thru 0x036b44 (0xdd92 bytes) 000249 1787861875.455133 mapwrite 0x03f683 thru 0x03ffff (0x97d bytes) 000250 1787861875.484047 mapwrite 0x038d6f thru 0x03ffff (0x7291 bytes) 000251 1787861875.500319 mapread 0x0245e7 thru 0x025c0c (0x1626 bytes) 000252 1787861875.501302 write 0x03f2e1 thru 0x03ffff (0xd1f bytes) 000253 1787861875.502873 trunc from 0x040000 to 0x03ad78 (0x5287 bytes) 000254 1787861875.518429 punch from 0x01f2d4 to 0x02e357 (0xf084 bytes) 000255 1787861875.527087 write 0x0363ba thru 0x03d4a6 (0x70ed bytes) 000256 1787861875.528317 punch from 0x02c077 to 0x032f22 (0x6eac bytes) 000258 1787861875.535586 fallocate 0x0138ea thru 0x01abc9 (0x72e0 bytes) 000259 1787861875.546675 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4a7): Mirror 4: 0x4aa266e0 (stale) Mirror 6: 0x4aa266e0 (stale) Mirror 7: 0x4aa266e0 (stale) Mirror 8: 0x4aa266e0 (stale) Mirror 10: 0x4aa266e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4a7): Mirror 4: 0x4aa266e0 (stale) Mirror 6: 0x4aa266e0 (stale) Mirror 7: 0x4aa266e0 (stale) Mirror 8: 0x4aa266e0 (stale) Mirror 10: 0x4aa266e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4a7: skipped 000260 1787861876.077992 trunc from 0x009f34 to 0x00a67b (0x748 bytes) 000261 1787861876.089277 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d4a7): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4a7): Mirror 4: 0x4aa266e0 (stale) Mirror 6: 0x4aa266e0 (stale) Mirror 7: 0x4aa266e0 (stale) Mirror 8: 0x4aa266e0 (stale) Mirror 10: 0x4aa266e0 (stale) CRC-32 checksum value for chunk [0, 0x3d4a7): Mirror 1: 0x7a8be6b5 Mirror 11: 0x7a8be6b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4a7: skipped 000262 1787861877.380046 trunc from 0x03a122 to 0x03d4a6 (0x3385 bytes) 000264 1787861877.472871 write 0x00c49b thru 0x01b9ac (0xf512 bytes) 000265 1787861877.517558 write 0x02cf63 thru 0x039b93 (0xcc31 bytes) 000266 1787861877.539412 trunc from 0x03d4a7 to 0x026fd1 (0x164d5 bytes) 000267 1787861877.581182 punch from 0x020184 to 0x026fd1 (0x6e4e bytes) 000268 1787861877.587398 mapwrite 0x021cd0 thru 0x026c45 (0x4f76 bytes) 000271 1787861877.713069 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x26fd2): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26fd2): Mirror 4: 0x99cc0929 (stale) Mirror 6: 0x99cc0929 (stale) Mirror 7: 0x99cc0929 (stale) Mirror 8: 0x99cc0929 (stale) Mirror 10: 0x99cc0929 (stale) Mirror 11: 0x99cc0929 (stale) CRC-32 checksum value for chunk [0, 0x26fd2): Mirror 1: 0x7286a6df Mirror 12: 0x7286a6df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26fd2: skipped 000272 1787861879.101786 fallocate 0x0271d8 thru 0x02e9d9 (0x7802 bytes) 000273 1787861879.138963 trunc from 0x01eb9f to 0x026fd1 (0x8433 bytes) 000274 1787861879.145088 read 0x011e48 thru 0x01247f (0x638 bytes) 000275 1787861879.149229 mapread 0x01121b thru 0x019000 (0x7de6 bytes) 000276 1787861879.173547 read 0x001a8a thru 0x00af58 (0x94cf bytes) 000277 1787861879.190360 fallocate 0x006b87 thru 0x008679 (0x1af3 bytes) 000278 1787861879.194745 fallocate 0x02dd12 thru 0x033561 (0x5850 bytes) 000279 1787861879.202960 MIRROR_RESYNC 8 mirrors 000280 1787861880.536130 mapread 0x01b966 thru 0x01f6dd (0x3d78 bytes) 000281 1787861880.551562 fallocate 0x014da4 thru 0x021bfb (0xce58 bytes) 000282 1787861880.591084 punch from 0x014259 to 0x01a93b (0x66e3 bytes) 000283 1787861880.596200 write 0x027a44 thru 0x02809e (0x65b bytes) 000284 1787861880.597208 write 0x02f731 thru 0x03ec3b (0xf50b bytes) 000286 1787861880.600613 trunc from 0x023004 to 0x031eb8 (0xeeb5 bytes) 000287 1787861880.617047 write 0x008d49 thru 0x0182ff (0xf5b7 bytes) 000288 1787861880.629211 trunc from 0x03ec3c to 0x006612 (0x38629 bytes) 000289 1787861880.650968 trunc from 0x006613 to 0x00b707 (0x50f5 bytes) 000290 1787861880.669053 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb708): Mirror 6: 0x7e2d01c0 (stale) Mirror 7: 0x7e2d01c0 (stale) Mirror 8: 0x7e2d01c0 (stale) Mirror 10: 0x7e2d01c0 (stale) Mirror 11: 0x7e2d01c0 (stale) Mirror 12: 0x7e2d01c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb708): Mirror 6: 0x7e2d01c0 (stale) Mirror 7: 0x7e2d01c0 (stale) Mirror 8: 0x7e2d01c0 (stale) Mirror 10: 0x7e2d01c0 (stale) Mirror 11: 0x7e2d01c0 (stale) Mirror 12: 0x7e2d01c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb708: skipped 000291 1787861881.011860 mapread 0x004e36 thru 0x00b707 (0x68d2 bytes) 000292 1787861881.031445 fallocate 0x036ae7 thru 0x03f2fe (0x8818 bytes) 000293 1787861881.038531 write 0x02c651 thru 0x03680a (0xa1ba bytes) 000294 1787861881.052925 mapread 0x02710d thru 0x0333ff (0xc2f3 bytes) 000295 1787861881.064869 read 0x012d18 thru 0x01b349 (0x8632 bytes) 000297 1787861881.089049 MIRROR_RESYNC 7 mirrors 000298 1787861883.428136 mapwrite 0x0329c2 thru 0x03ffff (0xd63e bytes) 000299 1787861883.744417 punch from 0x03658c to 0x039913 (0x3388 bytes) 000300 1787861883.757437 mapread 0x033321 thru 0x03b071 (0x7d51 bytes) 000301 1787861883.785357 read 0x0313fe thru 0x03ffff (0xec02 bytes) 000302 1787861883.793341 fallocate 0x02d205 thru 0x02de45 (0xc41 bytes) 000303 1787861883.806466 write 0x026e3e thru 0x0358a2 (0xea65 bytes) 000304 1787861883.845592 trunc from 0x02cbee to 0x033745 (0x6b58 bytes) 000306 1787861883.911591 write 0x02e1b3 thru 0x02ec67 (0xab5 bytes) 000307 1787861883.919526 read 0x00daf7 thru 0x0187b8 (0xacc2 bytes) 000308 1787861883.967679 write 0x02978f thru 0x02c3c2 (0x2c34 bytes) 000309 1787861883.970720 punch from 0x03c106 to 0x03ffff (0x3efa bytes) 000310 1787861883.999474 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x4d1f4f75 (stale) Mirror 8: 0x4d1f4f75 (stale) Mirror 10: 0x4d1f4f75 (stale) Mirror 11: 0x4d1f4f75 (stale) Mirror 12: 0x4d1f4f75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x4d1f4f75 (stale) Mirror 8: 0x4d1f4f75 (stale) Mirror 10: 0x4d1f4f75 (stale) Mirror 11: 0x4d1f4f75 (stale) Mirror 12: 0x4d1f4f75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000313 1787861884.653082 fallocate 0x031b58 thru 0x032c71 (0x111a bytes) 000314 1787861884.664147 mapread 0x00caae thru 0x010eb8 (0x440b bytes) 000315 1787861884.782986 mapread 0x0162dc thru 0x025c2e (0xf953 bytes) 000316 1787861884.861251 trunc from 0x040000 to 0x033225 (0xcdda bytes) 000317 1787861884.905238 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33226): Mirror 7: 0x47e23f19 (stale) Mirror 10: 0x47e23f19 (stale) Mirror 11: 0x47e23f19 (stale) Mirror 12: 0x47e23f19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33226): Mirror 7: 0x47e23f19 (stale) Mirror 10: 0x47e23f19 (stale) Mirror 11: 0x47e23f19 (stale) Mirror 12: 0x47e23f19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33226: skipped 000318 1787861885.823337 trunc from 0x033226 to 0x0036bf (0x2fb66 bytes) 000319 1787861885.851014 trunc from 0x0036c0 to 0x035968 (0x322a9 bytes) 000320 1787861885.898498 trunc from 0x035969 to 0x00f51b (0x2644d bytes) 000321 1787861885.938761 punch from 0x00f021 to 0x00f51b (0x4fb bytes) 000322 1787861885.949179 mapwrite 0x00ccfe thru 0x018732 (0xba35 bytes) 000323 1787861886.112028 trunc from 0x018733 to 0x01dead (0x577b bytes) 000324 1787861886.143097 fallocate 0x033324 thru 0x0377dc (0x44b9 bytes) 000325 1787861886.151345 trunc from 0x01587c to 0x01bd61 (0x64e6 bytes) 000326 1787861886.170326 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1deae): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1deae): Mirror 7: 0x75634df2 (stale) Mirror 10: 0x75634df2 (stale) Mirror 11: 0x75634df2 (stale) Mirror 12: 0x75634df2 (stale) CRC-32 checksum value for chunk [0, 0x1deae): Mirror 1: 0xa5cf87b2 Mirror 13: 0xa5cf87b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1deae: skipped 000328 1787861886.910193 trunc from 0x006361 to 0x012214 (0xbeb4 bytes) 000329 1787861886.950225 mapwrite 0x03de3b thru 0x03ffff (0x21c5 bytes) 000330 1787861886.997214 punch from 0x039bf2 to 0x03ffff (0x640e bytes) 000331 1787861887.001780 trunc from 0x040000 to 0x038f55 (0x70aa bytes) 000332 1787861887.021423 punch from 0x00b289 to 0x0127dc (0x7554 bytes) 000333 1787861887.027598 trunc from 0x038f56 to 0x038114 (0xe41 bytes) 000334 1787861887.050864 write 0x024c12 thru 0x029346 (0x4735 bytes) 000335 1787861887.066754 punch from 0x033298 to 0x038114 (0x4e7d bytes) 000336 1787861887.077119 read 0x01cee7 thru 0x02b64c (0xe766 bytes) 000337 1787861887.094743 mapwrite 0x036bd1 thru 0x03ffff (0x942f bytes) 000338 1787861887.154184 read 0x02c1b4 thru 0x036f4b (0xad98 bytes) 000339 1787861887.155162 mapread 0x02b2ab thru 0x02c101 (0xe57 bytes) 000340 1787861887.157974 mapread 0x018413 thru 0x01ebea (0x67d8 bytes) 000341 1787861887.175082 trunc from 0x040000 to 0x003a3d (0x3c5c2 bytes) 000342 1787861887.193921 trunc from 0x003a3e to 0x00ae99 (0x745c bytes) 000343 1787861887.218589 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae9a): Mirror 10: 0xb9d181f1 (stale) Mirror 11: 0xb9d181f1 (stale) Mirror 12: 0xb9d181f1 (stale) Mirror 13: 0xb9d181f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae9a): Mirror 10: 0xb9d181f1 (stale) Mirror 11: 0xb9d181f1 (stale) Mirror 12: 0xb9d181f1 (stale) Mirror 13: 0xb9d181f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae9a: skipped 000345 1787861887.659351 read 0x007fcf thru 0x00822c (0x25e bytes) 000346 1787861887.667124 mapread 0x009a76 thru 0x00ae99 (0x1424 bytes) 000347 1787861887.680813 read 0x003020 thru 0x00a841 (0x7822 bytes) 000348 1787861887.691410 punch from 0x007f41 to 0x00a879 (0x2939 bytes) 000349 1787861887.702912 MIRROR_RESYNC 5 mirrors 000350 1787861888.801751 write 0x034ad8 thru 0x03b08c (0x65b5 bytes) 000351 1787861888.839279 write 0x00478c thru 0x01096b (0xc1e0 bytes) 000352 1787861888.852450 write 0x019b38 thru 0x023e7a (0xa343 bytes) 000353 1787861888.870012 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b08d): Mirror 10: 0x727cd6bf (stale) Mirror 12: 0x727cd6bf (stale) Mirror 13: 0x727cd6bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b08d): Mirror 10: 0x727cd6bf (stale) Mirror 12: 0x727cd6bf (stale) Mirror 13: 0x727cd6bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b08d: skipped 000354 1787861889.242494 fallocate 0x03eadb thru 0x03ffff (0x1525 bytes) 000355 1787861889.249144 trunc from 0x03b08d to 0x0026ea (0x389a2 bytes) 000356 1787861889.281159 mapwrite 0x037ee6 thru 0x03f93b (0x7a56 bytes) 000357 1787861889.374484 trunc from 0x03f93c to 0x039344 (0x65f7 bytes) 000359 1787861889.395117 trunc from 0x039345 to 0x02eec3 (0xa481 bytes) 000360 1787861889.415900 read 0x019ade thru 0x01fffd (0x6520 bytes) 000361 1787861889.423449 punch from 0x017efe to 0x01868f (0x792 bytes) 000362 1787861889.426498 punch from 0x00991f to 0x0126b3 (0x8d95 bytes) 000363 1787861889.430461 read 0x0255fc thru 0x02d823 (0x8228 bytes) 000364 1787861889.443415 read 0x014f14 thru 0x01f775 (0xa862 bytes) 000366 1787861889.445070 write 0x00da5c thru 0x012d85 (0x532a bytes) 000367 1787861889.446985 mapwrite 0x03f02d thru 0x03ffff (0xfd3 bytes) 000368 1787861889.488558 mapread 0x033c82 thru 0x03c4ab (0x882a bytes) 000369 1787861889.489569 punch from 0x026bd2 to 0x02ac99 (0x40c8 bytes) 000370 1787861889.494932 fallocate 0x026cfd thru 0x028cbb (0x1fbf bytes) 000371 1787861889.500302 mapwrite 0x012bff thru 0x0178d5 (0x4cd7 bytes) 000372 1787861889.519039 fallocate 0x025d30 thru 0x0359fe (0xfccf bytes) 000373 1787861889.524264 fallocate 0x031a2c thru 0x033335 (0x190a bytes) 000374 1787861889.530425 fallocate 0x006184 thru 0x00999f (0x381c bytes) 000375 1787861889.536384 mapread 0x03ae96 thru 0x03dc71 (0x2ddc bytes) 000376 1787861889.542735 MIRROR_RESYNC 4 mirrors 000377 1787861890.735950 trunc from 0x040000 to 0x01e19f (0x21e60 bytes) 000378 1787861890.796243 mapread 0x00a2be thru 0x01333f (0x9082 bytes) 000379 1787861890.845799 read 0x012869 thru 0x0153d3 (0x2b6b bytes) 000382 1787861890.875261 punch from 0x016ca6 to 0x01afb9 (0x4314 bytes) 000383 1787861890.921019 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e1a0): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e1a0): Mirror 10: 0xaea9ba17 (stale) Mirror 12: 0xaea9ba17 (stale) Mirror 13: 0xaea9ba17 (stale) CRC-32 checksum value for chunk [0, 0x1e1a0): Mirror 1: 0xbe5f0b03 Mirror 14: 0xbe5f0b03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e1a0: skipped 000384 1787861892.607404 punch from 0x0058fb to 0x00fc6a (0xa370 bytes) 000385 1787861892.671449 fallocate 0x01e0bb thru 0x02613c (0x8082 bytes) 000386 1787861892.683649 trunc from 0x02613d to 0x035ba2 (0xfa66 bytes) 000387 1787861892.733934 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35ba3): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ba3): Mirror 10: 0xf33452c3 (stale) Mirror 12: 0xf33452c3 (stale) Mirror 13: 0xf33452c3 (stale) Mirror 14: 0xfed78c62 (stale) CRC-32 checksum value for chunk [0, 0x35ba3): Mirror 1: 0xcdce04d7 Mirror 15: 0xcdce04d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ba3: skipped 000388 1787861893.954139 MIRROR_RESYNC 6 mirrors 000389 1787861895.501384 trunc from 0x01a1f4 to 0x0257c4 (0xb5d1 bytes) 000390 1787861895.530357 punch from 0x0164b3 to 0x01bcd4 (0x5822 bytes) 000391 1787861895.534876 punch from 0x02fab4 to 0x035ba2 (0x60ef bytes) 000392 1787861895.544663 MIRROR_RESYNC 6 mirrors 000393 1787861896.666314 write 0x01f0a5 thru 0x023034 (0x3f90 bytes) 000394 1787861896.700490 mapread 0x033894 thru 0x035ba2 (0x230f bytes) 000395 1787861896.714848 read 0x028952 thru 0x032cff (0xa3ae bytes) 000396 1787861896.721967 read 0x031412 thru 0x035ba2 (0x4791 bytes) 000397 1787861896.722759 trunc from 0x035ba3 to 0x011b4f (0x24053 bytes) 000398 1787861896.738353 mapread 0x004730 thru 0x00a251 (0x5b22 bytes) 000399 1787861896.749368 fallocate 0x01bae7 thru 0x027b91 (0xc0ab bytes) 000400 1787861896.754241 write 0x0188f7 thru 0x0202b4 (0x79be bytes) 000401 1787861896.755409 trunc from 0x015a0c to 0x01c6be (0x6cb3 bytes) 000402 1787861896.769268 mapread 0x0155e3 thru 0x0202b4 (0xacd2 bytes) 000403 1787861896.779782 mapwrite 0x029b14 thru 0x0365d7 (0xcac4 bytes) 000404 1787861896.829911 write 0x03fe6c thru 0x03ffff (0x194 bytes) 000405 1787861896.831086 trunc from 0x00d5f6 to 0x00fae1 (0x24ec bytes) 000406 1787861896.838410 mapwrite 0x02bd55 thru 0x0336a8 (0x7954 bytes) 000407 1787861896.860663 read 0x032613 thru 0x03c580 (0x9f6e bytes) 000408 1787861896.870886 trunc from 0x040000 to 0x012ec0 (0x2d13f bytes) 000409 1787861896.887032 write 0x002ac0 thru 0x00fc01 (0xd142 bytes) 000410 1787861896.888391 punch from 0x0032f4 to 0x00f09c (0xbda9 bytes) 000411 1787861896.901993 trunc from 0x012ec1 to 0x01b94a (0x8a8a bytes) 000412 1787861896.914308 trunc from 0x01b94b to 0x012e78 (0x8ad2 bytes) 000413 1787861896.930218 punch from 0x00aa42 to 0x012e78 (0x8437 bytes) 000415 1787861896.935030 fallocate 0x01ca87 thru 0x02a4c1 (0xda3b bytes) 000416 1787861896.939001 mapwrite 0x001555 thru 0x0069c7 (0x5473 bytes) 000417 1787861896.970257 trunc from 0x011a1c to 0x018786 (0x6d6b bytes) 000418 1787861896.974917 read 0x010fef thru 0x01c2ab (0xb2bd bytes) 000419 1787861896.995052 fallocate 0x0294ea thru 0x036764 (0xd27b bytes) 000420 1787861897.007043 trunc from 0x026706 to 0x031a13 (0xb30e bytes) 000421 1787861897.028727 mapread 0x01f8e8 thru 0x026c0d (0x7326 bytes) 000422 1787861897.043908 write 0x03e482 thru 0x03ffff (0x1b7e bytes) 000423 1787861897.047022 write 0x007b60 thru 0x016d65 (0xf206 bytes) 000424 1787861897.049864 write 0x00deeb thru 0x01ada3 (0xceb9 bytes) 000426 1787861897.051750 fallocate 0x0037fa thru 0x006db7 (0x35be bytes) 000427 1787861897.059723 punch from 0x012c9c to 0x01e3d3 (0xb738 bytes) 000428 1787861897.089150 mapwrite 0x00dc2d thru 0x00eff5 (0x13c9 bytes) 000429 1787861897.099579 write 0x00fb6a thru 0x018595 (0x8a2c bytes) 000430 1787861897.107293 mapwrite 0x035c1c thru 0x03ffff (0xa3e4 bytes) 000431 1787861897.137899 punch from 0x02fcc2 to 0x02fe56 (0x195 bytes) 000432 1787861897.143941 trunc from 0x040000 to 0x012a4d (0x2d5b2 bytes) 000433 1787861897.162431 trunc from 0x012a4e to 0x03c427 (0x299da bytes) 000434 1787861897.186160 mapread 0x036a8a thru 0x038c8c (0x2203 bytes) 000435 1787861897.202312 trunc from 0x03c428 to 0x0361e9 (0x623e bytes) 000439 1787861897.227407 mapread 0x01741d thru 0x0270d0 (0xfcb4 bytes) 000441 1787861897.396021 fallocate 0x020646 thru 0x02b947 (0xb302 bytes) 000442 1787861897.413677 trunc from 0x0361ea to 0x00b36c (0x2ae7d bytes) 000443 1787861897.438707 trunc from 0x00b36d to 0x038454 (0x2d0e8 bytes) 000444 1787861897.459761 trunc from 0x037ae4 to 0x038454 (0x971 bytes) 000445 1787861897.463467 read 0x01eb7c thru 0x02a8a0 (0xbd25 bytes) 000447 1787861897.472326 fallocate 0x0354a3 thru 0x0379e3 (0x2541 bytes) 000449 1787861897.475824 write 0x027121 thru 0x02cea2 (0x5d82 bytes) 000450 1787861897.476946 trunc from 0x038455 to 0x036536 (0x1f1e bytes) 000451 1787861897.496958 mapwrite 0x03c682 thru 0x03eefe (0x287d bytes) 000452 1787861897.572828 read 0x00691f thru 0x012fb6 (0xc698 bytes) 000453 1787861897.573390 write 0x031d09 thru 0x033974 (0x1c6c bytes) 000454 1787861897.597250 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eeff): Mirror 12: 0x14d09440 (stale) Mirror 13: 0x14d09440 (stale) Mirror 14: 0x14d09440 (stale) Mirror 15: 0x14d09440 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eeff): Mirror 12: 0x14d09440 (stale) Mirror 13: 0x14d09440 (stale) Mirror 14: 0x14d09440 (stale) Mirror 15: 0x14d09440 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eeff: skipped 000457 1787861898.046939 mapwrite 0x037631 thru 0x03ffff (0x89cf bytes) 000459 1787861898.151298 read 0x02d6c3 thru 0x038d38 (0xb676 bytes) 000460 1787861898.165088 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa4e4625c (stale) Mirror 13: 0xa4e4625c (stale) Mirror 14: 0xa4e4625c (stale) Mirror 15: 0xa4e4625c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x20d3db43 Mirror 16: 0x20d3db43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000461 1787861899.340045 MIRROR_RESYNC 6 mirrors 000462 1787861900.932145 punch from 0x007075 to 0x016cb2 (0xfc3e bytes) 000463 1787861900.965424 write 0x03b77f thru 0x03ffff (0x4881 bytes) 000464 1787861900.984927 punch from 0x02e012 to 0x034161 (0x6150 bytes) 000465 1787861900.993237 mapwrite 0x00e2ef thru 0x015c23 (0x7935 bytes) 000466 1787861901.029777 trunc from 0x02ee9e to 0x0343e9 (0x554c bytes) 000467 1787861901.037406 write 0x0132c2 thru 0x01a5f2 (0x7331 bytes) 000468 1787861901.041594 mapread 0x014f3b thru 0x01d66b (0x8731 bytes) 000469 1787861901.053316 fallocate 0x03513e thru 0x037a36 (0x28f9 bytes) 000470 1787861901.059053 mapwrite 0x035b37 thru 0x03df0b (0x83d5 bytes) 000471 1787861901.090790 fallocate 0x00e33e thru 0x01c28e (0xdf51 bytes) 000472 1787861901.098378 mapwrite 0x0152e7 thru 0x0215f8 (0xc312 bytes) 000473 1787861901.146601 read 0x0100f8 thru 0x01eb6e (0xea77 bytes) 000475 1787861901.151694 trunc from 0x040000 to 0x03dd4a (0x22b5 bytes) 000476 1787861901.182864 fallocate 0x03f360 thru 0x03ffff (0xca0 bytes) 000477 1787861901.190732 mapread 0x036303 thru 0x03ffe8 (0x9ce6 bytes) 000478 1787861901.198401 trunc from 0x01c385 to 0x0241b2 (0x7e2e bytes) 000480 1787861901.205459 MIRROR_RESYNC 6 mirrors 000482 1787861903.127794 punch from 0x004fb1 to 0x012beb (0xdc3b bytes) 000483 1787861903.192722 fallocate 0x00edc2 thru 0x015926 (0x6b65 bytes) 000484 1787861903.203119 punch from 0x032d02 to 0x0335a7 (0x8a6 bytes) 000485 1787861903.211481 fallocate 0x01a495 thru 0x025fd7 (0xbb43 bytes) 000486 1787861903.220685 mapwrite 0x037713 thru 0x03f829 (0x8117 bytes) 000488 1787861903.286866 punch from 0x01d2cc to 0x021258 (0x3f8d bytes) 000489 1787861903.295243 fallocate 0x004a72 thru 0x00b608 (0x6b97 bytes) 000490 1787861903.302824 punch from 0x036e0a to 0x0384ed (0x16e4 bytes) 000491 1787861903.311685 trunc from 0x040000 to 0x01ea82 (0x2157d bytes) 000492 1787861903.362562 punch from 0x00346a to 0x00b8f3 (0x848a bytes) 000493 1787861903.373640 punch from 0x010227 to 0x017b1f (0x78f9 bytes) 000494 1787861903.386913 mapwrite 0x01b32f thru 0x01e11e (0x2df0 bytes) 000496 1787861903.447617 read 0x004e06 thru 0x00890d (0x3b08 bytes) 000497 1787861903.471989 trunc from 0x01ea83 to 0x0287c0 (0x9d3e bytes) 000498 1787861903.522940 mapwrite 0x039307 thru 0x03ffff (0x6cf9 bytes) 000501 1787861903.659862 trunc from 0x040000 to 0x0309b0 (0xf64f bytes) 000502 1787861903.743983 trunc from 0x01c3b4 to 0x0237c5 (0x7412 bytes) 000503 1787861903.780609 punch from 0x014908 to 0x01c0e3 (0x77dc bytes) 000505 1787861903.812908 write 0x036c2c thru 0x03eabd (0x7e92 bytes) 000506 1787861903.850927 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eabe): Mirror 13: 0x35b83e65 (stale) Mirror 14: 0x35b83e65 (stale) Mirror 15: 0x35b83e65 (stale) Mirror 16: 0x35b83e65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabe): Mirror 13: 0x35b83e65 (stale) Mirror 14: 0x35b83e65 (stale) Mirror 15: 0x35b83e65 (stale) Mirror 16: 0x35b83e65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabe: skipped 000507 1787861904.748321 trunc from 0x006557 to 0x0139fd (0xd4a7 bytes) 000508 1787861904.759975 trunc from 0x03eabe to 0x035bf2 (0x8ecb bytes) 000510 1787861904.804107 mapread 0x028a59 thru 0x035143 (0xc6eb bytes) 000511 1787861904.831334 trunc from 0x035bf3 to 0x00e998 (0x2725a bytes) 000512 1787861904.851155 MIRROR_RESYNC 5 mirrors 000513 1787861905.969937 trunc from 0x00e999 to 0x01e19d (0xf805 bytes) 000514 1787861906.080906 punch from 0x00548d to 0x00ce05 (0x7979 bytes) 000515 1787861906.095600 mapread 0x01d30b thru 0x01e19d (0xe93 bytes) 000516 1787861906.143636 trunc from 0x01e19e to 0x02c4f4 (0xe357 bytes) 000517 1787861906.204499 trunc from 0x02c4f5 to 0x03e13e (0x11c4a bytes) 000518 1787861906.278326 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e13f): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e13f): Mirror 13: 0x85a31271 (stale) Mirror 14: 0x85a31271 (stale) Mirror 15: 0x85a31271 (stale) Mirror 16: 0x85a31271 (stale) CRC-32 checksum value for chunk [0, 0x3e13f): Mirror 1: 0x85a31271 Mirror 17: 0x85a31271 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e13f: skipped 000519 1787861908.311233 write 0x018dba thru 0x01ba87 (0x2cce bytes) 000520 1787861908.359571 mapwrite 0x034751 thru 0x03c785 (0x8035 bytes) 000521 1787861908.393383 trunc from 0x03e13f to 0x028f02 (0x1523c bytes) 000522 1787861908.422079 MIRROR_RESYNC 6 mirrors 000523 1787861910.143429 trunc from 0x01495c to 0x016ad7 (0x217c bytes) 000525 1787861910.247306 punch from 0x00b9d1 to 0x018600 (0xcc30 bytes) 000526 1787861910.255649 read 0x026e10 thru 0x0289c9 (0x1bba bytes) 000527 1787861910.272251 mapwrite 0x00f5c8 thru 0x01cc8b (0xd6c4 bytes) 000528 1787861910.384409 mapwrite 0x0282a6 thru 0x031a40 (0x979b bytes) 000529 1787861910.493680 mapread 0x0261c0 thru 0x0289f0 (0x2831 bytes) 000530 1787861910.494800 write 0x01d4a0 thru 0x02c456 (0xefb7 bytes) 000534 1787861910.501261 trunc from 0x031a41 to 0x03f6cc (0xdc8c bytes) 000535 1787861910.540448 mapwrite 0x011358 thru 0x01b4dd (0xa186 bytes) 000536 1787861910.619459 fallocate 0x036e0b thru 0x037f42 (0x1138 bytes) 000537 1787861910.670356 MIRROR_RESYNC 6 mirrors 000538 1787861913.714347 mapread 0x00f3a5 thru 0x016dee (0x7a4a bytes) 000539 1787861913.756115 write 0x007523 thru 0x00fa1a (0x84f8 bytes) 000540 1787861913.821095 mapwrite 0x039d25 thru 0x03b5dc (0x18b8 bytes) 000541 1787861913.860069 mapwrite 0x0132aa thru 0x01422b (0xf82 bytes) 000542 1787861913.898586 fallocate 0x023cc1 thru 0x03379b (0xfadb bytes) 000544 1787861913.917396 mapread 0x028905 thru 0x02a200 (0x18fc bytes) 000545 1787861913.931243 trunc from 0x03f6cd to 0x029bf2 (0x15ada bytes) 000549 1787861913.965345 fallocate 0x02b719 thru 0x030e09 (0x56f1 bytes) 000550 1787861913.969947 trunc from 0x003d63 to 0x010258 (0xc4f6 bytes) 000551 1787861913.973189 fallocate 0x017933 thru 0x01ddad (0x647b bytes) 000552 1787861913.976822 read 0x00575c thru 0x012ef3 (0xd798 bytes) 000553 1787861913.984945 write 0x03b0cf thru 0x03f0e2 (0x4014 bytes) 000554 1787861913.985906 mapread 0x03e6ca thru 0x03f0e2 (0xa19 bytes) 000555 1787861913.986493 punch from 0x030f10 to 0x03379d (0x288e bytes) 000556 1787861913.989294 read 0x02d7e1 thru 0x03015a (0x297a bytes) 000557 1787861913.999145 write 0x01bd0f thru 0x020990 (0x4c82 bytes) 000559 1787861914.000629 fallocate 0x000361 thru 0x003f13 (0x3bb3 bytes) 000560 1787861914.008662 trunc from 0x03f0e3 to 0x013b93 (0x2b54f bytes) 000561 1787861914.061121 trunc from 0x007c04 to 0x013b93 (0xbf90 bytes) 000562 1787861914.077435 read 0x007d23 thru 0x00ce65 (0x5143 bytes) 000563 1787861914.094388 mapwrite 0x02f344 thru 0x0302da (0xf97 bytes) 000564 1787861914.166648 mapwrite 0x02f251 thru 0x034e5c (0x5c0c bytes) 000565 1787861914.221343 write 0x013104 thru 0x0178d2 (0x47cf bytes) 000566 1787861914.223802 write 0x026d5b thru 0x0335a4 (0xc84a bytes) 000567 1787861914.226104 punch from 0x032b0a to 0x034e5c (0x2353 bytes) 000568 1787861914.247604 read 0x02cc04 thru 0x034e5c (0x8259 bytes) 000569 1787861914.253417 write 0x0267d0 thru 0x034324 (0xdb55 bytes) 000570 1787861914.257911 trunc from 0x034e5d to 0x00358e (0x318ce bytes) 000571 1787861914.304079 fallocate 0x02733d thru 0x02b457 (0x411b bytes) 000572 1787861914.311991 mapread 0x000582 thru 0x00358e (0x300d bytes) 000573 1787861914.313572 read 0x000212 thru 0x00358e (0x337d bytes) 000574 1787861914.314187 mapwrite 0x0259dd thru 0x02d5ca (0x7bee bytes) 000575 1787861914.428615 mapwrite 0x01c457 thru 0x01c732 (0x2dc bytes) 000576 1787861914.441409 read 0x0253eb thru 0x02d4f1 (0x8107 bytes) 000577 1787861914.442186 write 0x02ce1d thru 0x03c463 (0xf647 bytes) 000578 1787861914.447826 fallocate 0x0140f5 thru 0x0169a9 (0x28b5 bytes) 000579 1787861914.459731 trunc from 0x03c464 to 0x024ef9 (0x1756a bytes) 000580 1787861914.482649 read 0x00e5af thru 0x0181bf (0x9c11 bytes) 000582 1787861914.483626 mapread 0x00104b thru 0x001e11 (0xdc7 bytes) 000583 1787861914.520498 MIRROR_RESYNC 6 mirrors 000584 1787861916.394380 fallocate 0x016c48 thru 0x0200c5 (0x947e bytes) 000585 1787861916.464108 punch from 0x016d9a to 0x021db1 (0xb018 bytes) 000586 1787861916.479165 read 0x022df7 thru 0x024249 (0x1453 bytes) 000587 1787861916.492860 trunc from 0x024efa to 0x038691 (0x13798 bytes) 000588 1787861916.536751 mapread 0x017249 thru 0x021ff6 (0xadae bytes) 000589 1787861916.572260 punch from 0x02a56a to 0x037141 (0xcbd8 bytes) 000590 1787861916.579302 fallocate 0x01763c thru 0x0187e0 (0x11a5 bytes) 000591 1787861916.598259 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38692): Mirror 14: 0x1b8b0c9d (stale) Mirror 15: 0x1b8b0c9d (stale) Mirror 16: 0x1b8b0c9d (stale) Mirror 17: 0x1b8b0c9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38692): Mirror 14: 0x1b8b0c9d (stale) Mirror 15: 0x1b8b0c9d (stale) Mirror 16: 0x1b8b0c9d (stale) Mirror 17: 0x1b8b0c9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38692: skipped 000592 1787861917.079782 punch from 0x026619 to 0x02f25b (0x8c43 bytes) 000593 1787861917.085485 punch from 0x00156f to 0x00f117 (0xdba9 bytes) 000594 1787861917.097456 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38692): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38692): Mirror 14: 0x1b8b0c9d (stale) Mirror 15: 0x1b8b0c9d (stale) Mirror 16: 0x1b8b0c9d (stale) Mirror 17: 0x1b8b0c9d (stale) CRC-32 checksum value for chunk [0, 0x38692): Mirror 1: 0x73b077ad Mirror 18: 0x73b077ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38692: skipped 000595 1787861917.938955 trunc from 0x02ce3c to 0x03343d (0x6602 bytes) 000597 1787861917.985761 read 0x01df47 thru 0x01f69e (0x1758 bytes) 000598 1787861917.994579 read 0x02744e thru 0x0331a4 (0xbd57 bytes) 000599 1787861918.024372 mapwrite 0x02e4b3 thru 0x02ea60 (0x5ae bytes) 000600 1787861918.039931 write 0x006777 thru 0x014b6f (0xe3f9 bytes) 000601 1787861918.048674 write 0x00325a thru 0x00e1fe (0xafa5 bytes) 000602 1787861918.050875 trunc from 0x038692 to 0x0073f2 (0x3129f bytes) 000603 1787861918.079393 fallocate 0x0053ec thru 0x00d19c (0x7db1 bytes) 000604 1787861918.086260 trunc from 0x00d19d to 0x022e23 (0x15c87 bytes) 000605 1787861918.160731 trunc from 0x022e24 to 0x011eb0 (0x10f73 bytes) 000606 1787861918.224640 punch from 0x001a12 to 0x009091 (0x7680 bytes) 000607 1787861918.319391 mapwrite 0x02c98c thru 0x0361d4 (0x9849 bytes) 000608 1787861918.652651 trunc from 0x01a0d6 to 0x01fabe (0x59e9 bytes) 000609 1787861918.686987 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361d5): Mirror 14: 0xad97750b (stale) Mirror 15: 0xad97750b (stale) Mirror 16: 0xad97750b (stale) Mirror 18: 0xea203e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361d5): Mirror 14: 0xad97750b (stale) Mirror 15: 0xad97750b (stale) Mirror 16: 0xad97750b (stale) Mirror 18: 0xea203e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361d5: skipped 000611 1787861919.453419 write 0x017d98 thru 0x01f1ce (0x7437 bytes) 000613 1787861919.480265 write 0x017bf6 thru 0x023f6b (0xc376 bytes) 000615 1787861919.497615 fallocate 0x01789b thru 0x020f5f (0x96c5 bytes) 000616 1787861919.503650 fallocate 0x00967f thru 0x010a6b (0x73ed bytes) 000617 1787861919.512977 punch from 0x02e36d to 0x0361d4 (0x7e68 bytes) 000619 1787861919.533242 write 0x003cc8 thru 0x00f62c (0xb965 bytes) 000620 1787861919.556482 write 0x0095d3 thru 0x011a11 (0x843f bytes) 000621 1787861919.585792 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x361d5): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361d5): Mirror 14: 0xad97750b (stale) Mirror 15: 0xad97750b (stale) Mirror 16: 0xad97750b (stale) Mirror 18: 0xea203e77 (stale) CRC-32 checksum value for chunk [0, 0x361d5): Mirror 1: 0x3012aae8 Mirror 19: 0x3012aae8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361d5: skipped 000622 1787861921.690217 trunc from 0x00cc57 to 0x012dd8 (0x6182 bytes) 000624 1787861921.765341 fallocate 0x03bdfc thru 0x03ffff (0x4204 bytes) 000625 1787861921.771031 trunc from 0x025a1a to 0x031503 (0xbaea bytes) 000627 1787861921.780223 trunc from 0x040000 to 0x00932b (0x36cd4 bytes) 000628 1787861921.864706 trunc from 0x006add to 0x00932b (0x284f bytes) 000629 1787861921.872938 fallocate 0x035ec4 thru 0x03bfe4 (0x6121 bytes) 000630 1787861921.903157 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x932c): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x932c): Mirror 14: 0x40b23e49 (stale) Mirror 15: 0x40b23e49 (stale) Mirror 16: 0x40b23e49 (stale) Mirror 18: 0x73d3903d (stale) Mirror 19: 0xc9723f6d (stale) CRC-32 checksum value for chunk [0, 0x932c): Mirror 1: 0xe448eb4c Mirror 20: 0xe448eb4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x932c: skipped 000631 1787861923.841553 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x932c): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x932c): Mirror 14: 0x40b23e49 (stale) Mirror 15: 0x40b23e49 (stale) Mirror 16: 0x40b23e49 (stale) Mirror 18: 0x73d3903d (stale) Mirror 19: 0xc9723f6d (stale) CRC-32 checksum value for chunk [0, 0x932c): Mirror 1: 0xe448eb4c Mirror 20: 0xe448eb4c Mirror 21: 0xe448eb4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x932c: skipped 000632 1787861925.583959 read 0x0062db thru 0x00932b (0x3051 bytes) 000633 1787861925.601320 write 0x019478 thru 0x023051 (0x9bda bytes) 000634 1787861925.761830 read 0x00ecb5 thru 0x0141f7 (0x5543 bytes) 000635 1787861925.790324 trunc from 0x01103b to 0x01bea2 (0xae68 bytes) 000636 1787861925.843917 punch from 0x0110ba to 0x01dec4 (0xce0b bytes) 000637 1787861925.852869 mapread 0x0199d7 thru 0x019cc8 (0x2f2 bytes) 000638 1787861925.865055 mapread 0x01d05d thru 0x0218a1 (0x4845 bytes) 000639 1787861925.895329 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x23052): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23052): Mirror 14: 0xc781141d (stale) Mirror 15: 0xc781141d (stale) Mirror 16: 0xc781141d (stale) Mirror 18: 0x5f9cdf15 (stale) Mirror 19: 0xbbb60ec1 (stale) Mirror 20: 0x481a911f (stale) Mirror 21: 0x481a911f (stale) CRC-32 checksum value for chunk [0, 0x23052): Mirror 1: 0xbebbfa61 Mirror 22: 0xbebbfa61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23052: skipped 000641 1787861927.646115 read 0x00d237 thru 0x01336d (0x6137 bytes) 000642 1787861927.683388 punch from 0x00a2f6 to 0x01023f (0x5f4a bytes) 000643 1787861927.777265 trunc from 0x01379c to 0x020c5b (0xd4c0 bytes) 000644 1787861927.794332 write 0x012919 thru 0x016a0a (0x40f2 bytes) 000645 1787861927.815754 trunc from 0x00a827 to 0x00b39e (0xb78 bytes) 000647 1787861927.822355 mapwrite 0x005486 thru 0x00e840 (0x93bb bytes) 000648 1787861927.857673 punch from 0x0111b5 to 0x01a477 (0x92c3 bytes) 000649 1787861927.862700 read 0x013535 thru 0x01e5c2 (0xb08e bytes) 000651 1787861927.897763 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23052): Mirror 14: 0xc781141d (stale) Mirror 15: 0xc781141d (stale) Mirror 16: 0xc781141d (stale) Mirror 18: 0x5f9cdf15 (stale) Mirror 20: 0x481a911f (stale) Mirror 21: 0x481a911f (stale) Mirror 22: 0xbebbfa61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23052): Mirror 14: 0xc781141d (stale) Mirror 15: 0xc781141d (stale) Mirror 16: 0xc781141d (stale) Mirror 18: 0x5f9cdf15 (stale) Mirror 20: 0x481a911f (stale) Mirror 21: 0x481a911f (stale) Mirror 22: 0xbebbfa61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23052: skipped 000652 1787861928.953458 fallocate 0x02c153 thru 0x031a58 (0x5906 bytes) 000653 1787861928.958438 trunc from 0x02c917 to 0x031a58 (0x5142 bytes) 000654 1787861928.969133 write 0x015ce8 thru 0x0202f5 (0xa60e bytes) 000655 1787861928.991389 write 0x017745 thru 0x019056 (0x1912 bytes) 000656 1787861928.992461 mapwrite 0x002dc6 thru 0x00f49c (0xc6d7 bytes) 000657 1787861929.075701 write 0x03a7f6 thru 0x03ffff (0x580a bytes) 000658 1787861929.089832 read 0x0099a5 thru 0x00c1da (0x2836 bytes) 000659 1787861929.102232 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x98a98443 (stale) Mirror 15: 0x98a98443 (stale) Mirror 16: 0x98a98443 (stale) Mirror 18: 0x70e89704 (stale) Mirror 20: 0x9a4c64fa (stale) Mirror 21: 0x9a4c64fa (stale) Mirror 22: 0x5af6fbd8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8a957ea Mirror 23: 0xe8a957ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1787861930.780296 read 0x01f91d thru 0x02c04b (0xc72f bytes) 000661 1787861930.796072 trunc from 0x036d4a to 0x03ffff (0x92b6 bytes) 000662 1787861930.857435 trunc from 0x040000 to 0x022da3 (0x1d25c bytes) 000663 1787861930.946250 mapwrite 0x01ca72 thru 0x027456 (0xa9e5 bytes) 000664 1787861931.037261 mapread 0x003988 thru 0x00d69e (0x9d17 bytes) 000665 1787861931.049581 punch from 0x020e0c to 0x024a62 (0x3c57 bytes) 000667 1787861931.064160 fallocate 0x0168d3 thru 0x025741 (0xee6f bytes) 000668 1787861931.071231 punch from 0x01ed7e to 0x027456 (0x86d9 bytes) 000669 1787861931.085346 mapwrite 0x02e880 thru 0x031b57 (0x32d8 bytes) 000670 1787861931.139374 trunc from 0x0036e6 to 0x003f1f (0x83a bytes) 000671 1787861931.151454 mapread 0x00bef6 thru 0x00d668 (0x1773 bytes) 000672 1787861931.153709 fallocate 0x016686 thru 0x018fb7 (0x2932 bytes) 000673 1787861931.161318 read 0x02fce9 thru 0x031b57 (0x1e6f bytes) 000674 1787861931.170160 trunc from 0x02c3f7 to 0x02c464 (0x6e bytes) 000675 1787861931.182286 MIRROR_RESYNC 9 mirrors 000676 1787861934.240635 read 0x02e561 thru 0x031b57 (0x35f7 bytes) 000677 1787861934.260102 MIRROR_RESYNC 9 mirrors 000678 1787861935.004985 write 0x013c8f thru 0x020c84 (0xcff6 bytes) 000679 1787861935.120144 fallocate 0x021307 thru 0x025786 (0x4480 bytes) 000680 1787861935.124804 punch from 0x013cae to 0x02249a (0xe7ed bytes) 000681 1787861935.146754 read 0x02c57c thru 0x031b57 (0x55dc bytes) 000682 1787861935.187679 read 0x01498b thru 0x01b788 (0x6dfe bytes) 000683 1787861935.261273 fallocate 0x0011f6 thru 0x00618e (0x4f99 bytes) 000684 1787861935.274584 mapwrite 0x01f40d thru 0x01f8bf (0x4b3 bytes) 000685 1787861935.311026 fallocate 0x02eac0 thru 0x03274f (0x3c90 bytes) 000686 1787861935.322382 mapread 0x0113c8 thru 0x014585 (0x31be bytes) 000687 1787861935.326433 mapread 0x011eef thru 0x01dfe3 (0xc0f5 bytes) 000688 1787861935.344094 trunc from 0x031b58 to 0x03ab91 (0x903a bytes) 000689 1787861935.399437 mapread 0x00e3e0 thru 0x019d91 (0xb9b2 bytes) 000690 1787861935.402987 fallocate 0x02ad67 thru 0x038a00 (0xdc9a bytes) 000691 1787861935.411126 punch from 0x00646a to 0x006b9d (0x734 bytes) 000692 1787861935.418262 mapwrite 0x004afb thru 0x011276 (0xc77c bytes) 000693 1787861935.462099 mapread 0x013dce thru 0x018845 (0x4a78 bytes) 000694 1787861935.469136 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 14: 0x1549f180 (stale) Mirror 15: 0x1549f180 (stale) Mirror 16: 0x1549f180 (stale) Mirror 18: 0x1549f180 (stale) Mirror 20: 0x1549f180 (stale) Mirror 21: 0x1549f180 (stale) Mirror 22: 0x1549f180 (stale) Mirror 23: 0x1549f180 (stale) CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 1: 0xf6757ca2 Mirror 24: 0xf6757ca2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab92: skipped 000695 1787861936.630943 mapread 0x019b9c thru 0x01e2a1 (0x4706 bytes) 000696 1787861936.654463 mapwrite 0x00c2bf thru 0x00c5cd (0x30f bytes) 000697 1787861936.750200 trunc from 0x01d844 to 0x022a70 (0x522d bytes) 000698 1787861936.758211 trunc from 0x000af5 to 0x00694d (0x5e59 bytes) 000699 1787861936.763664 fallocate 0x015ec2 thru 0x025e39 (0xff78 bytes) 000700 1787861936.770177 punch from 0x010c1c to 0x013603 (0x29e8 bytes) 000701 1787861936.781742 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 15: 0x1549f180 (stale) Mirror 16: 0x1549f180 (stale) Mirror 18: 0x1549f180 (stale) Mirror 20: 0x1549f180 (stale) Mirror 21: 0x1549f180 (stale) Mirror 22: 0x1549f180 (stale) Mirror 23: 0x1549f180 (stale) Mirror 24: 0xf6757ca2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 15: 0x1549f180 (stale) Mirror 16: 0x1549f180 (stale) Mirror 18: 0x1549f180 (stale) Mirror 20: 0x1549f180 (stale) Mirror 21: 0x1549f180 (stale) Mirror 22: 0x1549f180 (stale) Mirror 23: 0x1549f180 (stale) Mirror 24: 0xf6757ca2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab92: skipped 000702 1787861937.390226 trunc from 0x03ab92 to 0x03a2ae (0x8e3 bytes) 000703 1787861937.456140 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2af): Mirror 15: 0xbeeb0b66 (stale) Mirror 16: 0xbeeb0b66 (stale) Mirror 18: 0xbeeb0b66 (stale) Mirror 20: 0xbeeb0b66 (stale) Mirror 21: 0xbeeb0b66 (stale) Mirror 22: 0xbeeb0b66 (stale) Mirror 23: 0xbeeb0b66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2af): Mirror 15: 0xbeeb0b66 (stale) Mirror 16: 0xbeeb0b66 (stale) Mirror 18: 0xbeeb0b66 (stale) Mirror 20: 0xbeeb0b66 (stale) Mirror 21: 0xbeeb0b66 (stale) Mirror 22: 0xbeeb0b66 (stale) Mirror 23: 0xbeeb0b66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2af: skipped 000704 1787861937.985165 punch from 0x02d81b to 0x0316af (0x3e95 bytes) 000705 1787861937.995675 trunc from 0x03a2af to 0x01d802 (0x1caac bytes) 000706 1787861938.019183 read 0x01cdc4 thru 0x01d802 (0xa3f bytes) 000707 1787861938.024597 trunc from 0x006511 to 0x00d519 (0x7009 bytes) 000709 1787861938.031251 mapread 0x006130 thru 0x0124fd (0xc3ce bytes) 000710 1787861938.057299 read 0x00c439 thru 0x018a02 (0xc5ca bytes) 000711 1787861938.058656 read 0x003dbd thru 0x00a3ca (0x660e bytes) 000712 1787861938.059180 mapwrite 0x0022fe thru 0x0083c2 (0x60c5 bytes) 000713 1787861938.075257 mapread 0x00bde7 thru 0x011290 (0x54aa bytes) 000714 1787861938.076456 fallocate 0x03a743 thru 0x03ffff (0x58bd bytes) 000715 1787861938.082087 trunc from 0x00279c to 0x00f70c (0xcf71 bytes) 000716 1787861938.087317 mapread 0x03f061 thru 0x03ffff (0xf9f bytes) 000717 1787861938.106001 punch from 0x011133 to 0x016b97 (0x5a65 bytes) 000718 1787861938.124415 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4ff622ab (stale) Mirror 18: 0x4ff622ab (stale) Mirror 20: 0x4ff622ab (stale) Mirror 21: 0x4ff622ab (stale) Mirror 22: 0x4ff622ab (stale) Mirror 23: 0x4ff622ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4ff622ab (stale) Mirror 18: 0x4ff622ab (stale) Mirror 20: 0x4ff622ab (stale) Mirror 21: 0x4ff622ab (stale) Mirror 22: 0x4ff622ab (stale) Mirror 23: 0x4ff622ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000719 1787861938.669046 trunc from 0x00ebf9 to 0x0135a9 (0x49b1 bytes) 000720 1787861938.673757 write 0x0309dd thru 0x0336d6 (0x2cfa bytes) 000721 1787861938.685902 write 0x001aff thru 0x00c0ec (0xa5ee bytes) 000722 1787861938.703220 write 0x03a669 thru 0x03ffff (0x5997 bytes) 000723 1787861938.711325 mapread 0x00e820 thru 0x01a720 (0xbf01 bytes) 000724 1787861938.742984 punch from 0x01e0e8 to 0x027e37 (0x9d50 bytes) 000725 1787861938.747690 write 0x03d0f7 thru 0x03ffff (0x2f09 bytes) 000726 1787861938.749994 fallocate 0x00605e thru 0x007317 (0x12ba bytes) 000727 1787861938.754829 mapwrite 0x03218b thru 0x039dc7 (0x7c3d bytes) 000728 1787861938.785858 punch from 0x01fdd5 to 0x02cb32 (0xcd5e bytes) 000729 1787861938.790544 mapread 0x00855b thru 0x00f6fa (0x71a0 bytes) 000730 1787861938.797637 MIRROR_RESYNC 7 mirrors 000732 1787861940.286595 punch from 0x01e855 to 0x0213fb (0x2ba7 bytes) 000733 1787861940.334941 mapwrite 0x029151 thru 0x02a05b (0xf0b bytes) 000734 1787861940.369075 write 0x031ca9 thru 0x03e1bc (0xc514 bytes) 000735 1787861940.393470 MIRROR_RESYNC 7 mirrors 000736 1787861942.217076 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000737 1787861942.655809 mapread 0x01f032 thru 0x025fc3 (0x6f92 bytes) 000738 1787861942.703201 fallocate 0x034743 thru 0x03ffff (0xb8bd bytes) 000739 1787861942.792653 write 0x02525c thru 0x030da3 (0xbb48 bytes) 000740 1787861942.815413 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xdf9732da (stale) Mirror 20: 0xdf9732da (stale) Mirror 21: 0xdf9732da (stale) Mirror 22: 0xdf9732da (stale) Mirror 23: 0xdf9732da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c773d91 Mirror 24: 0x3c773d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000741 1787861944.346150 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xdf9732da (stale) Mirror 20: 0xdf9732da (stale) Mirror 21: 0xdf9732da (stale) Mirror 22: 0xdf9732da (stale) Mirror 23: 0xdf9732da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c773d91 Mirror 24: 0x3c773d91 Mirror 25: 0x3c773d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000742 1787861945.424132 fallocate 0x0087fd thru 0x01752c (0xed30 bytes) 000743 1787861945.487065 read 0x032f4a thru 0x03b40b (0x84c2 bytes) 000744 1787861945.509719 MIRROR_RESYNC 8 mirrors 000745 1787861947.304967 fallocate 0x01c839 thru 0x02b1ed (0xe9b5 bytes) 000746 1787861947.351963 read 0x03aacc thru 0x03e9f2 (0x3f27 bytes) 000747 1787861947.364067 mapread 0x01b953 thru 0x0297d1 (0xde7f bytes) 000748 1787861947.424420 mapwrite 0x03ba4b thru 0x03d9ed (0x1fa3 bytes) 000749 1787861947.454328 read 0x028033 thru 0x035a79 (0xda47 bytes) 000750 1787861947.477850 punch from 0x01818f to 0x01f6e8 (0x755a bytes) 000751 1787861947.499666 MIRROR_RESYNC 8 mirrors 000752 1787861949.010439 trunc from 0x040000 to 0x01358f (0x2ca70 bytes) 000753 1787861949.045894 mapwrite 0x018c08 thru 0x020dca (0x81c3 bytes) 000755 1787861949.091369 mapwrite 0x0326b3 thru 0x03c1f3 (0x9b41 bytes) 000756 1787861949.153753 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1f4): Mirror 16: 0x5b377d65 (stale) Mirror 20: 0x5b377d65 (stale) Mirror 21: 0x5b377d65 (stale) Mirror 22: 0x5b377d65 (stale) Mirror 23: 0x5b377d65 (stale) Mirror 24: 0x5b377d65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1f4): Mirror 16: 0x5b377d65 (stale) Mirror 20: 0x5b377d65 (stale) Mirror 21: 0x5b377d65 (stale) Mirror 22: 0x5b377d65 (stale) Mirror 23: 0x5b377d65 (stale) Mirror 24: 0x5b377d65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1f4: skipped 000757 1787861949.643048 trunc from 0x03c1f4 to 0x00b2bd (0x30f36 bytes) 000758 1787861949.670212 trunc from 0x00b2be to 0x00f2e1 (0x4024 bytes) 000759 1787861949.690252 read 0x00aace thru 0x00f2e1 (0x4814 bytes) 000760 1787861949.722708 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2e2): Mirror 16: 0xe25559d6 (stale) Mirror 21: 0xe25559d6 (stale) Mirror 22: 0xe25559d6 (stale) Mirror 23: 0xe25559d6 (stale) Mirror 24: 0xe25559d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2e2): Mirror 16: 0xe25559d6 (stale) Mirror 21: 0xe25559d6 (stale) Mirror 22: 0xe25559d6 (stale) Mirror 23: 0xe25559d6 (stale) Mirror 24: 0xe25559d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2e2: skipped 000761 1787861950.180101 trunc from 0x00f2e2 to 0x00b23c (0x40a5 bytes) 000762 1787861950.202344 punch from 0x0022e9 to 0x003e7f (0x1b97 bytes) 000764 1787861950.214841 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb23d): Mirror 16: 0xbc8a8bfc (stale) Mirror 22: 0xbc8a8bfc (stale) Mirror 23: 0xbc8a8bfc (stale) Mirror 24: 0xbc8a8bfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb23d): Mirror 16: 0xbc8a8bfc (stale) Mirror 22: 0xbc8a8bfc (stale) Mirror 23: 0xbc8a8bfc (stale) Mirror 24: 0xbc8a8bfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb23d: skipped 000766 1787861950.621040 trunc from 0x00b23d to 0x026b6f (0x1b933 bytes) 000768 1787861950.642828 write 0x000ede thru 0x0096e1 (0x8804 bytes) 000769 1787861950.653206 trunc from 0x026b70 to 0x032f7e (0xc40f bytes) 000770 1787861950.668009 read 0x001746 thru 0x00e38f (0xcc4a bytes) 000771 1787861950.675220 punch from 0x016ffb to 0x023760 (0xc766 bytes) 000772 1787861950.679750 write 0x005a9a thru 0x0092f3 (0x385a bytes) 000773 1787861950.680830 mapread 0x02fcb3 thru 0x0304e2 (0x830 bytes) 000774 1787861950.688279 trunc from 0x032f7f to 0x018c15 (0x1a369 bytes) 000775 1787861950.700882 write 0x0180e3 thru 0x019525 (0x1443 bytes) 000776 1787861950.704828 mapread 0x005e87 thru 0x0115f0 (0xb76a bytes) 000778 1787861950.711713 mapwrite 0x015c60 thru 0x01994d (0x3cee bytes) 000780 1787861950.769019 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: 0x5b49277b (stale) Mirror 22: 0x5b49277b (stale) Mirror 23: 0x5b49277b (stale) Mirror 24: 0x5b49277b (stale) CRC-32 checksum value for chunk [0, 0x1994e): Mirror 1: 0x762c0708 Mirror 25: 0x762c0708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1994e: skipped 000781 1787861951.949765 read 0x003d52 thru 0x0081b4 (0x4463 bytes) 000782 1787861951.959390 read 0x003390 thru 0x00f618 (0xc289 bytes) 000783 1787861952.005018 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: 0x5b49277b (stale) Mirror 22: 0x5b49277b (stale) Mirror 23: 0x5b49277b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: 0x5b49277b (stale) Mirror 22: 0x5b49277b (stale) Mirror 23: 0x5b49277b (stale) CRC-32 checksum value for chunk [0, 0x1994e): Mirror 1: 0x762c0708 Mirror 25: 0x762c0708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1994e: skipped 000785 1787861952.888060 read 0x00f009 thru 0x01994d (0xa945 bytes) 000786 1787861952.920934 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1994e): Mirror 16: 0x5b49277b (stale) Mirror 22: 0x5b49277b (stale) Mirror 23: 0x5b49277b (stale) CRC-32 checksum value for chunk [0, 0x1994e): Mirror 1: 0x762c0708 Mirror 25: 0x762c0708 Mirror 26: 0x762c0708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1994e: skipped 000787 1787861953.756764 mapwrite 0x026391 thru 0x0347d8 (0xe448 bytes) 000788 1787861953.897725 trunc from 0x0163c9 to 0x0213e4 (0xb01c bytes) 000789 1787861953.905958 mapwrite 0x000401 thru 0x0016f2 (0x12f2 bytes) 000790 1787861953.940746 MIRROR_RESYNC 6 mirrors 000791 1787861955.146177 write 0x02d6ac thru 0x02ee32 (0x1787 bytes) 000792 1787861955.184151 read 0x01ba6a thru 0x025f12 (0xa4a9 bytes) 000793 1787861955.192158 write 0x01eb59 thru 0x020ba1 (0x2049 bytes) 000794 1787861955.193340 fallocate 0x03ff5c thru 0x03ffff (0xa4 bytes) 000795 1787861955.198474 mapwrite 0x03a135 thru 0x03ffff (0x5ecb bytes) 000796 1787861955.233118 trunc from 0x040000 to 0x02d86b (0x12794 bytes) 000797 1787861955.259387 trunc from 0x02c77d to 0x02d86b (0x10ef bytes) 000798 1787861955.265276 read 0x0161c9 thru 0x022770 (0xc5a8 bytes) 000799 1787861955.279151 write 0x01c473 thru 0x024e9c (0x8a2a bytes) 000800 1787861955.280619 trunc from 0x02d86c to 0x00d4f8 (0x20373 bytes) 000802 1787861955.305586 MIRROR_RESYNC 6 mirrors 000803 1787861956.211022 trunc from 0x007955 to 0x00c783 (0x4e2f bytes) 000804 1787861956.250836 punch from 0x00a435 to 0x00d4f8 (0x30c4 bytes) 000805 1787861956.257540 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xd4f9): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd4f9): Mirror 16: 0x1547b6c9 (stale) Mirror 22: 0x1547b6c9 (stale) Mirror 23: 0x1547b6c9 (stale) Mirror 25: 0x1547b6c9 (stale) Mirror 26: 0x1547b6c9 (stale) CRC-32 checksum value for chunk [0, 0xd4f9): Mirror 1: 0x1ff4e92f Mirror 27: 0x1ff4e92f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd4f9: skipped 000806 1787861957.374076 mapwrite 0x03f4a8 thru 0x03ffff (0xb58 bytes) 000807 1787861957.451752 trunc from 0x040000 to 0x0263ca (0x19c35 bytes) 000809 1787861957.476172 read 0x0241cd thru 0x0263ca (0x21fe bytes) 000810 1787861957.482822 trunc from 0x0263cb to 0x01a9e9 (0xb9e1 bytes) 000811 1787861957.506646 mapwrite 0x031d1d thru 0x03757f (0x5863 bytes) 000812 1787861957.556871 punch from 0x005ce2 to 0x009b9e (0x3ebd bytes) 000813 1787861957.561577 trunc from 0x037580 to 0x028921 (0xec5e bytes) 000814 1787861957.582043 read 0x011df8 thru 0x019727 (0x7930 bytes) 000815 1787861957.597680 read 0x011685 thru 0x01bbc3 (0xa53f bytes) 000816 1787861957.598995 trunc from 0x028922 to 0x035fab (0xd68a bytes) 000817 1787861957.614726 trunc from 0x035fac to 0x01c39e (0x19c0d bytes) 000818 1787861957.637874 fallocate 0x014deb thru 0x0224a1 (0xd6b7 bytes) 000819 1787861957.643752 write 0x00b665 thru 0x01831b (0xccb7 bytes) 000820 1787861957.650092 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c39f): Mirror 16: 0xdccde491 (stale) Mirror 22: 0xdccde491 (stale) Mirror 25: 0xdccde491 (stale) Mirror 26: 0xdccde491 (stale) Mirror 27: 0xf9869157 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c39f): Mirror 16: 0xdccde491 (stale) Mirror 22: 0xdccde491 (stale) Mirror 25: 0xdccde491 (stale) Mirror 26: 0xdccde491 (stale) Mirror 27: 0xf9869157 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c39f: skipped 000821 1787861958.023860 trunc from 0x00bef0 to 0x00ddb8 (0x1ec9 bytes) 000822 1787861958.029324 trunc from 0x00c083 to 0x012b80 (0x6afe bytes) 000823 1787861958.036435 trunc from 0x00fd38 to 0x01c39e (0xc667 bytes) 000824 1787861958.045867 mapread 0x00a242 thru 0x017cc2 (0xda81 bytes) 000825 1787861958.078867 punch from 0x001ed0 to 0x00746a (0x559b bytes) 000826 1787861958.085634 mapread 0x006883 thru 0x011646 (0xadc4 bytes) 000827 1787861958.095943 read 0x008d93 thru 0x018640 (0xf8ae bytes) 000828 1787861958.096940 write 0x03cbba thru 0x03ffff (0x3446 bytes) 000829 1787861958.103688 mapread 0x01c485 thru 0x02b8cf (0xf44b bytes) 000830 1787861958.120768 read 0x027e7a thru 0x02f04f (0x71d6 bytes) 000831 1787861958.130095 fallocate 0x01dab3 thru 0x02940f (0xb95d bytes) 000832 1787861958.136317 fallocate 0x021010 thru 0x02d059 (0xc04a bytes) 000833 1787861958.142783 mapwrite 0x012850 thru 0x01ce50 (0xa601 bytes) 000834 1787861958.171129 mapwrite 0x005769 thru 0x00b10a (0x59a2 bytes) 000835 1787861958.190957 trunc from 0x040000 to 0x00a680 (0x3597f bytes) 000837 1787861958.211789 mapwrite 0x000f80 thru 0x006d9c (0x5e1d bytes) 000838 1787861958.233769 MIRROR_RESYNC 6 mirrors 000839 1787861958.974296 read 0x00957c thru 0x00a680 (0x1105 bytes) 000840 1787861958.984871 mapread 0x003653 thru 0x00a680 (0x702e bytes) 000841 1787861959.007073 punch from 0x0004a2 to 0x000974 (0x4d3 bytes) 000842 1787861959.046484 write 0x007c6d thru 0x008567 (0x8fb bytes) 000843 1787861959.055914 punch from 0x0066b2 to 0x00a680 (0x3fcf bytes) 000844 1787861959.069215 mapwrite 0x012b15 thru 0x01f855 (0xcd41 bytes) 000846 1787861959.166411 punch from 0x00bb67 to 0x015844 (0x9cde bytes) 000847 1787861959.173661 punch from 0x0184cb to 0x01f855 (0x738b bytes) 000848 1787861959.179021 mapwrite 0x037a87 thru 0x03ffff (0x8579 bytes) 000849 1787861959.238140 punch from 0x003f86 to 0x00dafe (0x9b79 bytes) 000850 1787861959.251248 MIRROR_RESYNC 6 mirrors 000851 1787861960.830494 mapread 0x008883 thru 0x0176f7 (0xee75 bytes) 000852 1787861960.883818 fallocate 0x0034bf thru 0x00a1e3 (0x6d25 bytes) 000853 1787861960.931309 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x784203ba (stale) Mirror 22: 0x784203ba (stale) Mirror 25: 0x784203ba (stale) Mirror 26: 0x784203ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x784203ba (stale) Mirror 22: 0x784203ba (stale) Mirror 25: 0x784203ba (stale) Mirror 26: 0x784203ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000854 1787861961.492435 fallocate 0x031fb3 thru 0x03d212 (0xb260 bytes) 000855 1787861961.500588 trunc from 0x00aa3b to 0x01237b (0x7941 bytes) 000856 1787861961.508773 mapwrite 0x0286a8 thru 0x033529 (0xae82 bytes) 000857 1787861961.603305 trunc from 0x015157 to 0x02505e (0xff08 bytes) 000859 1787861961.611607 fallocate 0x00f9dc thru 0x01e178 (0xe79d bytes) 000860 1787861961.618470 read 0x0179a3 thru 0x022d89 (0xb3e7 bytes) 000861 1787861961.628915 write 0x00094a thru 0x00bd77 (0xb42e bytes) 000862 1787861961.646569 fallocate 0x00a24d thru 0x016069 (0xbe1d bytes) 000863 1787861961.652569 read 0x0166d5 thru 0x01b57c (0x4ea8 bytes) 000864 1787861961.664253 trunc from 0x040000 to 0x016609 (0x299f6 bytes) 000865 1787861961.683354 trunc from 0x01254a to 0x016609 (0x40c0 bytes) 000867 1787861961.694331 MIRROR_RESYNC 5 mirrors 000869 1787861962.519654 trunc from 0x006320 to 0x0066d1 (0x3b2 bytes) 000871 1787861962.597073 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1660a): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1660a): Mirror 16: 0xe76c5911 (stale) Mirror 22: 0xe76c5911 (stale) Mirror 25: 0xe76c5911 (stale) Mirror 26: 0xe76c5911 (stale) CRC-32 checksum value for chunk [0, 0x1660a): Mirror 1: 0x34df1700 Mirror 27: 0x34df1700 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1660a: skipped 000872 1787861963.845424 mapwrite 0x032db7 thru 0x03c54b (0x9795 bytes) 000873 1787861963.980451 trunc from 0x026921 to 0x034c3b (0xe31b bytes) 000875 1787861963.991023 trunc from 0x037eb5 to 0x03c54b (0x4697 bytes) 000877 1787861963.999169 write 0x02703f thru 0x02ca5b (0x5a1d bytes) 000878 1787861964.013148 fallocate 0x0210e7 thru 0x024a81 (0x399b bytes) 000880 1787861964.021200 trunc from 0x03c54c to 0x0093cb (0x33180 bytes) 000881 1787861964.055317 write 0x03d278 thru 0x03eeeb (0x1c74 bytes) 000882 1787861964.058279 trunc from 0x03eeec to 0x022a9d (0x1c44e bytes) 000883 1787861964.092466 trunc from 0x013db7 to 0x0200a7 (0xc2f1 bytes) 000884 1787861964.106313 MIRROR_RESYNC 6 mirrors 000886 1787861965.207802 trunc from 0x01e34a to 0x022a9d (0x4754 bytes) 000887 1787861965.260449 mapwrite 0x00ae64 thru 0x01590d (0xaaaa bytes) 000888 1787861965.294960 punch from 0x019ebc to 0x022a9d (0x8be2 bytes) 000889 1787861965.298051 read 0x01676e thru 0x016c3c (0x4cf bytes) 000890 1787861965.298659 fallocate 0x017609 thru 0x021398 (0x9d90 bytes) 000891 1787861965.301607 mapwrite 0x00d7cc thru 0x017222 (0x9a57 bytes) 000892 1787861965.311719 punch from 0x00b404 to 0x010977 (0x5574 bytes) 000893 1787861965.317101 write 0x03c056 thru 0x03d819 (0x17c4 bytes) 000894 1787861965.318381 trunc from 0x03d81a to 0x01c7ee (0x2102b bytes) 000895 1787861965.343695 fallocate 0x03d33e thru 0x03e4a0 (0x1163 bytes) 000896 1787861965.357221 write 0x0376e4 thru 0x03ffff (0x891c bytes) 000897 1787861965.362832 trunc from 0x026f72 to 0x02c269 (0x52f8 bytes) 000898 1787861965.368554 mapread 0x03e0f3 thru 0x03ffff (0x1f0d bytes) 000900 1787861965.375245 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x47810641 (stale) Mirror 22: 0x47810641 (stale) Mirror 25: 0x47810641 (stale) Mirror 26: 0x47810641 (stale) Mirror 27: 0x47810641 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xda2ab1fd Mirror 28: 0xda2ab1fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000901 1787861966.461032 mapwrite 0x00e689 thru 0x00f82c (0x11a4 bytes) 000902 1787861966.543793 mapwrite 0x028704 thru 0x0375c4 (0xeec1 bytes) 000903 1787861966.638913 read 0x032921 thru 0x038ded (0x64cd bytes) 000904 1787861966.639697 write 0x024294 thru 0x02aca4 (0x6a11 bytes) 000905 1787861966.645189 write 0x01cb49 thru 0x0238c6 (0x6d7e bytes) 000906 1787861966.654073 read 0x00dd1f thru 0x0162d8 (0x85ba bytes) 000907 1787861966.668109 MIRROR_RESYNC 7 mirrors 000908 1787861968.447116 trunc from 0x00e407 to 0x019810 (0xb40a bytes) 000910 1787861968.502376 read 0x022e9a thru 0x02f9cc (0xcb33 bytes) 000911 1787861968.516162 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x3d598180 (stale) Mirror 22: 0x3d598180 (stale) Mirror 25: 0x3d598180 (stale) Mirror 26: 0x3d598180 (stale) Mirror 27: 0x3d598180 (stale) Mirror 28: 0x3d598180 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f95c1fe Mirror 29: 0x2f95c1fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1787861969.538838 trunc from 0x015c51 to 0x023079 (0xd429 bytes) 000913 1787861969.609081 trunc from 0x00c001 to 0x01aae6 (0xeae6 bytes) 000914 1787861969.619662 fallocate 0x02e9bd thru 0x033776 (0x4dba bytes) 000915 1787861969.625550 write 0x034a16 thru 0x03ed98 (0xa383 bytes) 000916 1787861969.640633 fallocate 0x013cb2 thru 0x022aa8 (0xedf7 bytes) 000917 1787861969.649895 MIRROR_RESYNC 8 mirrors 000918 1787861970.814033 mapread 0x02b2dd thru 0x03ab60 (0xf884 bytes) 000919 1787861970.844322 mapwrite 0x0061fa thru 0x00a9d7 (0x47de bytes) 000921 1787861970.953350 trunc from 0x040000 to 0x030da5 (0xf25a bytes) 000922 1787861970.985892 read 0x02fa74 thru 0x030da5 (0x1332 bytes) 000923 1787861970.995323 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30da6): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30da6): Mirror 16: 0x97c40c66 (stale) Mirror 22: 0x97c40c66 (stale) Mirror 25: 0x97c40c66 (stale) Mirror 26: 0x97c40c66 (stale) Mirror 27: 0x97c40c66 (stale) Mirror 28: 0x97c40c66 (stale) Mirror 29: 0x97c40c66 (stale) CRC-32 checksum value for chunk [0, 0x30da6): Mirror 1: 0xf54a9601 Mirror 30: 0xf54a9601 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30da6: skipped 000925 1787861972.115558 mapwrite 0x020383 thru 0x0209a2 (0x620 bytes) 000927 1787861972.211360 trunc from 0x00c631 to 0x00dc02 (0x15d2 bytes) 000928 1787861972.222080 punch from 0x008ed9 to 0x014698 (0xb7c0 bytes) 000929 1787861972.235648 mapread 0x00dd8d thru 0x016d32 (0x8fa6 bytes) 000930 1787861972.267450 trunc from 0x030da6 to 0x0363cf (0x562a bytes) 000931 1787861972.316224 punch from 0x0064cb to 0x0129c6 (0xc4fc bytes) 000933 1787861972.326744 trunc from 0x02fa12 to 0x0319e6 (0x1fd5 bytes) 000934 1787861972.333451 trunc from 0x0363d0 to 0x029fdb (0xc3f4 bytes) 000935 1787861972.352342 punch from 0x0005a4 to 0x00e506 (0xdf63 bytes) 000936 1787861972.361829 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x29fdc): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fdc): Mirror 16: 0xc2b36b5e (stale) Mirror 22: 0xc2b36b5e (stale) Mirror 25: 0xc2b36b5e (stale) Mirror 26: 0xc2b36b5e (stale) Mirror 27: 0xc2b36b5e (stale) Mirror 28: 0xc2b36b5e (stale) Mirror 29: 0xc2b36b5e (stale) Mirror 30: 0x32f87f1c (stale) CRC-32 checksum value for chunk [0, 0x29fdc): Mirror 1: 0xeb431c7e Mirror 31: 0xeb431c7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fdc: skipped 000937 1787861973.446609 MIRROR_RESYNC 10 mirrors 000939 1787861975.532554 read 0x005682 thru 0x009ef6 (0x4875 bytes) 000940 1787861975.540954 punch from 0x00105e to 0x010c96 (0xfc39 bytes) 000941 1787861975.599902 write 0x0257c9 thru 0x0347c6 (0xeffe bytes) 000942 1787861975.611882 trunc from 0x0347c7 to 0x03154f (0x3277 bytes) 000943 1787861975.679721 trunc from 0x0014cf to 0x00dce0 (0xc812 bytes) 000944 1787861975.685198 write 0x00d50f thru 0x01b6dc (0xe1ce bytes) 000945 1787861975.706759 mapread 0x01324b thru 0x0213d5 (0xe18b bytes) 000946 1787861975.737649 fallocate 0x03f1e7 thru 0x03ffff (0xe19 bytes) 000947 1787861975.747413 mapwrite 0x016ba9 thru 0x01c43d (0x5895 bytes) 000948 1787861975.770312 trunc from 0x031550 to 0x00469d (0x2ceb2 bytes) 000949 1787861975.787638 punch from 0x0001fc to 0x00469d (0x44a2 bytes) 000950 1787861975.795862 trunc from 0x00469e to 0x02e7c4 (0x2a127 bytes) 000951 1787861975.817261 trunc from 0x02c4db to 0x02e7c4 (0x22ea bytes) 000952 1787861975.827974 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e7c5): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7c5): Mirror 16: 0xc5068b12 (stale) Mirror 22: 0xc5068b12 (stale) Mirror 25: 0xc5068b12 (stale) Mirror 26: 0xc5068b12 (stale) Mirror 27: 0xc5068b12 (stale) Mirror 28: 0xc5068b12 (stale) Mirror 29: 0xc5068b12 (stale) Mirror 30: 0xc5068b12 (stale) Mirror 31: 0xc5068b12 (stale) CRC-32 checksum value for chunk [0, 0x2e7c5): Mirror 1: 0xaf6ccc6b Mirror 32: 0xaf6ccc6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7c5: skipped 000953 1787861976.861593 mapwrite 0x00e2d8 thru 0x01a6f9 (0xc422 bytes) 000954 1787861976.981495 mapwrite 0x0132b9 thru 0x01c482 (0x91ca bytes) 000955 1787861976.999579 fallocate 0x03b79a thru 0x03ffff (0x4866 bytes) 000956 1787861977.004935 punch from 0x016649 to 0x0169ed (0x3a5 bytes) 000957 1787861977.009790 mapwrite 0x026249 thru 0x026333 (0xeb bytes) 000958 1787861977.031025 mapwrite 0x01b201 thru 0x023f18 (0x8d18 bytes) 000959 1787861977.049795 read 0x01eefc thru 0x02c30d (0xd412 bytes) 000960 1787861977.050536 mapread 0x01c780 thru 0x027e29 (0xb6aa bytes) 000961 1787861977.051483 fallocate 0x0346d5 thru 0x03a772 (0x609e bytes) 000962 1787861977.058581 mapread 0x0014ba thru 0x010e5e (0xf9a5 bytes) 000963 1787861977.068548 trunc from 0x02e7c5 to 0x00233a (0x2c48a bytes) 000964 1787861977.098862 punch from 0x00125d to 0x00233a (0x10de bytes) 000965 1787861977.104014 trunc from 0x00233b to 0x03f5bd (0x3d283 bytes) 000966 1787861977.132948 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f5be): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5be): Mirror 16: 0x8df173f3 (stale) Mirror 22: 0x8df173f3 (stale) Mirror 25: 0x8df173f3 (stale) Mirror 26: 0x8df173f3 (stale) Mirror 27: 0x8df173f3 (stale) Mirror 28: 0x8df173f3 (stale) Mirror 29: 0x8df173f3 (stale) Mirror 30: 0x8df173f3 (stale) Mirror 31: 0x8df173f3 (stale) Mirror 32: 0x42a82076 (stale) CRC-32 checksum value for chunk [0, 0x3f5be): Mirror 1: 0x42a82076 Mirror 33: 0x42a82076 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5be: skipped 000968 1787861978.112177 write 0x026723 thru 0x02ca71 (0x634f bytes) 000969 1787861978.184436 trunc from 0x03f5be to 0x032e88 (0xc735 bytes) 000970 1787861978.204005 mapwrite 0x0052e6 thru 0x009493 (0x41ae bytes) 000971 1787861978.252850 trunc from 0x032e89 to 0x0200c5 (0x12dc3 bytes) 000972 1787861978.271020 punch from 0x000a0a to 0x002bd8 (0x21cf bytes) 000973 1787861978.276946 mapwrite 0x0397e0 thru 0x03ffff (0x6820 bytes) 000974 1787861978.319211 write 0x00c555 thru 0x016e6e (0xa91a bytes) 000976 1787861978.327193 mapwrite 0x0372a4 thru 0x03ffff (0x8d5c bytes) 000977 1787861978.347573 mapwrite 0x036d0e thru 0x03ffff (0x92f2 bytes) 000978 1787861978.365759 write 0x01ab9e thru 0x025a6c (0xaecf bytes) 000979 1787861978.379934 mapwrite 0x02f168 thru 0x03db5a (0xe9f3 bytes) 000980 1787861978.420989 trunc from 0x02733c to 0x029847 (0x250c bytes) 000981 1787861978.425768 read 0x00726c thru 0x007b4b (0x8e0 bytes) 000982 1787861978.426413 trunc from 0x040000 to 0x02163b (0x1e9c4 bytes) 000983 1787861978.454164 mapread 0x00484f thru 0x0091a8 (0x495a bytes) 000985 1787861978.461469 MIRROR_RESYNC 12 mirrors 000986 1787861981.088434 trunc from 0x00d9a1 to 0x0101aa (0x280a bytes) 000988 1787861981.139403 trunc from 0x02163c to 0x030316 (0xecdb bytes) 000989 1787861981.163217 mapread 0x00bf4e thru 0x00c9b9 (0xa6c bytes) 000991 1787861981.180975 read 0x002a4c thru 0x00cb9b (0xa150 bytes) 000992 1787861981.198535 mapwrite 0x00818a thru 0x0137d0 (0xb647 bytes) 000993 1787861981.239485 mapwrite 0x0016bb thru 0x00f22a (0xdb70 bytes) 000994 1787861981.282452 punch from 0x023c0f to 0x02967f (0x5a71 bytes) 000995 1787861981.296575 mapread 0x0087b5 thru 0x00c550 (0x3d9c bytes) 000996 1787861981.298740 mapwrite 0x0229f0 thru 0x024555 (0x1b66 bytes) 000997 1787861981.318863 mapwrite 0x028a80 thru 0x0386e7 (0xfc68 bytes) 000998 1787861981.369780 read 0x00bb7b thru 0x018cd5 (0xd15b bytes) 000999 1787861981.378530 trunc from 0x009464 to 0x00ba9a (0x2637 bytes) 001000 1787861981.383340 trunc from 0x0386e8 to 0x00fe06 (0x288e1 bytes) All operations completed A-OK! PASS 70a (130s) == sanity-flr test 71: check mirror extend parent fid ==== 16:19:46 (1787861986) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 16:19:57 (1787861997) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246201 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680989 s, 15.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 16:20:07 (1787862007) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3857408 .. Extending file size to 4847840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8423712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6137632 .. Extending file size to 7780192 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8549600 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8629280 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8892416 .. Extending file size to 8802656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62777 62778 62779 62781 62784 failed 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:21:31 (1787862091) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4909376 .. Extending file size to 2334688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7574944 .. Extending file size to 5824544 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7122432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7389568 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7942912 .. Extending file size to 8254624 .. Extending file size to 8882592 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8639680 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8815200 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 64313 64314 64316 64317 64319 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:22:51 (1787862171) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1268820 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24188 3582832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18552 3588468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42740 7171300 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1268820 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24188 3582832 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18552 3588468 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42740 7171300 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151608 s, 13.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170681 s, 12.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x62d77201 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:23:11 (1787862191) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0806703 s, 26.0 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 16:23:17 (1787862197) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fb:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:23:25 (1787862205) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0969124 s, 21.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:23:32 (1787862212) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101159 s, 20.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:23:40 (1787862220) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164107 s, 12.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:24:13 (1787862253) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882953 s, 11.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:25:12 (1787862312) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142698 s, 14.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:26:03 (1787862363) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg138-server: Pool lustre.test_204f created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195135 s, 10.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg138-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:27:27 (1787862447) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:27:36 (1787862456) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:27:44 (1787862464) oleg138-server: Pool lustre.test_206 created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg138-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:28:14 (1787862494) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752625 s, 13.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264181 s, 39.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf36f023d Mirror 3: 0xf36f023d PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:28:26 (1787862506) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:28:30 (1787862510) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:28:34 (1787862514) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.068914 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2210000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x8d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x8d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x8d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9e0000 Read this (4 bytes): BBBB 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:29:14 (1787862554) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0188424 s, 435 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:29:23 (1787862563) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643928 s, 16.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x308:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611639 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:29:25.000000000 -0400 Modify: 2026-08-27 16:29:26.000000000 -0400 Change: 2026-08-27 16:29:27.000000000 -0400 Birth: 2026-08-27 16:29:25.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:29:33 (1787862573) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608172 s, 17.2 MB/s oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg138-server: O/280000401/d7/327: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:29:56 (1787862596) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.176753 s, 232 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0252889 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 16:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 16:29 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:30:03 (1787862603) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:30:10 (1787862610) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1964 sec =========== 16:30:20 (1787862620) osc.lustre-OST0000-osc-ffff94b91a168800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff94b91a168800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1787862622.867042200 secs.nsecs start_time 1787861145.143529801 secs.nsecs elapsed_time 1477.723512399 secs.nsecs uncompressed_objects 2946 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1787862622.867361160 secs.nsecs start_time 1787861185.649437659 secs.nsecs elapsed_time 1437.217923501 secs.nsecs uncompressed_objects 2934 samples [reqs] === sanity-flr: start cleanup 16:30:23 (1787862623) === === sanity-flr: finish cleanup 16:30:31 (1787862631) ===