-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 07:28:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg357-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:28:42 (1787830122) === oleg357-client.virtnet: executing check_config_client /mnt/lustre oleg357-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg357-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93af98a4a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93af98a4a800.idle_timeout=debug disable quota as required oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:28:59 (1787830139) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8207] == sanity-flr test 0a: lfs mirror create with -N option == 07:28:59 (1787830139) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:29:06 (1787830146) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:29:07 (1787830147) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:29:09 (1787830149) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:29:19 (1787830159) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:29:21 (1787830161) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:29:22 (1787830162) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93af98a4a800.stats=clear osc.lustre-OST0001-osc-ffff93af98a4a800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:29:26 (1787830166) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:29:30 (1787830170) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:29:34 (1787830174) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:29:35 (1787830175) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:29:39 (1787830179) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 07:29:43 (1787830183) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:29:47 (1787830187) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:29:50 (1787830190) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:29:54 (1787830194) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:29:59 (1787830199) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:30:02 (1787830202) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.330732 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443457 s, 23.6 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:30:05 (1787830205) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0648982 s, 80.8 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:30:08 (1787830208) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:30:13 (1787830213) Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:30:45 (1787830245) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.101451 s, 41.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:31:03 (1787830263) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:31:39 (1787830299) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45137 s, 36.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 07:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.357s user 0m0.000s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.386s user 0m0.000s sys 0m0.157s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.450s user 0m0.000s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93af98a4a800.ost_server_uuid in FULL state after 0 sec PASS 33b (40s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:32:20 (1787830340) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:32:21 (1787830341) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:32:23 (1787830343) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 07:32:24 (1787830344) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 07:32:27 (1787830347) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.208041 s, 60.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:32:32 (1787830352) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138192 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164813 s, 63.6 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:32:35 (1787830355) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:32:39 (1787830359) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000665316 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000738725 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000707401 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000776727 s, 1.3 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 07:32:42 (1787830362) create test files with size 15754959 .. b977c3bc3466170acb0c6f4d2984e050 - 39b82b2c75bb587801183c5b81d47aeb - 48e2886fabe9323a50265d0457646279 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 07:32:50 (1787830370) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:32:56 (1787830376) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:32:59 (1787830379) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400628 s, 26.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156397 s, 67.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 07:33:03 (1787830383) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.050833 s, 41.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.144023 s, 29.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.087362 s, 24.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 07:33:09 (1787830389) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 07:33:11 (1787830391) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:33:12 (1787830392) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145961 s, 71.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 43c: read prefer uncompressed mirror == 07:33:16 (1787830396) llite.lustre-ffff93af86764000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (3s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 07:33:19 (1787830399) llite.lustre-ffff93af86764000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143185 s, 73.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150732 s, 69.6 MB/s **verify components PASS 43d (3s) == sanity-flr test 44a: lfs mirror split check =========== 07:33:22 (1787830402) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.120255 s, 26.2 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:33:27 (1787830407) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:33:31 (1787830411) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.144845 s, 72.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (7s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:33:38 (1787830418) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff93af82e82000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.119655 s, 87.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93af99116000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:33:44 (1787830424) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:34:00 (1787830440) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:34:15 (1787830455) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:34:17 (1787830457) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.109789 s, 28.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0724829 s, 43.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8d951fe8 (nosync,stale) Mirror 2: 0x8d951fe8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 388de84aadea6ca3dfa4e6c283fad06e - mirror 1 checksum: c6c75959bb72cd229c1ae3f22b705fd4 - mirror 2 checksum: 388de84aadea6ca3dfa4e6c283fad06e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:34:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:34:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 388de84aadea6ca3dfa4e6c283fad06e - mirror 1 checksum: c6c75959bb72cd229c1ae3f22b705fd4 - mirror 2 checksum: c6c75959bb72cd229c1ae3f22b705fd4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:34:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:34:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:34:21 (1787830461) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152523 s, 68.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:34:24 (1787830464) Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server 07:34:33 (1787830473) shut down facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server Failover mds1 to oleg357-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 07:34:49 (1787830489) targets are mounted 07:34:49 (1787830489) facet_failover done oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219442 s, 47.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:35:02 (1787830502) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0238924 s, 857 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128752 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:35:07 (1787830507) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0285227 s, 718 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00682807 s, 3.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:35:12 (1787830512) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00448312 s, 3.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:35:16 (1787830516) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0226664 s, 904 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:35:20 (1787830520) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00519529 s, 788 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0133555 s, 307 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:35:23 (1787830523) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:35:32 (1787830532) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:35:54 (1787830554) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:36:13 (1787830573) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:36:32 (1787830592) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 07:36:35 (1787830595) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58762 mirror split pid 58763 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58762 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58763 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 07:37:38 (1787830658) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 660 skipping zero size read 000002 1787830660.276671 MIRROR_EXTEND to 2 mirrors 000003 1787830660.458282 write 0x00fffa thru 0x01e8ef (0xe8f6 bytes) 000004 1787830660.479009 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000005 1787830660.586001 write 0x02490f thru 0x025da0 (0x1492 bytes) fallocating to largest ever: 0x12b39 000006 1787830660.590480 fallocate 0x00342e thru 0x012b38 (0xf70b bytes) 000007 1787830660.595491 MIRROR_EXTEND to 2 mirrors 000008 1787830660.968090 trunc from 0x025da1 to 0x012acc (0x132d4 bytes) 000009 1787830661.017977 trunc from 0x00fb80 to 0x012acc (0x2f4d bytes) fallocating to largest ever: 0x40000 000010 1787830661.024382 fallocate 0x03db2e thru 0x03ffff (0x24d2 bytes) 000011 1787830661.031846 trunc from 0x01fb8c to 0x023e5e (0x42d3 bytes) 000012 1787830661.037553 punch from 0x03b52d to 0x03ffff (0x4ad3 bytes) 000013 1787830661.043875 punch from 0x01a11e to 0x025dba (0xbc9d bytes) 000014 1787830661.050778 trunc from 0x027eca to 0x0285f4 (0x72b bytes) 000015 1787830661.056484 write 0x002a4a thru 0x00bf3e (0x94f5 bytes) 000016 1787830661.068380 write 0x0383d5 thru 0x03ffff (0x7c2b bytes) 000017 1787830661.073349 mapwrite 0x03f7ac thru 0x03ffff (0x854 bytes) 000018 1787830661.097726 trunc from 0x03fab9 to 0x03ffff (0x547 bytes) 000019 1787830661.104538 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000020 1787830661.230763 trunc from 0x040000 to 0x001b4d (0x3e4b2 bytes) 000021 1787830661.261862 read 0x00040d thru 0x001b4d (0x1741 bytes) 000023 1787830661.268669 trunc from 0x000054 to 0x001b4d (0x1afa bytes) 000024 1787830661.274391 read 0x0017ab thru 0x001b4d (0x3a3 bytes) 000027 1787830661.280485 read 0x000189 thru 0x001b4d (0x19c5 bytes) 000028 1787830661.281315 trunc from 0x0005a4 to 0x000e4e (0x8ab bytes) 000029 1787830661.289735 mapread 0x000f3c thru 0x001b4d (0xc12 bytes) 000030 1787830661.295400 mapread 0x001045 thru 0x001b4d (0xb09 bytes) 000031 1787830661.296425 trunc from 0x0000ae to 0x001b4d (0x1aa0 bytes) 000032 1787830661.301825 mapwrite 0x015fdf thru 0x023a54 (0xda76 bytes) 000033 1787830661.347304 punch from 0x01bcde to 0x01d3f0 (0x1713 bytes) 000034 1787830661.355977 fallocate 0x037bad thru 0x03a23b (0x268f bytes) 000035 1787830661.361200 mapread 0x0119b5 thru 0x011d6b (0x3b7 bytes) 000036 1787830661.363047 write 0x038df8 thru 0x03ebfd (0x5e06 bytes) 000037 1787830661.372512 trunc from 0x03ebfe to 0x0208fb (0x1e302 bytes) 000038 1787830661.390495 mapread 0x00fc73 thru 0x014a6f (0x4dfd bytes) 000039 1787830661.392268 trunc from 0x0208fc to 0x0280f3 (0x77f8 bytes) 000040 1787830661.404614 fallocate 0x001318 thru 0x00f677 (0xe360 bytes) 000041 1787830661.414235 mapread 0x00130a thru 0x008f6e (0x7c65 bytes) 000042 1787830661.419526 mapwrite 0x02461e thru 0x0267a7 (0x218a bytes) 000043 1787830661.448164 write 0x00d739 thru 0x01056e (0x2e36 bytes) 000044 1787830661.453646 MIRROR_EXTEND to 2 mirrors 000045 1787830662.003447 punch from 0x027100 to 0x0280f3 (0xff4 bytes) 000046 1787830662.039831 trunc from 0x0280f4 to 0x03b3fa (0x13307 bytes) 000047 1787830662.079087 mapread 0x001eab thru 0x00c9a0 (0xaaf6 bytes) 000048 1787830662.096115 punch from 0x00a3d5 to 0x00bcf0 (0x191c bytes) 000049 1787830662.108285 trunc from 0x01e349 to 0x02748d (0x9145 bytes) 000050 1787830662.113274 punch from 0x012313 to 0x01f8d1 (0xd5bf bytes) 000051 1787830662.117568 trunc from 0x03b3fb to 0x00b388 (0x30072 bytes) 000053 1787830662.135282 write 0x03fde1 thru 0x03ffff (0x21f bytes) 000054 1787830662.136428 trunc from 0x040000 to 0x005ee2 (0x3a11d bytes) 000055 1787830662.152125 mapwrite 0x011745 thru 0x0216dd (0xff99 bytes) 000056 1787830662.221188 mapread 0x01d270 thru 0x020d1b (0x3aac bytes) 000057 1787830662.222006 trunc from 0x0216de to 0x031f4f (0x10872 bytes) 000058 1787830662.237861 read 0x0056ab thru 0x013a06 (0xe35c bytes) 000059 1787830662.243304 mapread 0x01ebbe thru 0x02bd00 (0xd143 bytes) 000060 1787830662.271603 trunc from 0x0228f5 to 0x0251e5 (0x28f1 bytes) 000061 1787830662.276685 trunc from 0x031f50 to 0x037ef5 (0x5fa6 bytes) 000062 1787830662.289910 mapwrite 0x01dff5 thru 0x02445d (0x6469 bytes) 000063 1787830662.319245 read 0x00383d thru 0x00a304 (0x6ac8 bytes) 000064 1787830662.320102 mapwrite 0x024cca thru 0x0276a5 (0x29dc bytes) 000065 1787830662.329118 read 0x01ae20 thru 0x0222cd (0x74ae bytes) 000066 1787830662.329773 write 0x0148a5 thru 0x01663a (0x1d96 bytes) 000067 1787830662.331356 punch from 0x0142f3 to 0x016b4d (0x285b bytes) 000069 1787830662.338546 fallocate 0x0110ad thru 0x01bdaa (0xacfe bytes) 000070 1787830662.342004 write 0x035a5c thru 0x03b802 (0x5da7 bytes) 000071 1787830662.350095 mapwrite 0x038cfe thru 0x03dd8c (0x508f bytes) 000072 1787830662.374546 trunc from 0x01b0c9 to 0x01debf (0x2df7 bytes) 000073 1787830662.380053 trunc from 0x03dd8d to 0x02bf3b (0x11e51 bytes) 000074 1787830662.396182 trunc from 0x028d43 to 0x02bf3b (0x31f9 bytes) 000075 1787830662.404168 punch from 0x003a53 to 0x00e316 (0xa8c4 bytes) 000076 1787830662.411643 trunc from 0x022b8d to 0x02937b (0x67ef bytes) 000077 1787830662.415861 write 0x0191cb thru 0x022823 (0x9659 bytes) 000078 1787830662.421488 read 0x00f715 thru 0x00fca0 (0x58c bytes) 000079 1787830662.427264 MIRROR_RESYNC 2 mirrors 000080 1787830662.851677 write 0x000f57 thru 0x00741c (0x64c6 bytes) 000081 1787830662.885394 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000082 1787830662.993828 trunc from 0x0087a2 to 0x00952f (0xd8e bytes) 000083 1787830662.998100 fallocate 0x023c21 thru 0x02d051 (0x9431 bytes) 000084 1787830663.003328 mapwrite 0x023631 thru 0x033291 (0xfc61 bytes) 000085 1787830663.098632 MIRROR_EXTEND to 2 mirrors 000087 1787830663.650784 MIRROR_EXTEND to 3 mirrors 000088 1787830664.167737 MIRROR_EXTEND to 4 mirrors 000089 1787830664.707833 punch from 0x03140a to 0x033291 (0x1e88 bytes) 000091 1787830664.744555 MIRROR_RESYNC 4 mirrors 000092 1787830665.678244 write 0x00eee1 thru 0x012b25 (0x3c45 bytes) 000093 1787830665.723303 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33292): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33292): Mirror 2: 0x35ffa9f0 (stale) Mirror 3: 0x35ffa9f0 (stale) Mirror 4: 0x35ffa9f0 (stale) CRC-32 checksum value for chunk [0, 0x33292): Mirror 1: 0xfef4fded Mirror 5: 0xfef4fded lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33292: skipped 000094 1787830666.550398 mapwrite 0x02290f thru 0x02a6aa (0x7d9c bytes) 000095 1787830666.634997 write 0x02f189 thru 0x03958e (0xa406 bytes) 000096 1787830666.642460 punch from 0x01f775 to 0x026d25 (0x75b1 bytes) 000098 1787830666.651732 trunc from 0x03958f to 0x038ed8 (0x6b6 bytes) 000099 1787830666.687327 trunc from 0x02a2db to 0x037a60 (0xd786 bytes) 000100 1787830666.706890 punch from 0x038790 to 0x038ed8 (0x749 bytes) 000101 1787830666.711432 write 0x02b0ba thru 0x032f24 (0x7e6b bytes) 000102 1787830666.718884 punch from 0x00f50e to 0x01396b (0x445e bytes) 000103 1787830666.723310 trunc from 0x01288d to 0x01a4af (0x7c23 bytes) 000104 1787830666.728233 write 0x01ebc9 thru 0x02c22c (0xd664 bytes) 000105 1787830666.730416 trunc from 0x006200 to 0x0127b6 (0xc5b7 bytes) 000107 1787830666.736162 mapread 0x02f119 thru 0x038ed8 (0x9dc0 bytes) 000108 1787830666.749246 trunc from 0x038ed9 to 0x017266 (0x21c72 bytes) 000109 1787830666.772630 mapwrite 0x005e1d thru 0x01109e (0xb282 bytes) 000110 1787830666.813471 trunc from 0x000478 to 0x00c1b4 (0xbd3d bytes) 000111 1787830666.822912 fallocate 0x025d63 thru 0x034734 (0xe9d2 bytes) 000112 1787830666.834781 fallocate 0x03eedd thru 0x03ffff (0x1123 bytes) 000113 1787830666.851230 write 0x0294c8 thru 0x02cc87 (0x37c0 bytes) 000114 1787830666.863336 trunc from 0x040000 to 0x008345 (0x37cba bytes) 000115 1787830666.883521 trunc from 0x008346 to 0x01f4f4 (0x171af bytes) 000116 1787830666.904227 punch from 0x01c533 to 0x01f4f4 (0x2fc2 bytes) 000117 1787830666.908467 mapread 0x01adc3 thru 0x01f4f4 (0x4732 bytes) 000118 1787830666.921381 punch from 0x01db43 to 0x01ea98 (0xf56 bytes) 000119 1787830666.932300 MIRROR_RESYNC 5 mirrors 000120 1787830667.586747 MIRROR_RESYNC 5 mirrors 000122 1787830667.912668 trunc from 0x01a531 to 0x01e899 (0x4369 bytes) 000123 1787830667.941389 trunc from 0x01f4f5 to 0x020993 (0x149f bytes) 000124 1787830667.967770 read 0x001af1 thru 0x00398e (0x1e9e bytes) 000125 1787830667.985348 read 0x00e7c4 thru 0x00feac (0x16e9 bytes) 000126 1787830667.997600 trunc from 0x020994 to 0x01993e (0x7055 bytes) 000127 1787830668.017640 read 0x0140ac thru 0x0163fa (0x234f bytes) 000129 1787830668.019256 trunc from 0x01993f to 0x0199d4 (0x96 bytes) 000130 1787830668.038790 punch from 0x016986 to 0x0199d4 (0x304f bytes) 000131 1787830668.046050 trunc from 0x01300e to 0x0199d4 (0x69c7 bytes) 000132 1787830668.053363 fallocate 0x03c089 thru 0x03ffff (0x3f77 bytes) 000133 1787830668.064289 read 0x00b625 thru 0x010eed (0x58c9 bytes) 000135 1787830668.069901 read 0x029bba thru 0x02f21d (0x5664 bytes) 000136 1787830668.079970 fallocate 0x01e41d thru 0x02e2e0 (0xfec4 bytes) 000137 1787830668.091870 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 6: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1787830668.741814 trunc from 0x033eb0 to 0x03d787 (0x98d8 bytes) 000139 1787830668.779388 mapwrite 0x01f73c thru 0x020480 (0xd45 bytes) 000140 1787830668.814620 write 0x026082 thru 0x02c992 (0x6911 bytes) 000141 1787830668.836644 fallocate 0x015bb3 thru 0x01922c (0x367a bytes) 000144 1787830668.848126 trunc from 0x040000 to 0x0250e3 (0x1af1c bytes) 000145 1787830668.910857 mapread 0x01f2d8 thru 0x0250e3 (0x5e0c bytes) 000146 1787830668.924390 fallocate 0x01cb35 thru 0x020d1f (0x41eb bytes) 000147 1787830668.934449 trunc from 0x0250e4 to 0x014c2e (0x104b5 bytes) 000148 1787830668.959068 punch from 0x005d8e to 0x0118f8 (0xbb6b bytes) 000149 1787830668.968506 trunc from 0x014c2f to 0x014fad (0x37f bytes) 000150 1787830668.989890 trunc from 0x01253f to 0x014fad (0x2a6f bytes) 000151 1787830668.995688 write 0x03e282 thru 0x03f348 (0x10c7 bytes) 000152 1787830668.997891 read 0x030c20 thru 0x03b214 (0xa5f5 bytes) 000153 1787830669.005776 trunc from 0x03f349 to 0x0174f4 (0x27e54 bytes) 000154 1787830669.021235 trunc from 0x0174f5 to 0x009967 (0xdb8d bytes) 000155 1787830669.035239 mapread 0x0027ad thru 0x009967 (0x71bb bytes) 000156 1787830669.052213 mapwrite 0x014600 thru 0x01e136 (0x9b37 bytes) 000157 1787830669.107466 trunc from 0x0149b9 to 0x01e136 (0x977e bytes) 000158 1787830669.113293 read 0x0127da thru 0x01a7a3 (0x7fca bytes) 000159 1787830669.121690 fallocate 0x026bde thru 0x030aff (0x9f22 bytes) 000160 1787830669.126608 fallocate 0x00e2af thru 0x014c67 (0x69b9 bytes) 000161 1787830669.135089 mapread 0x00e7f4 thru 0x01999b (0xb1a8 bytes) 000162 1787830669.136535 punch from 0x01d5c9 to 0x01e136 (0xb6e bytes) 000163 1787830669.149248 MIRROR_RESYNC 6 mirrors 000164 1787830670.035592 trunc from 0x0149cd to 0x01e136 (0x976a bytes) 000165 1787830670.070226 mapwrite 0x0310c9 thru 0x03f9ca (0xe902 bytes) 000166 1787830670.158992 fallocate 0x02cc83 thru 0x032d85 (0x6103 bytes) 000167 1787830670.172706 mapwrite 0x009849 thru 0x015e77 (0xc62f bytes) 000168 1787830670.232587 fallocate 0x00c0a8 thru 0x016c07 (0xab60 bytes) 000169 1787830670.250704 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9cb): Mirror 2: 0x7788d45b (stale) Mirror 3: 0x7788d45b (stale) Mirror 5: 0x7788d45b (stale) Mirror 6: 0x7788d45b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9cb): Mirror 2: 0x7788d45b (stale) Mirror 3: 0x7788d45b (stale) Mirror 5: 0x7788d45b (stale) Mirror 6: 0x7788d45b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9cb: skipped 000170 1787830670.563396 mapread 0x009f2f thru 0x00ad42 (0xe14 bytes) 000171 1787830670.571470 write 0x00fb09 thru 0x010d03 (0x11fb bytes) 000172 1787830670.582131 write 0x03f8e0 thru 0x03ffff (0x720 bytes) 000173 1787830670.587413 punch from 0x03172b to 0x03a7ab (0x9081 bytes) 000174 1787830670.591500 mapwrite 0x001155 thru 0x00b36c (0xa218 bytes) 000175 1787830670.614630 mapwrite 0x00878c thru 0x00c0c0 (0x3935 bytes) 000176 1787830670.632112 write 0x02286b thru 0x0321d9 (0xf96f bytes) 000177 1787830670.640896 trunc from 0x03719c to 0x03ffff (0x8e64 bytes) 000178 1787830670.650188 fallocate 0x019450 thru 0x01e124 (0x4cd5 bytes) 000179 1787830670.661687 fallocate 0x03fb43 thru 0x03ffff (0x4bd bytes) 000180 1787830670.670736 mapwrite 0x008d2e thru 0x00a5c9 (0x189c bytes) 000181 1787830670.688700 punch from 0x0088e7 to 0x0119d5 (0x90ef bytes) 000182 1787830670.693599 write 0x02be9c thru 0x032bf0 (0x6d55 bytes) 000183 1787830670.695336 trunc from 0x010906 to 0x016c4a (0x6345 bytes) 000184 1787830670.702687 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbfeb81e6 (stale) Mirror 3: 0xbfeb81e6 (stale) Mirror 5: 0xbfeb81e6 (stale) Mirror 6: 0xbfeb81e6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc64a9461 Mirror 7: 0xc64a9461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000186 1787830671.388136 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbfeb81e6 (stale) Mirror 3: 0xbfeb81e6 (stale) Mirror 6: 0xbfeb81e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbfeb81e6 (stale) Mirror 3: 0xbfeb81e6 (stale) Mirror 6: 0xbfeb81e6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc64a9461 Mirror 7: 0xc64a9461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000187 1787830671.807789 trunc from 0x040000 to 0x016f3f (0x290c0 bytes) 000188 1787830671.869812 read 0x006e36 thru 0x00bb68 (0x4d33 bytes) 000189 1787830671.878830 trunc from 0x006c4b to 0x0149ae (0xdd64 bytes) 000190 1787830671.883421 punch from 0x0009c6 to 0x00c2b4 (0xb8ef bytes) 000191 1787830671.887780 read 0x00f2f0 thru 0x010e70 (0x1b81 bytes) 000193 1787830671.901911 fallocate 0x026d11 thru 0x02d210 (0x6500 bytes) 000194 1787830671.914057 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d211): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d211): Mirror 2: 0xf593cb72 (stale) Mirror 3: 0xf593cb72 (stale) Mirror 6: 0xf593cb72 (stale) Mirror 7: 0x59d30a3f (stale) CRC-32 checksum value for chunk [0, 0x2d211): Mirror 1: 0xfd7f73d8 Mirror 8: 0xfd7f73d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d211: skipped 000195 1787830672.555303 mapread 0x0068c3 thru 0x00fdae (0x94ec bytes) 000196 1787830672.574025 MIRROR_RESYNC 6 mirrors 000197 1787830673.192070 fallocate 0x01c97a thru 0x01d33e (0x9c5 bytes) 000198 1787830673.238002 punch from 0x029032 to 0x02d210 (0x41df bytes) 000199 1787830673.242021 write 0x02b4b5 thru 0x03b041 (0xfb8d bytes) 000200 1787830673.248119 write 0x0284b5 thru 0x0348f1 (0xc43d bytes) 000201 1787830673.258621 read 0x01a05b thru 0x028a6f (0xea15 bytes) 000202 1787830673.267671 trunc from 0x0002a1 to 0x00d716 (0xd476 bytes) 000203 1787830673.271314 punch from 0x007f6e to 0x0123af (0xa442 bytes) 000204 1787830673.275593 trunc from 0x03b042 to 0x0350ff (0x5f42 bytes) 000205 1787830673.293450 trunc from 0x003bdd to 0x008995 (0x4db9 bytes) 000208 1787830673.298406 mapwrite 0x00527e thru 0x010a6f (0xb7f2 bytes) 000209 1787830673.338411 trunc from 0x02b5fa to 0x02dd28 (0x272f bytes) 000211 1787830673.342461 mapwrite 0x01dfb8 thru 0x0206d4 (0x271d bytes) 000212 1787830673.351990 fallocate 0x03121b thru 0x032e33 (0x1c19 bytes) 000213 1787830673.357759 mapwrite 0x0237aa thru 0x023f1f (0x776 bytes) 000214 1787830673.370766 mapread 0x026537 thru 0x02bc70 (0x573a bytes) 000215 1787830673.376002 read 0x004a6d thru 0x0105df (0xbb73 bytes) 000216 1787830673.376916 mapread 0x001d3e thru 0x00ec08 (0xcecb bytes) 000217 1787830673.378019 mapread 0x029a9c thru 0x0315d3 (0x7b38 bytes) 000218 1787830673.383399 MIRROR_RESYNC 6 mirrors 000219 1787830674.831239 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000221 1787830675.085893 mapread 0x01e262 thru 0x02711d (0x8ebc bytes) 000222 1787830675.105682 write 0x00015c thru 0x004308 (0x41ad bytes) 000223 1787830675.158698 read 0x015624 thru 0x01d1a7 (0x7b84 bytes) 000224 1787830675.172192 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35100): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35100): Mirror 3: 0xcd6aec4a (stale) Mirror 6: 0xcd6aec4a (stale) Mirror 7: 0xcd6aec4a (stale) Mirror 8: 0xcd6aec4a (stale) CRC-32 checksum value for chunk [0, 0x35100): Mirror 1: 0xae3583bd Mirror 9: 0xae3583bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35100: skipped 000225 1787830675.902897 fallocate 0x02e025 thru 0x02fce7 (0x1cc3 bytes) 000226 1787830675.934193 read 0x01c127 thru 0x0290d6 (0xcfb0 bytes) 000228 1787830675.942358 write 0x0131bd thru 0x02180d (0xe651 bytes) 000230 1787830675.954382 trunc from 0x035100 to 0x007f10 (0x2d1ef bytes) 000231 1787830675.987730 punch from 0x00512e to 0x007f10 (0x2de3 bytes) 000232 1787830675.991474 mapread 0x003af0 thru 0x0062e5 (0x27f6 bytes) 000233 1787830676.006512 mapwrite 0x00a7fd thru 0x019bd3 (0xf3d7 bytes) 000234 1787830676.045210 read 0x012795 thru 0x0199b1 (0x721d bytes) 000235 1787830676.045848 write 0x0045d8 thru 0x00f03a (0xaa63 bytes) 000236 1787830676.047487 trunc from 0x0109fa to 0x019bd3 (0x91da bytes) 000237 1787830676.052080 mapread 0x010515 thru 0x019bd3 (0x96bf bytes) 000238 1787830676.060386 mapread 0x012f8c thru 0x019bd3 (0x6c48 bytes) 000240 1787830676.062064 trunc from 0x016dee to 0x019bd3 (0x2de6 bytes) 000241 1787830676.068042 trunc from 0x019bd4 to 0x02b906 (0x11d33 bytes) 000242 1787830676.085656 mapwrite 0x01f4c3 thru 0x025983 (0x64c1 bytes) 000243 1787830676.121897 trunc from 0x02b907 to 0x02684d (0x50b9 bytes) 000244 1787830676.138822 trunc from 0x02684e to 0x01a4f0 (0xc35d bytes) 000245 1787830676.163410 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a4f1): Mirror 3: 0xac930657 (stale) Mirror 7: 0xac930657 (stale) Mirror 8: 0xac930657 (stale) Mirror 9: 0xbf13cede (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a4f1): Mirror 3: 0xac930657 (stale) Mirror 7: 0xac930657 (stale) Mirror 8: 0xac930657 (stale) Mirror 9: 0xbf13cede (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a4f1: skipped 000246 1787830676.524006 mapwrite 0x02344c thru 0x027e25 (0x49da bytes) 000247 1787830676.577878 write 0x03ad05 thru 0x03b2e2 (0x5de bytes) 000249 1787830676.579919 fallocate 0x03ec9c thru 0x03ffff (0x1364 bytes) 000250 1787830676.589113 write 0x0082d3 thru 0x017ce1 (0xfa0f bytes) 000251 1787830676.598867 punch from 0x0055c8 to 0x012c3c (0xd675 bytes) 000253 1787830676.611605 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf5fb9a73 (stale) Mirror 8: 0xf5fb9a73 (stale) Mirror 9: 0xb700607a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf5fb9a73 (stale) Mirror 8: 0xf5fb9a73 (stale) Mirror 9: 0xb700607a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000254 1787830676.875148 mapwrite 0x02085d thru 0x02892c (0x80d0 bytes) 000255 1787830676.920298 fallocate 0x0343ba thru 0x034598 (0x1df bytes) 000256 1787830676.926503 trunc from 0x035c11 to 0x03ffff (0xa3ef bytes) 000257 1787830676.930511 mapwrite 0x01bcc7 thru 0x01c850 (0xb8a bytes) 000260 1787830676.942828 write 0x013f9b thru 0x0234b5 (0xf51b bytes) 000261 1787830676.952623 fallocate 0x037e90 thru 0x03b97a (0x3aeb bytes) 000262 1787830676.957558 fallocate 0x027945 thru 0x0336c4 (0xbd80 bytes) 000264 1787830676.963188 punch from 0x02677c to 0x030eaa (0xa72f bytes) 000265 1787830676.969575 write 0x00a79f thru 0x00de8a (0x36ec bytes) 000266 1787830676.979436 punch from 0x024d9e to 0x02aabb (0x5d1e bytes) 000267 1787830676.985300 trunc from 0x0090c8 to 0x0132b7 (0xa1f0 bytes) 000268 1787830677.000487 write 0x00bc7f thru 0x014084 (0x8406 bytes) 000269 1787830677.008467 write 0x007ca3 thru 0x00f649 (0x79a7 bytes) 000270 1787830677.022442 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf5fb9a73 (stale) Mirror 9: 0xb700607a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf5fb9a73 (stale) Mirror 9: 0xb700607a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000272 1787830677.359326 mapwrite 0x00ee7f thru 0x01b0e1 (0xc263 bytes) 000273 1787830677.409065 punch from 0x01cbaf to 0x0202b3 (0x3705 bytes) 000275 1787830677.414310 punch from 0x005d91 to 0x005f79 (0x1e9 bytes) 000276 1787830677.417637 fallocate 0x00ffca thru 0x01a53c (0xa573 bytes) 000277 1787830677.422202 fallocate 0x00fb16 thru 0x016a8c (0x6f77 bytes) 000278 1787830677.427760 read 0x0073d2 thru 0x00e80b (0x743a bytes) 000279 1787830677.437101 trunc from 0x040000 to 0x02138b (0x1ec74 bytes) 000280 1787830677.462984 mapwrite 0x01b94c thru 0x026fc8 (0xb67d bytes) 000281 1787830677.528479 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26fc9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26fc9): Mirror 8: 0xa426e070 (stale) Mirror 9: 0xd047790b (stale) CRC-32 checksum value for chunk [0, 0x26fc9): Mirror 1: 0x9833a255 Mirror 10: 0x9833a255 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26fc9: skipped 000283 1787830678.196483 fallocate 0x030e5e thru 0x030fe5 (0x188 bytes) 000284 1787830678.232594 mapwrite 0x0184c1 thru 0x020000 (0x7b40 bytes) 000286 1787830678.265179 read 0x00e69f thru 0x018b69 (0xa4cb bytes) 000287 1787830678.270830 trunc from 0x026fc9 to 0x035a08 (0xea40 bytes) 000288 1787830678.318199 read 0x00213e thru 0x006bd2 (0x4a95 bytes) 000289 1787830678.326261 trunc from 0x02c386 to 0x02df5c (0x1bd7 bytes) 000292 1787830678.336468 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35a09): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a09): Mirror 8: 0x633cfb4d (stale) Mirror 9: 0xacd2a412 (stale) Mirror 10: 0x56f8efa6 (stale) CRC-32 checksum value for chunk [0, 0x35a09): Mirror 1: 0x8e3773a9 Mirror 11: 0x8e3773a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a09: skipped 000293 1787830678.933863 mapwrite 0x01bce5 thru 0x02a86e (0xeb8a bytes) 000294 1787830679.019724 read 0x032bad thru 0x035a08 (0x2e5c bytes) 000295 1787830679.026154 trunc from 0x029058 to 0x0341c9 (0xb172 bytes) 000296 1787830679.031070 read 0x013a92 thru 0x01fd87 (0xc2f6 bytes) 000297 1787830679.041130 write 0x025993 thru 0x026272 (0x8e0 bytes) 000298 1787830679.047200 trunc from 0x035a09 to 0x01ca6a (0x18f9e bytes) 000299 1787830679.081250 punch from 0x011e11 to 0x012b1b (0xd0b bytes) 000300 1787830679.086238 punch from 0x00b3b0 to 0x0126c7 (0x7318 bytes) 000301 1787830679.095203 trunc from 0x01ca6b to 0x0014e6 (0x1b584 bytes) 000302 1787830679.127571 mapread 0x000c1b thru 0x0014e6 (0x8cc bytes) 000303 1787830679.128737 read 0x0012d2 thru 0x0014e6 (0x215 bytes) 000304 1787830679.129577 fallocate 0x000630 thru 0x00bab8 (0xb489 bytes) 000305 1787830679.133744 trunc from 0x0014e7 to 0x00e690 (0xd1aa bytes) 000306 1787830679.147255 mapwrite 0x031e00 thru 0x036ded (0x4fee bytes) 000307 1787830679.182292 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36dee): Mirror 8: 0x6c66cddd (stale) Mirror 9: 0x5a8bcb86 (stale) Mirror 10: 0x59bd57d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dee): Mirror 8: 0x6c66cddd (stale) Mirror 9: 0x5a8bcb86 (stale) Mirror 10: 0x59bd57d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dee: skipped 000308 1787830679.437790 write 0x03476d thru 0x036d3d (0x25d1 bytes) 000309 1787830679.448801 mapread 0x002870 thru 0x00f2c9 (0xca5a bytes) 000310 1787830679.461531 trunc from 0x03597e to 0x036ded (0x1470 bytes) 000312 1787830679.468574 mapread 0x028271 thru 0x02867e (0x40e bytes) 000313 1787830679.477954 punch from 0x01f170 to 0x02431d (0x51ae bytes) 000315 1787830679.485875 mapwrite 0x035301 thru 0x03c54d (0x724d bytes) 000316 1787830679.527823 MIRROR_RESYNC 4 mirrors 000317 1787830680.015224 mapread 0x00abf4 thru 0x00d67a (0x2a87 bytes) 000318 1787830680.034913 read 0x0162dd thru 0x01f203 (0x8f27 bytes) 000319 1787830680.046586 trunc from 0x030061 to 0x03a4f8 (0xa498 bytes) 000320 1787830680.071868 write 0x008af1 thru 0x00c068 (0x3578 bytes) 000321 1787830680.077971 fallocate 0x038c3e thru 0x03ffff (0x73c2 bytes) 000322 1787830680.081858 trunc from 0x040000 to 0x0082d6 (0x37d29 bytes) 000323 1787830680.117830 fallocate 0x0072c9 thru 0x012b0a (0xb842 bytes) 000324 1787830680.121813 write 0x02da46 thru 0x038d37 (0xb2f2 bytes) 000325 1787830680.123607 mapread 0x032ed1 thru 0x038d37 (0x5e67 bytes) 000326 1787830680.124811 mapwrite 0x022ac5 thru 0x02f271 (0xc7ad bytes) 000327 1787830680.150209 trunc from 0x02dba3 to 0x0351d0 (0x762e bytes) 000328 1787830680.154838 mapread 0x001156 thru 0x00694d (0x57f8 bytes) 000331 1787830680.165486 read 0x0197d5 thru 0x028daf (0xf5db bytes) 000332 1787830680.172591 trunc from 0x038d38 to 0x03f98a (0x6c53 bytes) 000333 1787830680.181367 read 0x0105c6 thru 0x01e80f (0xe24a bytes) 000334 1787830680.181876 fallocate 0x003e23 thru 0x006b8b (0x2d69 bytes) 000335 1787830680.189682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0xe2a81a86 (stale) Mirror 9: 0xe2a81a86 (stale) Mirror 10: 0xe2a81a86 (stale) CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 1: 0xec8b6f5c Mirror 11: 0xec8b6f5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f98b: skipped 000338 1787830680.826393 trunc from 0x00b723 to 0x010f9a (0x5878 bytes) 000339 1787830680.858720 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0xe2a81a86 (stale) Mirror 9: 0xe2a81a86 (stale) Mirror 11: 0xec8b6f5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0xe2a81a86 (stale) Mirror 9: 0xe2a81a86 (stale) Mirror 11: 0xec8b6f5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f98b: skipped 000340 1787830681.146882 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0xe2a81a86 (stale) Mirror 9: 0xe2a81a86 (stale) Mirror 11: 0xec8b6f5c (stale) CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 1: 0xec8b6f5c Mirror 12: 0xec8b6f5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f98b: skipped 000341 1787830681.959969 trunc from 0x03c149 to 0x03f98a (0x3842 bytes) 000342 1787830682.014471 read 0x00aef8 thru 0x01425f (0x9368 bytes) 000343 1787830682.033692 mapwrite 0x037214 thru 0x038a5c (0x1849 bytes) 000344 1787830682.064458 punch from 0x01b841 to 0x028aef (0xd2af bytes) 000345 1787830682.069763 mapwrite 0x02539e thru 0x025f62 (0xbc5 bytes) 000346 1787830682.097176 read 0x023a2f thru 0x02ee3c (0xb40e bytes) 000347 1787830682.109649 mapread 0x00213f thru 0x00eec9 (0xcd8b bytes) 000349 1787830682.114634 MIRROR_RESYNC 5 mirrors 000350 1787830683.382429 read 0x035a95 thru 0x039d83 (0x42ef bytes) 000351 1787830683.387328 punch from 0x009bf8 to 0x013a0f (0x9e18 bytes) 000352 1787830683.416415 write 0x00f6a4 thru 0x01f257 (0xfbb4 bytes) 000353 1787830683.423269 fallocate 0x01df5f thru 0x01e444 (0x4e6 bytes) 000354 1787830683.427062 mapread 0x03425b thru 0x03f98a (0xb730 bytes) 000355 1787830683.442381 punch from 0x0171c4 to 0x01b591 (0x43ce bytes) 000356 1787830683.452078 mapread 0x018105 thru 0x0181e2 (0xde bytes) 000357 1787830683.466509 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0x2f17b6f1 (stale) Mirror 9: 0x2f17b6f1 (stale) Mirror 11: 0x2f17b6f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f98b): Mirror 8: 0x2f17b6f1 (stale) Mirror 9: 0x2f17b6f1 (stale) Mirror 11: 0x2f17b6f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f98b: skipped 000358 1787830683.695301 MIRROR_RESYNC 4 mirrors 000359 1787830684.569910 mapread 0x007a47 thru 0x012ec9 (0xb483 bytes) 000360 1787830684.598578 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000361 1787830684.809582 MIRROR_RESYNC 3 mirrors 000362 1787830685.022814 trunc from 0x03f98b to 0x02c835 (0x13155 bytes) 000364 1787830685.060201 fallocate 0x03a76c thru 0x03ffff (0x5894 bytes) 000365 1787830685.064567 read 0x020da6 thru 0x024966 (0x3bc1 bytes) 000366 1787830685.070640 punch from 0x003ae3 to 0x012226 (0xe744 bytes) 000367 1787830685.074382 write 0x03f0e0 thru 0x03ffff (0xf20 bytes) 000368 1787830685.075893 punch from 0x00c9aa to 0x00eeda (0x2531 bytes) 000369 1787830685.080559 read 0x03d209 thru 0x03fd46 (0x2b3e bytes) 000370 1787830685.091890 trunc from 0x040000 to 0x02077d (0x1f882 bytes) 000371 1787830685.110213 trunc from 0x0146f1 to 0x02077d (0xc08d bytes) 000372 1787830685.114821 write 0x037465 thru 0x03e079 (0x6c15 bytes) 000373 1787830685.116649 trunc from 0x03e07a to 0x0012c8 (0x3cdb1 bytes) 000374 1787830685.129103 mapread 0x000054 thru 0x0012c8 (0x1275 bytes) 000375 1787830685.131908 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12c9): Mirror 9: 0xf6034da3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c9): Mirror 9: 0xf6034da3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c9: skipped 000376 1787830685.282176 trunc from 0x0012c9 to 0x02cb87 (0x2b8bf bytes) 000377 1787830685.292871 mapread 0x006dc9 thru 0x012fa5 (0xc1dd bytes) 000379 1787830685.307205 fallocate 0x03d900 thru 0x03ffff (0x2700 bytes) 000380 1787830685.312166 punch from 0x032f8a to 0x03e1af (0xb226 bytes) 000381 1787830685.316155 write 0x02f1ab thru 0x03694b (0x77a1 bytes) 000382 1787830685.322105 mapwrite 0x03c684 thru 0x03c938 (0x2b5 bytes) 000383 1787830685.342769 trunc from 0x02792b to 0x02f321 (0x79f7 bytes) 000384 1787830685.347529 mapread 0x01d1b6 thru 0x02ae9c (0xdce7 bytes) 000385 1787830685.362853 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x839f5975 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x368d1497 Mirror 10: 0x368d1497 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1787830685.909684 trunc from 0x040000 to 0x00ae9a (0x35165 bytes) 000387 1787830685.974430 write 0x03fbe2 thru 0x03ffff (0x41e bytes) 000388 1787830685.977381 fallocate 0x0182ec thru 0x01fac6 (0x77db bytes) 000389 1787830685.981985 trunc from 0x00c052 to 0x014921 (0x88d0 bytes) 000390 1787830685.986235 fallocate 0x032a68 thru 0x0373b3 (0x494c bytes) 000392 1787830685.991483 trunc from 0x03094b to 0x0371cd (0x6883 bytes) 000393 1787830685.996214 punch from 0x02df76 to 0x03113d (0x31c8 bytes) 000394 1787830686.003223 fallocate 0x00c1ad thru 0x018bbf (0xca13 bytes) 000396 1787830686.010725 fallocate 0x01c3de thru 0x024f8b (0x8bae bytes) 000397 1787830686.017816 punch from 0x016a9b to 0x01f5fe (0x8b64 bytes) 000398 1787830686.030818 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x839f5975 (stale) Mirror 10: 0x368d1497 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x114a13a9 Mirror 11: 0x114a13a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000400 1787830686.549235 MIRROR_RESYNC 4 mirrors 000401 1787830687.017640 mapwrite 0x0263b7 thru 0x029b7e (0x37c8 bytes) 000402 1787830687.088439 mapread 0x03929a thru 0x03ffff (0x6d66 bytes) 000403 1787830687.097648 fallocate 0x038147 thru 0x038332 (0x1ec bytes) 000404 1787830687.102182 fallocate 0x00eb9e thru 0x0131fa (0x465d bytes) 000405 1787830687.106423 fallocate 0x00344b thru 0x008000 (0x4bb6 bytes) 000406 1787830687.111049 punch from 0x01e8c1 to 0x02c560 (0xdca0 bytes) 000407 1787830687.114727 mapwrite 0x00f7a5 thru 0x01053e (0xd9a bytes) 000408 1787830687.137183 MIRROR_RESYNC 4 mirrors 000409 1787830687.736413 fallocate 0x02a283 thru 0x02d10e (0x2e8c bytes) 000410 1787830687.758120 punch from 0x024680 to 0x025107 (0xa88 bytes) 000411 1787830687.762261 write 0x0274bd thru 0x0317f7 (0xa33b bytes) 000412 1787830687.769502 mapread 0x01de11 thru 0x02714a (0x933a bytes) 000413 1787830687.781197 trunc from 0x0023b7 to 0x005ddb (0x3a25 bytes) 000414 1787830687.786199 read 0x009646 thru 0x00b519 (0x1ed4 bytes) 000415 1787830687.790752 trunc from 0x037125 to 0x03e439 (0x7315 bytes) 000416 1787830687.796806 punch from 0x03243c to 0x036f14 (0x4ad9 bytes) 000417 1787830687.804196 trunc from 0x01d31f to 0x021b1d (0x47ff bytes) 000418 1787830687.808020 read 0x02b928 thru 0x031163 (0x583c bytes) 000419 1787830687.808407 read 0x0247ab thru 0x02ccd0 (0x8526 bytes) 000420 1787830687.808880 trunc from 0x040000 to 0x0122d0 (0x2dd2f bytes) 000421 1787830687.837132 punch from 0x005a65 to 0x010641 (0xabdd bytes) 000422 1787830687.842191 read 0x0071e5 thru 0x0122d0 (0xb0ec bytes) 000424 1787830687.852386 mapwrite 0x005a62 thru 0x01245d (0xc9fc bytes) 000425 1787830687.887405 mapread 0x0082cd thru 0x01245d (0xa191 bytes) 000426 1787830687.888255 trunc from 0x0096fe to 0x01245d (0x8d60 bytes) 000427 1787830687.891599 read 0x00ed81 thru 0x00effc (0x27c bytes) 000428 1787830687.896607 read 0x00f661 thru 0x01245d (0x2dfd bytes) 000429 1787830687.897182 read 0x007f4c thru 0x00a275 (0x232a bytes) 000430 1787830687.897677 mapwrite 0x01b01e thru 0x0292c6 (0xe2a9 bytes) 000431 1787830687.929074 read 0x008c7d thru 0x010d02 (0x8086 bytes) 000432 1787830687.930330 fallocate 0x019ef1 thru 0x01f2af (0x53bf bytes) 000433 1787830687.935724 punch from 0x011892 to 0x018d82 (0x74f1 bytes) 000434 1787830687.941221 mapwrite 0x017dbe thru 0x02038d (0x85d0 bytes) 000435 1787830687.960461 write 0x007bb3 thru 0x00f124 (0x7572 bytes) 000436 1787830687.961558 write 0x03a6f2 thru 0x03e635 (0x3f44 bytes) 000437 1787830687.962548 trunc from 0x004555 to 0x00f1f2 (0xac9e bytes) 000438 1787830687.976122 fallocate 0x0106e9 thru 0x018554 (0x7e6c bytes) 000439 1787830687.986212 trunc from 0x01259d to 0x020608 (0xe06c bytes) 000440 1787830687.989624 trunc from 0x03e636 to 0x029630 (0x15005 bytes) 000441 1787830687.997826 trunc from 0x01618f to 0x025b30 (0xf9a2 bytes) 000442 1787830688.001926 mapwrite 0x034395 thru 0x0368bf (0x252b bytes) 000443 1787830688.033805 write 0x01d5a9 thru 0x02509e (0x7af6 bytes) 000444 1787830688.049108 write 0x02381f thru 0x025ea5 (0x2687 bytes) 000446 1787830688.051857 trunc from 0x0368c0 to 0x00c6d1 (0x2a1ee bytes) 000447 1787830688.074393 fallocate 0x026a11 thru 0x02b732 (0x4d22 bytes) 000448 1787830688.082444 trunc from 0x00b77d to 0x01ae56 (0xf6da bytes) 000449 1787830688.088054 trunc from 0x0178d8 to 0x01901d (0x1746 bytes) 000450 1787830688.090887 read 0x01e12d thru 0x028092 (0x9f66 bytes) 000451 1787830688.099882 read 0x00310f thru 0x012a3a (0xf92c bytes) 000452 1787830688.101206 punch from 0x017a94 to 0x018fd7 (0x1544 bytes) 000454 1787830688.107017 punch from 0x027207 to 0x02b732 (0x452c bytes) 000456 1787830688.112094 mapread 0x0246fd thru 0x02b732 (0x7036 bytes) 000457 1787830688.123781 trunc from 0x02b733 to 0x03c763 (0x11031 bytes) 000458 1787830688.136137 mapwrite 0x00e394 thru 0x018fdb (0xac48 bytes) 000459 1787830688.153153 read 0x01a93a thru 0x01cf9e (0x2665 bytes) 000460 1787830688.153786 mapread 0x00c0ac thru 0x012a42 (0x6997 bytes) 000461 1787830688.155227 fallocate 0x0110bf thru 0x015962 (0x48a4 bytes) 000462 1787830688.159887 fallocate 0x02ec0a thru 0x032093 (0x348a bytes) 000463 1787830688.164366 punch from 0x01ec72 to 0x0217f3 (0x2b82 bytes) 000464 1787830688.168156 trunc from 0x03c764 to 0x038248 (0x451b bytes) 000465 1787830688.178060 mapwrite 0x038fa7 thru 0x03d304 (0x435e bytes) 000466 1787830688.216845 fallocate 0x023330 thru 0x02b0f3 (0x7dc4 bytes) 000467 1787830688.220336 punch from 0x02faae to 0x032662 (0x2bb5 bytes) 000468 1787830688.229021 write 0x032f97 thru 0x0368b4 (0x391e bytes) 000469 1787830688.236652 write 0x018a9f thru 0x023e07 (0xb369 bytes) 000470 1787830688.244414 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d305): Mirror 9: 0x2f4bad8a (stale) Mirror 10: 0x2f4bad8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d305): Mirror 9: 0x2f4bad8a (stale) Mirror 10: 0x2f4bad8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d305: skipped 000471 1787830688.467064 read 0x03abcf thru 0x03d304 (0x2736 bytes) 000472 1787830688.475127 fallocate 0x03dc66 thru 0x03ffff (0x239a bytes) 000474 1787830688.485154 trunc from 0x01d490 to 0x01fcac (0x281d bytes) 000475 1787830688.493654 mapwrite 0x039f69 thru 0x03ffff (0x6097 bytes) 000476 1787830688.541898 mapwrite 0x0057b3 thru 0x013fe1 (0xe82f bytes) 000477 1787830688.573783 trunc from 0x040000 to 0x02f63a (0x109c5 bytes) 000478 1787830688.589602 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f63b): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f63b): Mirror 9: 0x35352efd (stale) Mirror 10: 0x35352efd (stale) CRC-32 checksum value for chunk [0, 0x2f63b): Mirror 1: 0x8dc9cac9 Mirror 11: 0x8dc9cac9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f63b: skipped 000479 1787830689.108690 fallocate 0x03c044 thru 0x03fa32 (0x39ef bytes) 000480 1787830689.141170 mapwrite 0x00cb79 thru 0x014b12 (0x7f9a bytes) 000481 1787830689.181246 punch from 0x037b32 to 0x03b317 (0x37e6 bytes) 000482 1787830689.187145 read 0x03c395 thru 0x03fa32 (0x369e bytes) 000483 1787830689.195543 trunc from 0x03fa33 to 0x003477 (0x3c5bb bytes) 000484 1787830689.226585 fallocate 0x014afb thru 0x022b1d (0xe023 bytes) 000485 1787830689.229665 fallocate 0x0224f4 thru 0x02b40e (0x8f1b bytes) 000486 1787830689.233379 trunc from 0x00b9cb to 0x019316 (0xd94c bytes) 000487 1787830689.237510 punch from 0x00b2e9 to 0x01a98e (0xf6a6 bytes) 000488 1787830689.241043 trunc from 0x02b40f to 0x0073e0 (0x2402e bytes) 000489 1787830689.249767 write 0x023066 thru 0x02306c (0x7 bytes) 000490 1787830689.250720 mapwrite 0x02b868 thru 0x03902e (0xd7c7 bytes) 000491 1787830689.288032 read 0x017c2e thru 0x026bcd (0xefa0 bytes) 000492 1787830689.297121 trunc from 0x00eb9a to 0x018644 (0x9aab bytes) 000493 1787830689.303839 MIRROR_RESYNC 4 mirrors 000495 1787830689.861416 trunc from 0x03902f to 0x02d76c (0xb8c2 bytes) 000496 1787830689.924471 mapwrite 0x01d4e4 thru 0x01d5e8 (0x105 bytes) 000497 1787830689.944619 trunc from 0x02d76d to 0x010383 (0x1d3e9 bytes) 000498 1787830689.958077 fallocate 0x0019d6 thru 0x008599 (0x6bc4 bytes) 000500 1787830689.961320 read 0x0069f1 thru 0x010383 (0x9993 bytes) 000501 1787830689.967217 trunc from 0x010384 to 0x032d9e (0x22a1b bytes) 000502 1787830689.979244 fallocate 0x029b13 thru 0x034a43 (0xaf31 bytes) 000503 1787830689.984804 read 0x034881 thru 0x034a43 (0x1c3 bytes) 000504 1787830689.992626 punch from 0x02f74d to 0x034a43 (0x52f7 bytes) 000505 1787830689.999577 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34a44): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a44): Mirror 9: 0x35c092cf (stale) Mirror 10: 0x35c092cf (stale) Mirror 11: 0x35c092cf (stale) CRC-32 checksum value for chunk [0, 0x34a44): Mirror 1: 0x1b2402db Mirror 12: 0x1b2402db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a44: skipped 000506 1787830690.455509 trunc from 0x002914 to 0x0057ae (0x2e9b bytes) 000507 1787830690.482764 trunc from 0x034a44 to 0x019100 (0x1b943 bytes) 000508 1787830690.496216 write 0x0392ab thru 0x03ffff (0x6d55 bytes) 000509 1787830690.497309 punch from 0x00f1ed to 0x00fd4c (0xb60 bytes) 000510 1787830690.501195 fallocate 0x007674 thru 0x00c11c (0x4aa9 bytes) 000511 1787830690.503876 mapread 0x00bfa5 thru 0x017e1d (0xbe79 bytes) 000512 1787830690.525741 trunc from 0x040000 to 0x00cfd6 (0x33029 bytes) 000514 1787830690.536877 fallocate 0x007dfb thru 0x00c662 (0x4868 bytes) 000515 1787830690.539927 mapwrite 0x014b13 thru 0x016660 (0x1b4e bytes) 000516 1787830690.581085 fallocate 0x01b6cc thru 0x01bf71 (0x8a6 bytes) 000517 1787830690.586700 fallocate 0x017640 thru 0x01fc08 (0x85c9 bytes) 000518 1787830690.591193 write 0x02478c thru 0x024a6c (0x2e1 bytes) 000519 1787830690.592564 fallocate 0x0264e4 thru 0x02bbf9 (0x5716 bytes) 000520 1787830690.596656 fallocate 0x027a74 thru 0x0353ad (0xd93a bytes) 000521 1787830690.603777 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: 0x2efb0859 (stale) Mirror 11: 0x2efb0859 (stale) Mirror 12: 0x7191929b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: 0x2efb0859 (stale) Mirror 11: 0x2efb0859 (stale) Mirror 12: 0x7191929b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353ae: skipped 000522 1787830690.846793 write 0x01a7e2 thru 0x025acd (0xb2ec bytes) 000523 1787830690.864754 mapread 0x01ef6f thru 0x025197 (0x6229 bytes) 000524 1787830690.871428 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: 0x2efb0859 (stale) Mirror 11: 0x2efb0859 (stale) Mirror 12: 0x7191929b (stale) CRC-32 checksum value for chunk [0, 0x353ae): Mirror 1: 0xb5f6c597 Mirror 13: 0xb5f6c597 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353ae: skipped 000525 1787830691.438664 write 0x010042 thru 0x01e4cd (0xe48c bytes) 000526 1787830691.465885 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353ae): Mirror 9: 0x2efb0859 (stale) Mirror 11: 0x2efb0859 (stale) Mirror 12: 0x7191929b (stale) Mirror 13: 0xb5f6c597 (stale) CRC-32 checksum value for chunk [0, 0x353ae): Mirror 1: 0xe98e49d Mirror 14: 0xe98e49d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353ae: skipped 000527 1787830692.020324 trunc from 0x0353ae to 0x02e603 (0x6daa bytes) 000528 1787830692.048179 trunc from 0x02e604 to 0x01b3e1 (0x13222 bytes) 000529 1787830692.062128 punch from 0x00c3f2 to 0x0130e3 (0x6cf2 bytes) 000530 1787830692.065160 write 0x023d41 thru 0x0243af (0x66f bytes) 000531 1787830692.068748 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x243b0): Mirror 9: 0x9e602cec (stale) Mirror 11: 0x9e602cec (stale) Mirror 12: 0xf5dc0748 (stale) Mirror 13: 0xf2c06b4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243b0): Mirror 9: 0x9e602cec (stale) Mirror 11: 0x9e602cec (stale) Mirror 12: 0xf5dc0748 (stale) Mirror 13: 0xf2c06b4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243b0: skipped 000532 1787830692.320008 trunc from 0x0243b0 to 0x01ae94 (0x951b bytes) 000533 1787830692.338874 fallocate 0x03be60 thru 0x03f28d (0x342e bytes) 000534 1787830692.346336 read 0x01c6fb thru 0x025748 (0x904e bytes) 000535 1787830692.354730 trunc from 0x03f28e to 0x010430 (0x2ee5d bytes) 000536 1787830692.366338 write 0x00fb98 thru 0x019058 (0x94c1 bytes) 000537 1787830692.374818 punch from 0x000133 to 0x00f1a6 (0xf074 bytes) 000538 1787830692.394100 fallocate 0x00c9fa thru 0x016456 (0x9a5d bytes) 000540 1787830692.400232 write 0x03faa9 thru 0x03ffff (0x557 bytes) 000541 1787830692.407268 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbeae4485 (stale) Mirror 12: 0x7b8b40b8 (stale) Mirror 13: 0x4eaaa646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbeae4485 (stale) Mirror 12: 0x7b8b40b8 (stale) Mirror 13: 0x4eaaa646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000542 1787830692.685845 write 0x00e3b0 thru 0x019a88 (0xb6d9 bytes) 000543 1787830692.696572 read 0x0009f5 thru 0x008123 (0x772f bytes) 000544 1787830692.707210 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7b8b40b8 (stale) Mirror 13: 0x4eaaa646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7b8b40b8 (stale) Mirror 13: 0x4eaaa646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1787830692.921614 mapwrite 0x03b1c1 thru 0x03ffff (0x4e3f bytes) 000546 1787830692.948900 read 0x0032dc thru 0x010d28 (0xda4d bytes) 000547 1787830692.954504 fallocate 0x034d6f thru 0x03f4a7 (0xa739 bytes) 000548 1787830692.961316 mapread 0x03b231 thru 0x03ffff (0x4dcf bytes) 000549 1787830692.962790 trunc from 0x040000 to 0x00b825 (0x347da bytes) 000550 1787830692.977896 punch from 0x00456e to 0x00b825 (0x72b8 bytes) 000551 1787830692.983398 trunc from 0x0098b5 to 0x00b825 (0x1f71 bytes) 000552 1787830692.988692 trunc from 0x007930 to 0x00b825 (0x3ef6 bytes) 000554 1787830692.999038 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb826): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb826): Mirror 12: 0x1bcee2d6 (stale) Mirror 13: 0x1bcee2d6 (stale) CRC-32 checksum value for chunk [0, 0xb826): Mirror 1: 0xc9576b95 Mirror 14: 0xc9576b95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb826: skipped 000555 1787830693.539058 mapread 0x00601c thru 0x00b825 (0x580a bytes) 000556 1787830693.549823 mapread 0x0040d5 thru 0x00a108 (0x6034 bytes) 000557 1787830693.550417 trunc from 0x009ff0 to 0x00b825 (0x1836 bytes) 000558 1787830693.568289 read 0x00194c thru 0x00557c (0x3c31 bytes) 000559 1787830693.573279 trunc from 0x006068 to 0x00b825 (0x57be bytes) 000560 1787830693.579135 fallocate 0x009a31 thru 0x00cb0d (0x30dd bytes) 000561 1787830693.582684 fallocate 0x02a83b thru 0x02b23f (0xa05 bytes) 000564 1787830693.588016 mapwrite 0x003d11 thru 0x00a45d (0x674d bytes) 000565 1787830693.606129 punch from 0x0084d3 to 0x00889e (0x3cc bytes) 000566 1787830693.609448 fallocate 0x004273 thru 0x00efa7 (0xad35 bytes) 000567 1787830693.613802 mapwrite 0x0157fe thru 0x01af83 (0x5786 bytes) 000568 1787830693.654901 write 0x00cb83 thru 0x01a98d (0xde0b bytes) 000569 1787830693.657064 write 0x03347a thru 0x03ae5e (0x79e5 bytes) 000570 1787830693.658925 fallocate 0x0349b1 thru 0x0361e2 (0x1832 bytes) 000571 1787830693.666138 punch from 0x01b2e0 to 0x028976 (0xd697 bytes) 000572 1787830693.670548 fallocate 0x00e0ba thru 0x018138 (0xa07f bytes) 000573 1787830693.674479 trunc from 0x03ae5f to 0x03bae3 (0xc85 bytes) 000574 1787830693.686303 read 0x00a4fa thru 0x019e27 (0xf92e bytes) 000575 1787830693.687151 read 0x02fd01 thru 0x03bae3 (0xbde3 bytes) 000576 1787830693.694888 trunc from 0x0090ea to 0x00e8b0 (0x57c7 bytes) 000577 1787830693.708789 write 0x00fb8a thru 0x012530 (0x29a7 bytes) 000578 1787830693.710237 trunc from 0x033143 to 0x03bae3 (0x89a1 bytes) 000579 1787830693.715009 mapread 0x00584a thru 0x008b12 (0x32c9 bytes) 000580 1787830693.718481 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bae4): Mirror 12: 0x196eadde (stale) Mirror 14: 0x92a2d4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bae4): Mirror 12: 0x196eadde (stale) Mirror 14: 0x92a2d4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bae4: skipped 000583 1787830693.948675 write 0x020277 thru 0x026940 (0x66ca bytes) 000584 1787830693.961690 mapwrite 0x012c72 thru 0x015ed1 (0x3260 bytes) 000585 1787830693.999182 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bae4): Mirror 14: 0x92a2d4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bae4): Mirror 14: 0x92a2d4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bae4: skipped 000586 1787830694.193762 fallocate 0x005b3e thru 0x00e762 (0x8c25 bytes) 000587 1787830694.197123 trunc from 0x02d1da to 0x03bae3 (0xe90a bytes) 000588 1787830694.203789 MIRROR_RESYNC 2 mirrors 000589 1787830694.690083 mapread 0x02a3d6 thru 0x036a7e (0xc6a9 bytes) 000590 1787830694.710157 mapwrite 0x019577 thru 0x025178 (0xbc02 bytes) 000591 1787830694.787288 MIRROR_RESYNC 2 mirrors 000592 1787830695.231487 trunc from 0x03bae4 to 0x02fc45 (0xbe9e bytes) 000593 1787830695.258722 mapread 0x013665 thru 0x01e45e (0xadfa bytes) 000594 1787830695.278590 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000595 1787830695.394927 punch from 0x006e29 to 0x00d921 (0x6af9 bytes) 000597 1787830695.402376 fallocate 0x004564 thru 0x0099e1 (0x547e bytes) 000598 1787830695.407532 read 0x02b394 thru 0x02fc45 (0x48b2 bytes) 000599 1787830695.413292 mapwrite 0x028f01 thru 0x02ded5 (0x4fd5 bytes) 000600 1787830695.443393 fallocate 0x006cc2 thru 0x007d34 (0x1073 bytes) 000601 1787830695.447471 punch from 0x02c58d to 0x02fc45 (0x36b9 bytes) 000603 1787830695.453445 mapwrite 0x0145f3 thru 0x020be8 (0xc5f6 bytes) 000604 1787830695.482343 read 0x011132 thru 0x01e843 (0xd712 bytes) 000606 1787830695.483655 trunc from 0x02fc46 to 0x00a0fe (0x25b47 bytes) 000607 1787830695.498623 read 0x000de7 thru 0x002448 (0x1662 bytes) 000608 1787830695.504960 mapread 0x0058dc thru 0x00a0fe (0x4823 bytes) 000609 1787830695.505855 punch from 0x007fa9 to 0x007fce (0x26 bytes) 000610 1787830695.509602 mapwrite 0x026a71 thru 0x034fe1 (0xe571 bytes) 000612 1787830695.546148 read 0x02f0bf thru 0x034fe1 (0x5f23 bytes) 000613 1787830695.546784 fallocate 0x0123b4 thru 0x018d5a (0x69a7 bytes) 000614 1787830695.550187 mapread 0x004634 thru 0x011190 (0xcb5d bytes) 000616 1787830695.554506 read 0x023f51 thru 0x02c6b5 (0x8765 bytes) 000617 1787830695.556495 trunc from 0x034fe2 to 0x007416 (0x2dbcb bytes) 000618 1787830695.566911 trunc from 0x007417 to 0x0271a0 (0x1fd8a bytes) 000619 1787830695.585015 mapread 0x009fa5 thru 0x01883e (0xe89a bytes) 000620 1787830695.592367 mapwrite 0x0136f1 thru 0x01a7ac (0x70bc bytes) 000621 1787830695.606799 punch from 0x00c1ba to 0x013fc9 (0x7e10 bytes) 000622 1787830695.610672 read 0x010554 thru 0x01f345 (0xedf2 bytes) 000623 1787830695.619073 read 0x00f327 thru 0x014bd2 (0x58ac bytes) 000624 1787830695.620320 mapwrite 0x027e13 thru 0x0357bd (0xd9ab bytes) 000625 1787830695.669422 trunc from 0x0357be to 0x039e61 (0x46a4 bytes) 000626 1787830695.680335 write 0x011327 thru 0x0186db (0x73b5 bytes) 000628 1787830695.687622 MIRROR_EXTEND to 2 mirrors 000629 1787830696.146322 mapwrite 0x00d78a thru 0x00f393 (0x1c0a bytes) 000630 1787830696.210052 read 0x02ebd8 thru 0x035278 (0x66a1 bytes) 000631 1787830696.216400 read 0x02af4d thru 0x0358f5 (0xa9a9 bytes) 000632 1787830696.222688 trunc from 0x039e62 to 0x024e9a (0x14fc7 bytes) 000633 1787830696.253027 punch from 0x01528a to 0x01ee0d (0x9b84 bytes) 000634 1787830696.256041 punch from 0x006419 to 0x009246 (0x2e2e bytes) 000635 1787830696.258743 trunc from 0x024e9b to 0x02ccef (0x7e55 bytes) 000636 1787830696.272465 trunc from 0x01312c to 0x01d915 (0xa7ea bytes) 000637 1787830696.275869 trunc from 0x0158cf to 0x021fb7 (0xc6e9 bytes) 000638 1787830696.279539 trunc from 0x01830d to 0x023464 (0xb158 bytes) 000639 1787830696.283662 mapread 0x021b18 thru 0x029eb5 (0x839e bytes) 000640 1787830696.317724 trunc from 0x00b0f6 to 0x015010 (0x9f1b bytes) 000641 1787830696.323389 read 0x00ffba thru 0x01e8b4 (0xe8fb bytes) 000642 1787830696.331256 mapwrite 0x021024 thru 0x02db98 (0xcb75 bytes) 000643 1787830696.365381 mapwrite 0x00de8b thru 0x01c434 (0xe5aa bytes) 000644 1787830696.380443 fallocate 0x012e28 thru 0x015b4c (0x2d25 bytes) 000646 1787830696.386013 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2db99): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db99): Mirror 2: 0x22782cbd (stale) CRC-32 checksum value for chunk [0, 0x2db99): Mirror 1: 0x18d6cee4 Mirror 3: 0x18d6cee4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db99: skipped 000647 1787830696.835769 trunc from 0x02db99 to 0x02d156 (0xa42 bytes) 000648 1787830696.872103 mapread 0x029d3f thru 0x02b924 (0x1be6 bytes) 000649 1787830696.883095 read 0x005bfe thru 0x010578 (0xa97b bytes) 000650 1787830696.887911 trunc from 0x02d157 to 0x0302b3 (0x315d bytes) 000651 1787830696.900104 punch from 0x02161d to 0x02f816 (0xe1fa bytes) 000652 1787830696.907109 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302b4): Mirror 3: 0x579ecf4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302b4): Mirror 3: 0x579ecf4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302b4: skipped 000653 1787830697.073576 fallocate 0x03194c thru 0x036439 (0x4aee bytes) 000654 1787830697.082074 punch from 0x0257ea to 0x02a1f6 (0x4a0d bytes) 000655 1787830697.085593 read 0x008096 thru 0x00d9a0 (0x590b bytes) 000656 1787830697.096856 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3643a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3643a): Mirror 3: 0xdfc3416 (stale) CRC-32 checksum value for chunk [0, 0x3643a): Mirror 1: 0xcc595e40 Mirror 4: 0xcc595e40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3643a: skipped 000658 1787830697.586804 fallocate 0x02b241 thru 0x02f859 (0x4619 bytes) 000659 1787830697.605213 trunc from 0x03643a to 0x012544 (0x23ef5 bytes) 000660 1787830697.628523 MIRROR_RESYNC 3 mirrors 000661 1787830698.022233 punch from 0x0005aa to 0x000f90 (0x9e7 bytes) 000662 1787830698.056272 MIRROR_RESYNC 3 mirrors 000663 1787830698.477733 mapread 0x000d97 thru 0x003201 (0x246b bytes) 000664 1787830698.493105 mapread 0x003a4b thru 0x012544 (0xeafa bytes) 000665 1787830698.497750 trunc from 0x002ccc to 0x012544 (0xf879 bytes) 000666 1787830698.518184 trunc from 0x012545 to 0x038fa2 (0x26a5e bytes) 000667 1787830698.534029 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: 0xe429d311 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: 0xe429d311 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fa3: skipped 000668 1787830698.737427 trunc from 0x019421 to 0x01fec3 (0x6aa3 bytes) 000669 1787830698.748085 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: 0xe429d311 (stale) CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 1: 0xa620b943 Mirror 5: 0xa620b943 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fa3: skipped 000670 1787830699.254493 write 0x016ff1 thru 0x02526a (0xe27a bytes) 000671 1787830699.283903 mapread 0x004409 thru 0x007bc8 (0x37c0 bytes) 000672 1787830699.299574 write 0x02b045 thru 0x03428e (0x924a bytes) 000673 1787830699.309289 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 4: 0xe429d311 (stale) Mirror 5: 0xa620b943 (stale) CRC-32 checksum value for chunk [0, 0x38fa3): Mirror 1: 0xaf3e8ae0 Mirror 6: 0xaf3e8ae0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fa3: skipped 000674 1787830699.796474 trunc from 0x038fa3 to 0x019b5c (0x1f446 bytes) 000675 1787830699.834908 trunc from 0x019b5d to 0x00b962 (0xe1fa bytes) 000676 1787830699.853799 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb963): Mirror 4: 0xda038802 (stale) Mirror 6: 0x1eed7647 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb963): Mirror 4: 0xda038802 (stale) Mirror 6: 0x1eed7647 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb963: skipped 000677 1787830700.051878 fallocate 0x0064b1 thru 0x01138c (0xaedc bytes) 000678 1787830700.059726 mapread 0x006c80 thru 0x00b479 (0x47fa bytes) 000679 1787830700.078563 read 0x008064 thru 0x00b962 (0x38ff bytes) 000680 1787830700.079309 fallocate 0x02a478 thru 0x03567c (0xb205 bytes) 000681 1787830700.088464 read 0x000e4b thru 0x006db5 (0x5f6b bytes) 000682 1787830700.098310 MIRROR_RESYNC 3 mirrors 000683 1787830700.441550 write 0x03a71b thru 0x03ffff (0x58e5 bytes) 000684 1787830700.456842 punch from 0x0338a0 to 0x0367ef (0x2f50 bytes) 000685 1787830700.460349 mapwrite 0x002a48 thru 0x006793 (0x3d4c bytes) 000686 1787830700.478792 fallocate 0x03f700 thru 0x03ffff (0x900 bytes) 000687 1787830700.482719 fallocate 0x030285 thru 0x03c24a (0xbfc6 bytes) 000689 1787830700.490306 mapread 0x005be6 thru 0x012b49 (0xcf64 bytes) 000690 1787830700.496993 fallocate 0x007737 thru 0x016ace (0xf398 bytes) 000691 1787830700.499659 trunc from 0x040000 to 0x002339 (0x3dcc6 bytes) 000692 1787830700.512574 trunc from 0x00233a to 0x0239ce (0x21695 bytes) 000693 1787830700.521731 trunc from 0x0239cf to 0x01895c (0xb072 bytes) 000694 1787830700.529352 punch from 0x0170fb to 0x01895c (0x1862 bytes) 000695 1787830700.532073 write 0x02ba2f thru 0x0318d2 (0x5ea4 bytes) 000696 1787830700.533203 write 0x002421 thru 0x009c2f (0x780f bytes) 000697 1787830700.536558 write 0x027853 thru 0x027986 (0x134 bytes) 000698 1787830700.539911 fallocate 0x022950 thru 0x029d5f (0x7410 bytes) 000699 1787830700.543205 mapwrite 0x02996d thru 0x02b1dc (0x1870 bytes) 000700 1787830700.560524 punch from 0x01502b to 0x0249c4 (0xf99a bytes) 000701 1787830700.567320 punch from 0x0047b0 to 0x00785e (0x30af bytes) 000702 1787830700.570371 mapwrite 0x005f0e thru 0x013372 (0xd465 bytes) 000703 1787830700.595815 mapwrite 0x01a0a7 thru 0x0214ff (0x7459 bytes) 000704 1787830700.615419 punch from 0x002c2a to 0x009edb (0x72b2 bytes) 000705 1787830700.620283 punch from 0x00b2f4 to 0x017937 (0xc644 bytes) 000706 1787830700.626418 mapwrite 0x013744 thru 0x014760 (0x101d bytes) 000707 1787830700.638216 punch from 0x020cf0 to 0x02e3be (0xd6cf bytes) 000708 1787830700.640820 write 0x00bfe3 thru 0x0163d8 (0xa3f6 bytes) 000709 1787830700.643910 read 0x00ebcb thru 0x01dd60 (0xf196 bytes) 000710 1787830700.644493 punch from 0x02190a to 0x02cf97 (0xb68e bytes) 000711 1787830700.647889 read 0x00dea7 thru 0x018272 (0xa3cc bytes) 000712 1787830700.648707 fallocate 0x03c069 thru 0x03ffff (0x3f97 bytes) 000714 1787830700.651860 write 0x001bf1 thru 0x00b579 (0x9989 bytes) 000716 1787830700.653643 punch from 0x03baeb to 0x03ffff (0x4515 bytes) 000717 1787830700.657067 mapwrite 0x004564 thru 0x00f904 (0xb3a1 bytes) 000718 1787830700.667733 write 0x03a82b thru 0x03ffff (0x57d5 bytes) 000720 1787830700.675797 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000721 1787830700.814750 mapwrite 0x031046 thru 0x032827 (0x17e2 bytes) 000722 1787830700.827905 fallocate 0x0218d8 thru 0x02284d (0xf76 bytes) 000723 1787830700.833357 MIRROR_RESYNC 2 mirrors 000724 1787830701.149409 trunc from 0x040000 to 0x0126fd (0x2d902 bytes) 000725 1787830701.172213 punch from 0x00fd86 to 0x0126fd (0x2978 bytes) 000727 1787830701.176080 read 0x006aec thru 0x00c72a (0x5c3f bytes) 000728 1787830701.182779 punch from 0x006ee7 to 0x009962 (0x2a7c bytes) 000729 1787830701.186105 trunc from 0x0126fe to 0x02e0fd (0x1ba00 bytes) 000730 1787830701.197095 trunc from 0x02e0fe to 0x02e6cf (0x5d2 bytes) 000731 1787830701.204305 trunc from 0x00c59d to 0x01841d (0xbe81 bytes) 000732 1787830701.207061 mapwrite 0x016ab8 thru 0x01e9cf (0x7f18 bytes) 000733 1787830701.229382 fallocate 0x001043 thru 0x00c6f7 (0xb6b5 bytes) 000734 1787830701.236638 MIRROR_RESYNC 2 mirrors 000735 1787830701.554705 trunc from 0x02e6d0 to 0x00c6cf (0x22000 bytes) 000736 1787830701.589059 write 0x01378f thru 0x019f4a (0x67bc bytes) 000737 1787830701.590360 fallocate 0x03814a thru 0x03ffff (0x7eb6 bytes) 000738 1787830701.593869 punch from 0x03b092 to 0x03ffff (0x4f6e bytes) 000739 1787830701.597213 write 0x01e18b thru 0x0279f0 (0x9866 bytes) 000740 1787830701.610493 trunc from 0x02cf80 to 0x02d7b7 (0x838 bytes) 000742 1787830701.614774 write 0x023bd4 thru 0x025853 (0x1c80 bytes) 000743 1787830701.615630 mapread 0x02a146 thru 0x02d21b (0x30d6 bytes) 000744 1787830701.628583 mapread 0x0289fa thru 0x030d3d (0x8344 bytes) 000746 1787830701.629810 write 0x02fbd5 thru 0x039f6a (0xa396 bytes) 000747 1787830701.637343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000749 1787830701.717236 trunc from 0x040000 to 0x02011c (0x1fee3 bytes) 000750 1787830701.732473 mapread 0x010b59 thru 0x01741f (0x68c7 bytes) 000751 1787830701.747518 read 0x016fb2 thru 0x01e0ec (0x713b bytes) 000753 1787830701.753537 punch from 0x0148d6 to 0x01891b (0x4046 bytes) 000755 1787830701.758781 fallocate 0x021182 thru 0x02e8f5 (0xd774 bytes) 000756 1787830701.762567 punch from 0x0127fd to 0x01b992 (0x9196 bytes) 000758 1787830701.769561 write 0x038c08 thru 0x03ffff (0x73f8 bytes) 000759 1787830701.771611 punch from 0x03ab0a to 0x03ffff (0x54f6 bytes) 000760 1787830701.781611 fallocate 0x02e94d thru 0x036d26 (0x83da bytes) 000761 1787830701.785142 write 0x038cc9 thru 0x03e946 (0x5c7e bytes) 000763 1787830701.792521 mapread 0x00df45 thru 0x012503 (0x45bf bytes) 000764 1787830701.799805 write 0x037350 thru 0x039990 (0x2641 bytes) 000765 1787830701.807197 read 0x02573e thru 0x0259d3 (0x296 bytes) 000767 1787830701.820434 mapread 0x020221 thru 0x02cb9f (0xc97f bytes) 000768 1787830701.821380 trunc from 0x040000 to 0x03fa93 (0x56c bytes) 000769 1787830701.841157 read 0x032c38 thru 0x037a1d (0x4de6 bytes) 000770 1787830701.841737 mapwrite 0x002384 thru 0x00cb02 (0xa77f bytes) 000771 1787830701.858954 mapread 0x015b82 thru 0x01d496 (0x7915 bytes) 000774 1787830701.861031 mapwrite 0x01396d thru 0x01954e (0x5be2 bytes) 000775 1787830701.880699 trunc from 0x03fa94 to 0x0208e7 (0x1f1ac bytes) 000776 1787830701.890663 read 0x0201d4 thru 0x0208e7 (0x714 bytes) 000777 1787830701.893698 MIRROR_EXTEND to 2 mirrors 000778 1787830702.198323 trunc from 0x017182 to 0x0208e7 (0x9766 bytes) 000779 1787830702.212872 mapread 0x002cd5 thru 0x00e7b2 (0xbade bytes) 000780 1787830702.222727 fallocate 0x035cbd thru 0x03dc69 (0x7fad bytes) 000781 1787830702.225652 write 0x01e2d7 thru 0x025d35 (0x7a5f bytes) 000782 1787830702.232322 mapwrite 0x02831a thru 0x031aae (0x9795 bytes) 000783 1787830702.249876 trunc from 0x02de51 to 0x02e197 (0x347 bytes) 000784 1787830702.252864 read 0x037c0a thru 0x03bbaa (0x3fa1 bytes) 000786 1787830702.256392 read 0x02b0fe thru 0x03ab8c (0xfa8f bytes) 000787 1787830702.261650 trunc from 0x012c01 to 0x0144fe (0x18fe bytes) 000788 1787830702.267424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000789 1787830702.333026 write 0x0371e4 thru 0x03c172 (0x4f8f bytes) 000790 1787830702.341388 trunc from 0x030c38 to 0x0372dd (0x66a6 bytes) 000792 1787830702.348983 punch from 0x000ec1 to 0x001c02 (0xd42 bytes) 000793 1787830702.353379 read 0x0341dc thru 0x03dc69 (0x9a8e bytes) 000794 1787830702.358904 write 0x016248 thru 0x01f562 (0x931b bytes) 000795 1787830702.366028 trunc from 0x0186e4 to 0x01f806 (0x7123 bytes) 000796 1787830702.375897 mapwrite 0x00e630 thru 0x017d85 (0x9756 bytes) 000797 1787830702.398883 trunc from 0x011f4b to 0x01f226 (0xd2dc bytes) 000799 1787830702.402644 write 0x013543 thru 0x01c082 (0x8b40 bytes) 000800 1787830702.411389 mapread 0x027b75 thru 0x031a74 (0x9f00 bytes) 000801 1787830702.421671 mapread 0x003b86 thru 0x00bc59 (0x80d4 bytes) 000802 1787830702.429777 trunc from 0x03dc6a to 0x02ac5e (0x1300b bytes) 000803 1787830702.439521 mapwrite 0x03982e thru 0x03ffff (0x67d2 bytes) 000804 1787830702.470627 write 0x0262c1 thru 0x03564b (0xf38b bytes) 000806 1787830702.474071 mapwrite 0x03cae1 thru 0x03ffff (0x351f bytes) 000807 1787830702.484145 write 0x027756 thru 0x0310a5 (0x9950 bytes) 000808 1787830702.485318 mapwrite 0x032254 thru 0x03893d (0x66ea bytes) 000811 1787830702.501458 trunc from 0x0326a7 to 0x03b6cb (0x9025 bytes) 000812 1787830702.504548 mapwrite 0x03f9fb thru 0x03ffff (0x605 bytes) 000813 1787830702.512108 read 0x01d0da thru 0x0263e4 (0x930b bytes) 000814 1787830702.517242 trunc from 0x0176e9 to 0x0254cc (0xdde4 bytes) 000815 1787830702.520802 mapwrite 0x02f941 thru 0x03bb24 (0xc1e4 bytes) 000816 1787830702.536516 mapread 0x01046b thru 0x016104 (0x5c9a bytes) 000817 1787830702.537483 fallocate 0x02a8fd thru 0x037bdf (0xd2e3 bytes) 000818 1787830702.540751 trunc from 0x040000 to 0x027f36 (0x180c9 bytes) 000819 1787830702.551132 mapwrite 0x039663 thru 0x03ffff (0x699d bytes) 000820 1787830702.585973 mapread 0x01a134 thru 0x0263b9 (0xc286 bytes) 000821 1787830702.626918 mapread 0x0093ff thru 0x00c105 (0x2d07 bytes) 000824 1787830702.629116 write 0x035be6 thru 0x03fbf8 (0xa013 bytes) 000826 1787830702.637698 read 0x002c6e thru 0x010b1c (0xdeaf bytes) 000827 1787830702.638765 mapread 0x02606b thru 0x030955 (0xa8eb bytes) 000828 1787830702.647413 write 0x02e870 thru 0x03e0e3 (0xf874 bytes) 000829 1787830702.648986 mapwrite 0x01cd09 thru 0x0241b0 (0x74a8 bytes) 000831 1787830702.670684 trunc from 0x00b6c0 to 0x016ecf (0xb810 bytes) 000832 1787830702.675107 mapread 0x00b220 thru 0x0120cb (0x6eac bytes) 000835 1787830702.707414 trunc from 0x040000 to 0x03d220 (0x2ddf bytes) 000836 1787830702.719007 mapread 0x00ad23 thru 0x0183d3 (0xd6b1 bytes) 000837 1787830702.732903 mapwrite 0x0295dd thru 0x031588 (0x7fac bytes) 000838 1787830702.745790 trunc from 0x03d221 to 0x018fb5 (0x2426b bytes) 000840 1787830702.760068 mapwrite 0x002e1a thru 0x009766 (0x694d bytes) 000841 1787830702.774750 punch from 0x00d496 to 0x0104c9 (0x3034 bytes) 000842 1787830702.779303 trunc from 0x018fb6 to 0x02b625 (0x12670 bytes) 000843 1787830702.790875 fallocate 0x016b66 thru 0x01ed31 (0x81cc bytes) 000844 1787830702.793660 trunc from 0x02b626 to 0x038b1b (0xd4f6 bytes) 000845 1787830702.802030 mapread 0x024cc1 thru 0x02aedd (0x621d bytes) 000846 1787830702.815991 fallocate 0x033666 thru 0x0362cf (0x2c6a bytes) 000847 1787830702.818459 mapread 0x00ade5 thru 0x00bb0e (0xd2a bytes) 000848 1787830702.819052 mapwrite 0x03aa5d thru 0x03ffff (0x55a3 bytes) 000849 1787830702.852501 read 0x01e041 thru 0x02662c (0x85ec bytes) 000851 1787830702.859618 trunc from 0x003d25 to 0x009f73 (0x624f bytes) 000852 1787830702.862723 trunc from 0x033868 to 0x03ffff (0xc798 bytes) 000853 1787830702.865686 punch from 0x02b896 to 0x034da7 (0x9512 bytes) 000854 1787830702.872621 MIRROR_EXTEND to 2 mirrors 000855 1787830703.198836 read 0x006813 thru 0x012605 (0xbdf3 bytes) 000857 1787830703.204651 punch from 0x01f2f2 to 0x02d113 (0xde22 bytes) 000858 1787830703.234164 mapread 0x00aee9 thru 0x01466f (0x9787 bytes) 000859 1787830703.243617 trunc from 0x040000 to 0x024507 (0x1baf8 bytes) 000860 1787830703.261190 read 0x020ea2 thru 0x023549 (0x26a8 bytes) 000861 1787830703.268352 fallocate 0x021e5d thru 0x02a62d (0x87d1 bytes) 000863 1787830703.275994 trunc from 0x01c8cb to 0x020ef3 (0x4629 bytes) 000864 1787830703.280525 write 0x02f5cb thru 0x03d9d0 (0xe406 bytes) 000865 1787830703.282569 read 0x02bf7e thru 0x038e13 (0xce96 bytes) 000866 1787830703.289383 punch from 0x0191a7 to 0x02535f (0xc1b9 bytes) 000867 1787830703.292222 write 0x01af7d thru 0x023042 (0x80c6 bytes) 000868 1787830703.300598 punch from 0x0094a3 to 0x012681 (0x91df bytes) 000869 1787830703.307144 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000871 1787830703.394348 write 0x03e7df thru 0x03ffff (0x1821 bytes) 000872 1787830703.399876 punch from 0x0024b7 to 0x007df3 (0x593d bytes) 000873 1787830703.409226 mapread 0x007171 thru 0x0100d9 (0x8f69 bytes) 000874 1787830703.421596 punch from 0x0221d4 to 0x02f464 (0xd291 bytes) 000875 1787830703.426085 mapwrite 0x03285e thru 0x039c6f (0x7412 bytes) 000876 1787830703.451311 write 0x03e876 thru 0x03eb50 (0x2db bytes) 000877 1787830703.452190 trunc from 0x019944 to 0x01f3d8 (0x5a95 bytes) 000878 1787830703.455533 mapwrite 0x031eb7 thru 0x03456c (0x26b6 bytes) 000879 1787830703.472221 trunc from 0x01aa03 to 0x0273f9 (0xc9f7 bytes) 000881 1787830703.478474 read 0x030589 thru 0x03271c (0x2194 bytes) 000883 1787830703.481602 punch from 0x020363 to 0x02e20d (0xdeab bytes) 000885 1787830703.486696 trunc from 0x040000 to 0x01e67c (0x21983 bytes) 000887 1787830703.503698 mapwrite 0x00e33a thru 0x01be50 (0xdb17 bytes) 000888 1787830703.539179 write 0x00f944 thru 0x0123cd (0x2a8a bytes) 000890 1787830703.541231 write 0x039756 thru 0x03ffff (0x68aa bytes) 000891 1787830703.542481 trunc from 0x01ff09 to 0x02cad8 (0xcbd0 bytes) 000892 1787830703.545604 write 0x0368be thru 0x03ce12 (0x6555 bytes) 000893 1787830703.548897 fallocate 0x02819c thru 0x02b86d (0x36d2 bytes) 000894 1787830703.551767 mapread 0x011a3c thru 0x01f18a (0xd74f bytes) 000895 1787830703.557760 read 0x01d858 thru 0x01de9a (0x643 bytes) 000896 1787830703.558370 fallocate 0x031f42 thru 0x03f7b9 (0xd878 bytes) 000897 1787830703.562624 fallocate 0x0374b5 thru 0x03ffff (0x8b4b bytes) 000898 1787830703.565598 punch from 0x012284 to 0x0203d9 (0xe156 bytes) 000900 1787830703.574983 write 0x0285c7 thru 0x0351a2 (0xcbdc bytes) 000901 1787830703.580639 trunc from 0x040000 to 0x0052cb (0x3ad34 bytes) 000902 1787830703.588587 fallocate 0x0378f5 thru 0x03ffff (0x870b bytes) 000903 1787830703.596403 MIRROR_EXTEND to 2 mirrors 000904 1787830703.866717 mapread 0x0046b5 thru 0x0052cb (0xc17 bytes) 000905 1787830703.873888 trunc from 0x0003b5 to 0x0052cb (0x4f17 bytes) 000906 1787830703.899227 trunc from 0x0052a6 to 0x0052cb (0x26 bytes) 000907 1787830703.902513 mapread 0x0028ac thru 0x0052cb (0x2a20 bytes) 000908 1787830703.909328 write 0x02a7e1 thru 0x0311dc (0x69fc bytes) 000909 1787830703.914956 trunc from 0x02beeb to 0x0311dc (0x52f2 bytes) 000910 1787830703.925682 punch from 0x00fb3c to 0x01794b (0x7e10 bytes) 000911 1787830703.933355 fallocate 0x03b0dd thru 0x03e5b3 (0x34d7 bytes) 000912 1787830703.937637 write 0x02e31d thru 0x039e51 (0xbb35 bytes) 000913 1787830703.942271 mapread 0x00dc46 thru 0x01320a (0x55c5 bytes) 000914 1787830703.947871 trunc from 0x039e52 to 0x016d45 (0x2310c bytes) 000915 1787830703.971500 trunc from 0x016d46 to 0x00bfb3 (0xad92 bytes) 000916 1787830703.981511 trunc from 0x00bfb4 to 0x03e983 (0x329d0 bytes) 000917 1787830703.997565 mapread 0x01486d thru 0x018d33 (0x44c7 bytes) 000919 1787830704.004796 trunc from 0x02acb2 to 0x02f63b (0x498a bytes) 000920 1787830704.009481 trunc from 0x03e984 to 0x029a1a (0x14f69 bytes) 000921 1787830704.025811 fallocate 0x02bf7b thru 0x034b3a (0x8bc0 bytes) 000922 1787830704.035347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34b3b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b3b): Mirror 2: 0x9961c482 (stale) CRC-32 checksum value for chunk [0, 0x34b3b): Mirror 1: 0x235cc16b Mirror 3: 0x235cc16b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b3b: skipped 000923 1787830704.503577 trunc from 0x034b3b to 0x01a8c7 (0x1a273 bytes) 000924 1787830704.540180 punch from 0x013926 to 0x01a8c7 (0x6fa2 bytes) 000925 1787830704.543732 trunc from 0x01376f to 0x018167 (0x49f9 bytes) 000926 1787830704.547626 write 0x01b82b thru 0x02251e (0x6cf4 bytes) 000927 1787830704.548779 fallocate 0x00afcd thru 0x00d2fd (0x2331 bytes) 000928 1787830704.553097 fallocate 0x02efea thru 0x035ddf (0x6df6 bytes) 000929 1787830704.558437 write 0x025b9c thru 0x03517c (0xf5e1 bytes) 000930 1787830704.570296 mapwrite 0x01ed79 thru 0x0211e1 (0x2469 bytes) 000931 1787830704.587050 read 0x00e7aa thru 0x015152 (0x69a9 bytes) 000932 1787830704.603506 read 0x00ca93 thru 0x01afa2 (0xe510 bytes) 000933 1787830704.620686 read 0x03041c thru 0x035ddf (0x59c4 bytes) 000934 1787830704.621386 trunc from 0x035de0 to 0x01b419 (0x1a9c6 bytes) 000935 1787830704.652035 read 0x003e5a thru 0x009937 (0x5ade bytes) 000936 1787830704.653872 read 0x00768f thru 0x0114ea (0x9e5c bytes) 000937 1787830704.655861 write 0x02a06b thru 0x02c384 (0x231a bytes) 000938 1787830704.657333 fallocate 0x03f8f1 thru 0x03ffff (0x70f bytes) 000939 1787830704.664784 mapread 0x00b459 thru 0x0194bd (0xe065 bytes) 000940 1787830704.666402 trunc from 0x02c385 to 0x0056e1 (0x26ca3 bytes) 000941 1787830704.689419 trunc from 0x0056e2 to 0x0076e5 (0x2004 bytes) 000942 1787830704.704520 fallocate 0x028955 thru 0x030dc9 (0x8475 bytes) 000943 1787830704.709882 punch from 0x003665 to 0x012d0d (0xf6a9 bytes) 000945 1787830704.714692 trunc from 0x030dca to 0x0027bc (0x2e60d bytes) 000946 1787830704.731712 mapread 0x000e81 thru 0x0027bc (0x193c bytes) 000947 1787830704.733226 punch from 0x000a7f to 0x0027bc (0x1d3e bytes) 000948 1787830704.738539 mapread 0x0021af thru 0x0027bc (0x60e bytes) 000949 1787830704.747648 write 0x0282fb thru 0x031ec4 (0x9bca bytes) 000950 1787830704.749568 punch from 0x025474 to 0x029794 (0x4321 bytes) 000951 1787830704.765730 fallocate 0x036bb6 thru 0x03e243 (0x768e bytes) 000952 1787830704.772039 mapwrite 0x016825 thru 0x02614c (0xf928 bytes) 000953 1787830704.808182 fallocate 0x0283ad thru 0x0349b9 (0xc60d bytes) 000954 1787830704.813118 punch from 0x0134fb to 0x018be3 (0x56e9 bytes) 000955 1787830704.826183 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x349ba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349ba): Mirror 2: 0xe2b5e10d (stale) Mirror 3: 0xa318b91b (stale) CRC-32 checksum value for chunk [0, 0x349ba): Mirror 1: 0x5dce4cd5 Mirror 4: 0x5dce4cd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349ba: skipped 000956 1787830705.356224 read 0x013143 thru 0x0185a4 (0x5462 bytes) 000957 1787830705.361168 punch from 0x02cfe0 to 0x032d0d (0x5d2e bytes) 000958 1787830705.382539 mapread 0x00a306 thru 0x00c6d0 (0x23cb bytes) 000959 1787830705.392376 write 0x039bd1 thru 0x03ffff (0x642f bytes) 000960 1787830705.397447 trunc from 0x00be8e to 0x01b8cc (0xfa3f bytes) 000961 1787830705.403058 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf49e84bb (stale) Mirror 4: 0xbf9c0f5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf49e84bb (stale) Mirror 4: 0xbf9c0f5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000963 1787830705.574091 fallocate 0x01b946 thru 0x0258df (0x9f9a bytes) 000964 1787830705.578658 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf49e84bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf49e84bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000965 1787830705.729847 mapread 0x02cd48 thru 0x037f30 (0xb1e9 bytes) 000966 1787830705.754323 trunc from 0x040000 to 0x02d86f (0x12790 bytes) 000967 1787830705.771095 mapwrite 0x021b22 thru 0x026f19 (0x53f8 bytes) 000968 1787830705.791138 trunc from 0x01a3d0 to 0x023fc8 (0x9bf9 bytes) 000969 1787830705.794990 fallocate 0x007575 thru 0x0131fb (0xbc87 bytes) 000970 1787830705.799338 fallocate 0x024b97 thru 0x0319fe (0xce68 bytes) 000971 1787830705.806207 trunc from 0x01198a to 0x015350 (0x39c7 bytes) 000972 1787830705.809358 fallocate 0x01d858 thru 0x02717a (0x9923 bytes) 000974 1787830705.813652 read 0x01e8e1 thru 0x02c291 (0xd9b1 bytes) 000975 1787830705.821897 trunc from 0x0319ff to 0x03fcb0 (0xe2b2 bytes) 000976 1787830705.829893 read 0x022953 thru 0x026b80 (0x422e bytes) 000977 1787830705.830414 read 0x01abd9 thru 0x028ab3 (0xdedb bytes) 000978 1787830705.841917 MIRROR_RESYNC 2 mirrors 000979 1787830706.213881 punch from 0x006b9c to 0x01075a (0x9bbf bytes) 000980 1787830706.232673 mapwrite 0x02f0bc thru 0x032ce2 (0x3c27 bytes) 000981 1787830706.250601 mapread 0x01d339 thru 0x027d2e (0xa9f6 bytes) 000983 1787830706.258134 mapwrite 0x02bbcf thru 0x02fcfc (0x412e bytes) 000984 1787830706.269300 mapread 0x02aa0c thru 0x034f8d (0xa582 bytes) 000986 1787830706.270801 punch from 0x016ed2 to 0x01a923 (0x3a52 bytes) 000987 1787830706.276013 trunc from 0x03fcb1 to 0x017c60 (0x28050 bytes) 000988 1787830706.288235 trunc from 0x017c61 to 0x01270f (0x5551 bytes) 000989 1787830706.305525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000990 1787830706.426782 trunc from 0x00dab5 to 0x01270f (0x4c5b bytes) 000991 1787830706.430951 mapread 0x007786 thru 0x00b510 (0x3d8b bytes) 000992 1787830706.442463 trunc from 0x00cd50 to 0x01270f (0x59c0 bytes) 000993 1787830706.445237 fallocate 0x020609 thru 0x023019 (0x2a11 bytes) 000994 1787830706.448579 punch from 0x00fee8 to 0x01f0b4 (0xf1cd bytes) 000995 1787830706.452194 punch from 0x000361 to 0x00aa87 (0xa727 bytes) 000996 1787830706.457143 fallocate 0x013acc thru 0x018474 (0x49a9 bytes) 000997 1787830706.461369 mapread 0x01ec60 thru 0x023019 (0x43ba bytes) 000998 1787830706.467026 fallocate 0x00c254 thru 0x014a99 (0x8846 bytes) 000999 1787830706.475194 mapread 0x01275b thru 0x013b50 (0x13f6 bytes) 001000 1787830706.483814 mapread 0x0038a7 thru 0x00d82e (0x9f88 bytes) All operations completed A-OK! PASS 70a (51s) == sanity-flr test 71: check mirror extend parent fid ==== 07:38:29 (1787830709) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x3d1:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 07:38:34 (1787830714) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123711 s, 17.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644626 s, 16.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 07:38:39 (1787830719) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5026624 .. Extending file size to 8773504 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6790400 .. Extending file size to 8175232 .. done Extending file size to 9141312 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8880640 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8978560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 63915 63916 63917 63919 63922 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) 192.168.203.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:39:47 (1787830787) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2214592 .. Extending file size to 2375328 .. Extending file size to 8445152 .. Extending file size to 4278304 .. Extending file size to 7342976 .. Extending file size to 7769728 .. Extending file size to 8625984 .. Extending file size to 8831296 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8994912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9025824 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 66435 66436 66437 66438 66440 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) 192.168.203.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:40:55 (1787830855) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17588 3589432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25140 3581880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42728 7171312 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2384 1268920 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17588 3589432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25140 3581880 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42728 7171312 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0708522 s, 29.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142768 s, 14.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xe0586012 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:41:10 (1787830870) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0561307 s, 37.4 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 07:41:17 (1787830877) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:41:23 (1787830883) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.085173 s, 24.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:41:29 (1787830889) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0535568 s, 39.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x500:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x500:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x501:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:41:34 (1787830894) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109131 s, 19.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:42:02 (1787830922) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346039 s, 30.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4ff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x503:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x502:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4ff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x503:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x502:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x500:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:42:45 (1787830965) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0545449 s, 38.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:43:28 (1787831008) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg357-server: Pool lustre.test_204f created Adding targets to pool oleg357-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg357-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0859237 s, 24.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg357-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg357-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg357-server: Pool lustre.test_204f destroyed PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:44:34 (1787831074) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:44:39 (1787831079) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:44:45 (1787831085) oleg357-server: Pool lustre.test_206 created Adding targets to pool oleg357-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg357-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg357-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg357-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg357-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:45:07 (1787831107) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601074 s, 17.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320191 s, 32.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x924e4ed3 Mirror 3: 0x924e4ed3 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:45:14 (1787831114) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:45:16 (1787831116) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:45:19 (1787831119) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0317491 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ba0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23c0000 Read this (4 bytes): BBBB 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) PASS 209a (27s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:45:46 (1787831146) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0205659 s, 398 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:45:51 (1787831151) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320121 s, 32.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9b lmm_fid: [0x200000405:0x9b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9b lmm_fid: [0x200000405:0x9b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611291 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 07:45:52.000000000 -0400 Modify: 2026-08-27 07:45:53.000000000 -0400 Change: 2026-08-27 07:45:53.000000000 -0400 Birth: 2026-08-27 07:45:53.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:45:57 (1787831157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231477 s, 45.3 MB/s oleg357-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg357-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg357-server: O/280000401/d26/15a: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:46:15 (1787831175) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.220265 s, 186 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0306344 s, 134 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 07:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa0:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 07:46 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:46:22 (1787831182) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:46:30 (1787831190) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1084 sec =========== 07:46:39 (1787831199) === sanity-flr: start cleanup 07:46:40 (1787831200) === === sanity-flr: finish cleanup 07:46:47 (1787831207) ===