-----============= acceptance-small: sanity-flr ============----- Thu Sep 3 17:18:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:19:05 (1788470345) === oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91538720b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91538720b800.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:19:56 (1788470396) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8237] == sanity-flr test 0a: lfs mirror create with -N option == 17:19:59 (1788470399) PASS 0a (26s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:20:25 (1788470425) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (6s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:20:32 (1788470432) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:20:36 (1788470436) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:21:03 (1788470463) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:21:06 (1788470466) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:21:09 (1788470469) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91538720b800.stats=clear osc.lustre-OST0001-osc-ffff91538720b800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:21:18 (1788470478) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:21:30 (1788470490) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:21:39 (1788470499) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:21:42 (1788470502) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:21:50 (1788470510) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 17:21:59 (1788470519) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:22:07 (1788470527) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:22:15 (1788470535) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:22:23 (1788470543) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:22:35 (1788470555) PASS 7 (11s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:22:46 (1788470566) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287844 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142247 s, 7.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:22:57 (1788470577) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.72634 s, 27.4 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:23:08 (1788470588) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:23:20 (1788470600) Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:24:44 (1788470684) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11931 s, 17.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:25:34 (1788470734) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (101s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:27:15 (1788470835) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.04607 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 3 17:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.968s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.286s user 0m0.000s sys 0m0.205s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.371s user 0m0.001s sys 0m0.199s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91538720b800.ost_server_uuid in FULL state after 0 sec PASS 33b (106s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:29:01 (1788470941) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:29:05 (1788470945) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:29:09 (1788470949) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 17:29:13 (1788470953) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 17:29:23 (1788470963) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.434578 s, 29.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:29:36 (1788470976) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531334 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381612 s, 27.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:29:45 (1788470985) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:29:54 (1788470994) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0032139 s, 319 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143309 s, 715 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148961 s, 687 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105489 s, 971 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 17:30:05 (1788471005) create test files with size 15737482 .. 13f11d5cb1298e66fff72e7c7a75cc78 - 3dc2557d9006e0ce6d9daaa5b8061cea - a2708b4c2fff699c330711b2ab13f117 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 17:30:33 (1788471033) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:30:50 (1788471050) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:30:58 (1788471058) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05948 s, 17.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507458 s, 20.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 17:31:08 (1788471068) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0963083 s, 21.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.25782 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201379 s, 10.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 17:31:22 (1788471082) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 17:31:26 (1788471086) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:31:31 (1788471091) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261841 s, 40.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 17:31:39 (1788471099) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.307335 s, 10.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:31:51 (1788471111) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:32:04 (1788471124) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.401272 s, 26.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:32:18 (1788471138) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff915387349000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.37235 s, 28.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9153924e0000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:32:36 (1788471156) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:33:26 (1788471206) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (51s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:34:17 (1788471257) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 17:34:21 (1788471261) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.254139 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.274608 s, 11.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xedd5c309 (nosync,stale) Mirror 2: 0xedd5c309 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7fef8afcdf1b7256d8ac19322ce9d393 - mirror 1 checksum: 88174c45dcd7af2d59d44f9ada24187a - mirror 2 checksum: 7fef8afcdf1b7256d8ac19322ce9d393 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T17:34:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T17:34:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7fef8afcdf1b7256d8ac19322ce9d393 - mirror 1 checksum: 88174c45dcd7af2d59d44f9ada24187a - mirror 2 checksum: 88174c45dcd7af2d59d44f9ada24187a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T17:34:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T17:34:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:34:33 (1788471273) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295854 s, 35.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:34:41 (1788471281) Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server 17:34:55 (1788471295) shut down facet: mds1 facet_host: oleg450-server facet_failover_host: oleg450-server Failover mds1 to oleg450-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 17:35:27 (1788471327) targets are mounted 17:35:27 (1788471327) facet_failover done oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342391 s, 30.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:35:45 (1788471345) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0353585 s, 579 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.011219 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:35:55 (1788471355) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0537368 s, 381 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0114707 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:36:05 (1788471365) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0123592 s, 1.3 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:36:15 (1788471375) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0911147 s, 225 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:36:23 (1788471383) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0208325 s, 197 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0449476 s, 91.1 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:36:33 (1788471393) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:36:55 (1788471415) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:37:24 (1788471444) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:37:49 (1788471469) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:38:15 (1788471495) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 17:38:24 (1788471504) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57605 mirror split pid 57606 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57605 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57606 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 17:39:32 (1788471572) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1575 skipping zero length zero range skipping zero size read skipping zero length zero range fallocating to largest ever: 0x36e25 000006 1788471575.520287 fallocate 0x02ac7a thru 0x036e24 (0xc1ab bytes) truncating to largest ever: 0x37333 000007 1788471575.526407 trunc from 0x036e25 to 0x037332 (0x50e bytes) 000008 1788471575.552336 punch from 0x026043 to 0x029cf1 (0x3caf bytes) 000011 1788471575.561371 trunc from 0x037333 to 0x021a94 (0x1589e bytes) fallocating to largest ever: 0x40000 000012 1788471575.584539 fallocate 0x03dad4 thru 0x03ffff (0x252c bytes) 000014 1788471575.591352 punch from 0x00b7d9 to 0x0174ee (0xbd16 bytes) 000015 1788471575.598296 read 0x008950 thru 0x017cdc (0xf38d bytes) 000017 1788471575.609714 mapwrite 0x007a90 thru 0x00b247 (0x37b8 bytes) 000018 1788471575.636221 mapread 0x001d32 thru 0x00c8af (0xab7e bytes) 000019 1788471575.655886 mapread 0x018653 thru 0x0203d4 (0x7d82 bytes) 000020 1788471575.668211 fallocate 0x012cb7 thru 0x013466 (0x7b0 bytes) 000021 1788471575.673925 trunc from 0x021a95 to 0x039cc9 (0x18235 bytes) 000022 1788471575.696043 mapread 0x034826 thru 0x039416 (0x4bf1 bytes) 000023 1788471575.707063 fallocate 0x005ef8 thru 0x00fd2d (0x9e36 bytes) 000025 1788471575.713317 mapwrite 0x019f3e thru 0x0208ae (0x6971 bytes) 000026 1788471575.738465 fallocate 0x0001ad thru 0x000749 (0x59d bytes) 000027 1788471575.744590 read 0x00ec67 thru 0x019cc3 (0xb05d bytes) 000028 1788471575.745216 mapread 0x0366a9 thru 0x03854f (0x1ea7 bytes) 000029 1788471575.746059 read 0x01b166 thru 0x01c030 (0xecb bytes) 000031 1788471575.747918 mapwrite 0x016ae7 thru 0x01f0c4 (0x85de bytes) 000032 1788471575.769010 fallocate 0x03fd6f thru 0x03ffff (0x291 bytes) 000033 1788471575.774608 write 0x01d899 thru 0x0271f9 (0x9961 bytes) 000034 1788471575.784026 trunc from 0x040000 to 0x03bf87 (0x4078 bytes) 000035 1788471575.800320 read 0x02f210 thru 0x03bf87 (0xcd78 bytes) 000036 1788471575.809465 trunc from 0x03bf88 to 0x036f8c (0x4ffb bytes) 000038 1788471575.825989 trunc from 0x036f8d to 0x0315eb (0x59a1 bytes) 000039 1788471575.839562 trunc from 0x0315ec to 0x03879b (0x71b0 bytes) 000040 1788471575.858317 punch from 0x00d0c9 to 0x0195b6 (0xc4ee bytes) 000041 1788471575.864084 trunc from 0x028c4c to 0x02c63a (0x39ef bytes) 000042 1788471575.868574 trunc from 0x03879c to 0x017c50 (0x20b4b bytes) 000043 1788471575.882272 trunc from 0x017c51 to 0x005617 (0x12639 bytes) 000044 1788471575.898691 trunc from 0x0036ae to 0x005617 (0x1f6a bytes) 000045 1788471575.903198 punch from 0x0029b3 to 0x005617 (0x2c65 bytes) 000046 1788471575.907811 read 0x003475 thru 0x005617 (0x21a3 bytes) 000047 1788471575.916361 punch from 0x005113 to 0x005617 (0x505 bytes) 000048 1788471575.921987 write 0x00b8f5 thru 0x00f856 (0x3f62 bytes) 000049 1788471575.924202 trunc from 0x005e03 to 0x009bae (0x3dac bytes) 000050 1788471575.929184 read 0x009a9c thru 0x00f856 (0x5dbb bytes) 000051 1788471575.936919 fallocate 0x02b063 thru 0x034174 (0x9112 bytes) 000054 1788471575.944686 mapwrite 0x027b6e thru 0x02b2a7 (0x373a bytes) 000055 1788471575.983734 read 0x013df4 thru 0x016055 (0x2262 bytes) 000056 1788471575.984310 write 0x0129e3 thru 0x01be06 (0x9424 bytes) 000057 1788471575.985512 read 0x01ce75 thru 0x029185 (0xc311 bytes) 000058 1788471575.987850 mapread 0x0166c8 thru 0x01b578 (0x4eb1 bytes) 000060 1788471575.991634 mapwrite 0x03f0fc thru 0x03ffff (0xf04 bytes) 000061 1788471576.047226 MIRROR_EXTEND to 2 mirrors 000062 1788471576.640255 fallocate 0x001ecb thru 0x007316 (0x544c bytes) 000063 1788471576.682444 trunc from 0x034a05 to 0x03f28c (0xa888 bytes) 000064 1788471576.687781 trunc from 0x001de8 to 0x007a26 (0x5c3f bytes) 000065 1788471576.692999 mapwrite 0x0062bb thru 0x00c1d2 (0x5f18 bytes) 000066 1788471576.724835 read 0x003693 thru 0x00c7e2 (0x9150 bytes) 000067 1788471576.729771 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x41f0a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc198cc3e Mirror 3: 0xc198cc3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1788471577.518249 trunc from 0x03b823 to 0x03ffff (0x47dd bytes) 000069 1788471577.553654 trunc from 0x03e0b2 to 0x03ffff (0x1f4e bytes) 000070 1788471577.557909 fallocate 0x018a1a thru 0x0216a6 (0x8c8d bytes) 000071 1788471577.563067 write 0x029e43 thru 0x02cf09 (0x30c7 bytes) 000072 1788471577.576052 mapread 0x000607 thru 0x000a5e (0x458 bytes) 000073 1788471577.581940 read 0x02d05c thru 0x03805b (0xb000 bytes) 000074 1788471577.591306 mapwrite 0x00d404 thru 0x0178fd (0xa4fa bytes) 000075 1788471577.635362 trunc from 0x030f67 to 0x034380 (0x341a bytes) 000076 1788471577.641768 trunc from 0x040000 to 0x003352 (0x3ccad bytes) 000077 1788471577.660180 punch from 0x000aee to 0x003352 (0x2865 bytes) 000078 1788471577.664752 mapwrite 0x0051a0 thru 0x00e36f (0x91d0 bytes) 000079 1788471577.704258 trunc from 0x00e370 to 0x020408 (0x12099 bytes) 000080 1788471577.719560 mapread 0x012c99 thru 0x020408 (0xd770 bytes) 000081 1788471577.740115 punch from 0x018888 to 0x01e4c1 (0x5c3a bytes) 000082 1788471577.745107 punch from 0x01f2af to 0x020408 (0x115a bytes) 000083 1788471577.749765 write 0x015211 thru 0x017e82 (0x2c72 bytes) 000084 1788471577.750893 trunc from 0x00e666 to 0x013ee8 (0x5883 bytes) 000085 1788471577.756439 read 0x00581c thru 0x0080b6 (0x289b bytes) 000086 1788471577.757331 mapwrite 0x008a1d thru 0x00c2ee (0x38d2 bytes) 000087 1788471577.775265 trunc from 0x020409 to 0x00b1ff (0x15209 bytes) 000088 1788471577.800432 trunc from 0x00b200 to 0x019991 (0xe792 bytes) 000089 1788471577.821183 trunc from 0x019992 to 0x00df9a (0xb9f7 bytes) 000090 1788471577.840832 trunc from 0x004cb6 to 0x00df9a (0x92e5 bytes) 000091 1788471577.846863 write 0x0385ab thru 0x03ab7b (0x25d1 bytes) 000092 1788471577.848775 mapwrite 0x038594 thru 0x03ca78 (0x44e5 bytes) 000093 1788471577.904441 punch from 0x00203a to 0x002400 (0x3c7 bytes) 000094 1788471577.916570 mapwrite 0x033486 thru 0x037e5b (0x49d6 bytes) 000095 1788471577.951400 fallocate 0x01946e thru 0x01e91d (0x54b0 bytes) 000096 1788471577.962077 fallocate 0x01c5d2 thru 0x023bd5 (0x7604 bytes) 000097 1788471577.972057 read 0x010333 thru 0x018b06 (0x87d4 bytes) 000098 1788471577.986334 MIRROR_RESYNC 3 mirrors 000099 1788471578.595306 punch from 0x0101ec to 0x019df5 (0x9c0a bytes) 000100 1788471578.627323 fallocate 0x0081a1 thru 0x00853b (0x39b bytes) 000101 1788471578.632139 trunc from 0x02c0e3 to 0x0350b5 (0x8fd3 bytes) 000102 1788471578.636946 fallocate 0x034afd thru 0x03569c (0xba0 bytes) 000103 1788471578.641204 trunc from 0x03ca79 to 0x01d625 (0x1f453 bytes) 000104 1788471578.660867 fallocate 0x02bb27 thru 0x036d3f (0xb219 bytes) 000105 1788471578.666135 trunc from 0x036d40 to 0x03b8bd (0x4b7e bytes) 000106 1788471578.682157 trunc from 0x017baf to 0x01e0b5 (0x6507 bytes) 000107 1788471578.687195 read 0x00c895 thru 0x01aec1 (0xe62d bytes) 000108 1788471578.696217 write 0x032512 thru 0x033244 (0xd33 bytes) 000109 1788471578.705237 fallocate 0x0366a7 thru 0x03dd3e (0x7698 bytes) 000110 1788471578.709359 trunc from 0x03b8be to 0x028763 (0x1315a bytes) 000111 1788471578.728045 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28764): Mirror 3: 0x8bb1bc0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28764): Mirror 3: 0x8bb1bc0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28764: skipped 000112 1788471578.984611 mapread 0x02263a thru 0x026f25 (0x48ec bytes) 000113 1788471579.002469 mapwrite 0x00b108 thru 0x00fb93 (0x4a8c bytes) 000114 1788471579.028052 punch from 0x00eeb0 to 0x01847c (0x95cd bytes) 000115 1788471579.033078 mapwrite 0x02a31c thru 0x035d46 (0xba2b bytes) 000116 1788471579.087293 punch from 0x022bcf to 0x0230d3 (0x505 bytes) 000118 1788471579.092886 fallocate 0x0217e4 thru 0x022d25 (0x1542 bytes) 000119 1788471579.097571 write 0x015855 thru 0x023270 (0xda1c bytes) 000120 1788471579.107026 read 0x004185 thru 0x004fcf (0xe4b bytes) 000121 1788471579.107728 read 0x0352d3 thru 0x035d46 (0xa74 bytes) 000122 1788471579.112712 MIRROR_RESYNC 2 mirrors 000124 1788471579.623355 read 0x00c22c thru 0x016400 (0xa1d5 bytes) 000125 1788471579.641807 write 0x00a6fd thru 0x010ce4 (0x65e8 bytes) 000126 1788471579.692526 fallocate 0x0245b4 thru 0x026e05 (0x2852 bytes) 000127 1788471579.700381 mapwrite 0x0198a5 thru 0x01d142 (0x389e bytes) 000128 1788471579.732510 mapwrite 0x00aad6 thru 0x01199c (0x6ec7 bytes) 000129 1788471579.756253 write 0x021705 thru 0x02278a (0x1086 bytes) 000130 1788471579.757318 read 0x025e5a thru 0x026a95 (0xc3c bytes) 000131 1788471579.766251 punch from 0x00222e to 0x011120 (0xeef3 bytes) 000132 1788471579.771728 read 0x00852d thru 0x008a53 (0x527 bytes) 000133 1788471579.782878 trunc from 0x035d47 to 0x03de64 (0x811e bytes) 000134 1788471579.821254 read 0x018ac1 thru 0x01b675 (0x2bb5 bytes) 000135 1788471579.822028 write 0x02304a thru 0x02aa02 (0x79b9 bytes) 000136 1788471579.824075 write 0x0012d0 thru 0x004f4a (0x3c7b bytes) 000137 1788471579.826114 mapwrite 0x01a53d thru 0x01cd02 (0x27c6 bytes) 000138 1788471579.848046 write 0x02a854 thru 0x02b963 (0x1110 bytes) 000139 1788471579.853021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000140 1788471580.002238 trunc from 0x03de65 to 0x03a44f (0x3a15 bytes) 000141 1788471580.021108 fallocate 0x02d5f8 thru 0x03bbd9 (0xe5e2 bytes) 000143 1788471580.026701 fallocate 0x006080 thru 0x0117f0 (0xb771 bytes) 000144 1788471580.033316 trunc from 0x03bbda to 0x03e3fd (0x2824 bytes) 000145 1788471580.077369 write 0x027494 thru 0x02bcb6 (0x4823 bytes) 000147 1788471580.093135 punch from 0x0131bf to 0x015ac3 (0x2905 bytes) 000148 1788471580.099332 write 0x01e287 thru 0x0240e3 (0x5e5d bytes) 000150 1788471580.117258 fallocate 0x00da29 thru 0x01c896 (0xee6e bytes) 000151 1788471580.123085 mapread 0x0280de thru 0x028579 (0x49c bytes) 000154 1788471580.125667 write 0x02172d thru 0x02fb74 (0xe448 bytes) 000155 1788471580.136354 fallocate 0x03a666 thru 0x03ffff (0x599a bytes) 000156 1788471580.142588 write 0x00e89e thru 0x01b097 (0xc7fa bytes) 000157 1788471580.152785 mapread 0x02e91f thru 0x02ea43 (0x125 bytes) 000158 1788471580.153983 read 0x006bd4 thru 0x011a87 (0xaeb4 bytes) 000159 1788471580.164201 read 0x011267 thru 0x018ecd (0x7c67 bytes) 000160 1788471580.165102 read 0x039388 thru 0x03ffff (0x6c78 bytes) 000162 1788471580.174860 trunc from 0x036660 to 0x03c8d8 (0x6279 bytes) 000163 1788471580.179595 trunc from 0x040000 to 0x01dc35 (0x223ca bytes) 000164 1788471580.194465 mapread 0x00a024 thru 0x015aa4 (0xba81 bytes) 000165 1788471580.195448 fallocate 0x005ded thru 0x0124c1 (0xc6d5 bytes) 000166 1788471580.201467 fallocate 0x01bf6a thru 0x02056a (0x4601 bytes) 000167 1788471580.208233 trunc from 0x019ebe to 0x02056a (0x66ad bytes) 000168 1788471580.227324 read 0x001e69 thru 0x004a02 (0x2b9a bytes) 000169 1788471580.232160 MIRROR_EXTEND to 2 mirrors 000170 1788471580.948963 mapwrite 0x03050a thru 0x036751 (0x6248 bytes) 000171 1788471581.094873 mapread 0x01fc9b thru 0x02ef67 (0xf2cd bytes) 000172 1788471581.112759 mapread 0x02bb78 thru 0x02cf97 (0x1420 bytes) 000173 1788471581.114115 write 0x026f9c thru 0x02d19f (0x6204 bytes) 000174 1788471581.116292 fallocate 0x00d2b4 thru 0x013a1f (0x676c bytes) 000175 1788471581.127670 trunc from 0x036752 to 0x01fe01 (0x16950 bytes) 000176 1788471581.150080 punch from 0x003ef6 to 0x013d1b (0xfe26 bytes) 000177 1788471581.155338 mapread 0x0004ae thru 0x002dd9 (0x292c bytes) 000178 1788471581.164538 trunc from 0x01fe02 to 0x023efa (0x40f9 bytes) 000179 1788471581.189717 write 0x03bf25 thru 0x03bfdc (0xb8 bytes) 000181 1788471581.193446 mapwrite 0x03013f thru 0x0318b5 (0x1777 bytes) 000182 1788471581.256169 MIRROR_RESYNC 2 mirrors 000184 1788471581.831162 mapwrite 0x000913 thru 0x000b11 (0x1ff bytes) 000185 1788471581.902433 write 0x028cbf thru 0x0300c0 (0x7402 bytes) 000186 1788471581.919952 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000187 1788471582.179474 mapwrite 0x026d92 thru 0x035b64 (0xedd3 bytes) 000188 1788471582.262241 mapread 0x01ae4c thru 0x024594 (0x9749 bytes) 000189 1788471582.283051 punch from 0x033fbd to 0x037b11 (0x3b55 bytes) 000191 1788471582.294159 trunc from 0x03bfdd to 0x00d92e (0x2e6ae bytes) 000192 1788471582.364147 write 0x032976 thru 0x03974e (0x6dd9 bytes) 000193 1788471582.370424 read 0x00d250 thru 0x017996 (0xa747 bytes) 000195 1788471582.390379 trunc from 0x03974f to 0x01964b (0x20103 bytes) 000197 1788471582.418316 trunc from 0x009803 to 0x00d260 (0x3a5e bytes) 000198 1788471582.423057 mapread 0x005c44 thru 0x014381 (0xe73e bytes) 000199 1788471582.439846 mapwrite 0x00cb0b thru 0x0144fd (0x79f3 bytes) 000200 1788471582.482898 fallocate 0x002e1d thru 0x00a998 (0x7b7c bytes) 000201 1788471582.495549 fallocate 0x0002cd thru 0x00501b (0x4d4f bytes) 000202 1788471582.512131 trunc from 0x01964c to 0x00813a (0x11511 bytes) 000203 1788471582.540212 write 0x013d16 thru 0x01d7bc (0x9aa7 bytes) 000204 1788471582.544031 fallocate 0x03a758 thru 0x03b60c (0xeb5 bytes) 000205 1788471582.551148 punch from 0x00610b to 0x00a7da (0x46d0 bytes) 000206 1788471582.556548 trunc from 0x02c058 to 0x037b48 (0xbaf1 bytes) 000208 1788471582.562970 mapwrite 0x031a9f thru 0x03b650 (0x9bb2 bytes) 000209 1788471582.624395 fallocate 0x0050f7 thru 0x01031a (0xb224 bytes) 000210 1788471582.630806 punch from 0x03a8bf to 0x03b650 (0xd92 bytes) 000211 1788471582.638281 trunc from 0x027d6b to 0x034851 (0xcae7 bytes) 000212 1788471582.649252 read 0x02da0b thru 0x036856 (0x8e4c bytes) 000213 1788471582.670785 trunc from 0x02523f to 0x02d237 (0x7ff9 bytes) 000214 1788471582.680056 read 0x02652a thru 0x02c212 (0x5ce9 bytes) 000215 1788471582.698529 punch from 0x02bde9 to 0x0381a5 (0xc3bd bytes) 000216 1788471582.709089 mapread 0x00c87f thru 0x00e095 (0x1817 bytes) 000217 1788471582.710228 write 0x014400 thru 0x01f52e (0xb12f bytes) 000218 1788471582.712318 trunc from 0x03b651 to 0x028493 (0x131bd bytes) 000219 1788471582.734664 mapread 0x020219 thru 0x027a71 (0x7859 bytes) 000220 1788471582.740212 trunc from 0x028494 to 0x03d1da (0x14d47 bytes) 000221 1788471582.765518 write 0x00cb3e thru 0x014ff0 (0x84b3 bytes) 000222 1788471582.766950 mapread 0x011a99 thru 0x01fe43 (0xe3ab bytes) 000223 1788471582.769015 read 0x006e92 thru 0x00d4a5 (0x6614 bytes) 000224 1788471582.770278 punch from 0x01ded9 to 0x021e82 (0x3faa bytes) 000226 1788471582.789617 fallocate 0x00ed92 thru 0x014a34 (0x5ca3 bytes) 000227 1788471582.797338 fallocate 0x02c781 thru 0x0303c9 (0x3c49 bytes) 000228 1788471582.804459 trunc from 0x03d1db to 0x038545 (0x4c95 bytes) 000229 1788471582.821239 mapread 0x00368d thru 0x005dd1 (0x2745 bytes) 000230 1788471582.822465 write 0x00b771 thru 0x01720d (0xba9d bytes) 000231 1788471582.827773 punch from 0x028d17 to 0x035d20 (0xd00a bytes) 000232 1788471582.834043 punch from 0x006b70 to 0x00b988 (0x4e19 bytes) 000233 1788471582.859286 MIRROR_EXTEND to 2 mirrors 000234 1788471583.687618 mapread 0x01615a thru 0x02605f (0xff06 bytes) 000235 1788471583.764779 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000237 1788471583.952105 fallocate 0x026f01 thru 0x0323e2 (0xb4e2 bytes) 000239 1788471583.965467 trunc from 0x031660 to 0x0372ed (0x5c8e bytes) 000241 1788471583.972438 write 0x03a460 thru 0x03ffff (0x5ba0 bytes) 000242 1788471583.977900 trunc from 0x0348af to 0x03c244 (0x7996 bytes) 000243 1788471583.990714 mapwrite 0x00b0f7 thru 0x0184bd (0xd3c7 bytes) 000244 1788471584.069794 mapwrite 0x028275 thru 0x0329da (0xa766 bytes) 000246 1788471584.111437 fallocate 0x001469 thru 0x00b48d (0xa025 bytes) 000247 1788471584.115415 trunc from 0x040000 to 0x01d1a5 (0x22e5a bytes) 000248 1788471584.137032 fallocate 0x03fc28 thru 0x03ffff (0x3d8 bytes) 000249 1788471584.144943 trunc from 0x01d1a6 to 0x03952d (0x1c388 bytes) 000250 1788471584.156819 trunc from 0x03952e to 0x0097f4 (0x2fd39 bytes) 000251 1788471584.171915 read 0x008537 thru 0x008666 (0x130 bytes) 000252 1788471584.172641 trunc from 0x006502 to 0x0097f4 (0x32f3 bytes) 000253 1788471584.179131 mapwrite 0x03fee0 thru 0x03ffff (0x120 bytes) 000254 1788471584.236812 mapread 0x029cb8 thru 0x0348ae (0xabf7 bytes) 000255 1788471584.272208 trunc from 0x040000 to 0x004762 (0x3b89d bytes) 000256 1788471584.297706 write 0x015647 thru 0x02431b (0xecd5 bytes) 000257 1788471584.299646 trunc from 0x01837f to 0x02431b (0xbf9d bytes) 000258 1788471584.330391 punch from 0x023f88 to 0x02431b (0x394 bytes) 000259 1788471584.336240 fallocate 0x024e23 thru 0x02e7cb (0x99a9 bytes) 000261 1788471584.341345 mapread 0x0207bb thru 0x02485a (0x40a0 bytes) 000262 1788471584.356434 punch from 0x01322a to 0x023184 (0xff5b bytes) 000264 1788471584.368589 mapwrite 0x0059ce thru 0x01349f (0xdad2 bytes) 000265 1788471584.425667 trunc from 0x009c05 to 0x0112ce (0x76ca bytes) 000266 1788471584.432520 write 0x007573 thru 0x01280b (0xb299 bytes) 000268 1788471584.437867 mapread 0x020e57 thru 0x024b06 (0x3cb0 bytes) 000269 1788471584.448949 fallocate 0x0262f0 thru 0x02a126 (0x3e37 bytes) 000270 1788471584.454214 trunc from 0x02e7cc to 0x034eb0 (0x66e5 bytes) 000271 1788471584.471364 trunc from 0x01e124 to 0x02dc6e (0xfb4b bytes) 000272 1788471584.475952 mapread 0x01bfd2 thru 0x01f743 (0x3772 bytes) 000274 1788471584.486174 punch from 0x02f28b to 0x034eb0 (0x5c26 bytes) 000275 1788471584.490134 trunc from 0x0005b6 to 0x002e21 (0x286c bytes) 000276 1788471584.495509 trunc from 0x0147b0 to 0x0231e8 (0xea39 bytes) 000278 1788471584.501026 mapread 0x003bbf thru 0x010436 (0xc878 bytes) 000279 1788471584.502125 punch from 0x01b24b to 0x01c0e2 (0xe98 bytes) 000280 1788471584.507098 mapwrite 0x028a51 thru 0x037d69 (0xf319 bytes) 000281 1788471584.568260 punch from 0x0110a3 to 0x018aaf (0x7a0d bytes) 000282 1788471584.573820 mapwrite 0x03adde thru 0x03ffff (0x5222 bytes) 000283 1788471584.621237 trunc from 0x02fe0e to 0x03c4b0 (0xc6a3 bytes) 000284 1788471584.626748 mapwrite 0x00dec8 thru 0x00e482 (0x5bb bytes) 000285 1788471584.648286 trunc from 0x0159c2 to 0x01d832 (0x7e71 bytes) 000287 1788471584.659194 trunc from 0x040000 to 0x00be3e (0x341c1 bytes) 000288 1788471584.685926 trunc from 0x00981f to 0x00be3e (0x2620 bytes) 000289 1788471584.697083 write 0x02c78c thru 0x03414f (0x79c4 bytes) 000290 1788471584.700932 fallocate 0x03bc55 thru 0x03ffff (0x43ab bytes) 000292 1788471584.711765 punch from 0x00baed to 0x013b66 (0x807a bytes) 000293 1788471584.719250 read 0x017dc8 thru 0x024da4 (0xcfdd bytes) 000294 1788471584.772855 trunc from 0x014f70 to 0x0179ce (0x2a5f bytes) 000295 1788471584.781208 trunc from 0x00269f to 0x00d393 (0xacf5 bytes) 000296 1788471584.789139 fallocate 0x017b52 thru 0x01fbb3 (0x8062 bytes) 000297 1788471584.798202 trunc from 0x005e8b to 0x013cdb (0xde51 bytes) 000298 1788471584.804930 trunc from 0x040000 to 0x004fc8 (0x3b037 bytes) 000299 1788471584.835195 MIRROR_EXTEND to 2 mirrors 000300 1788471585.750246 mapread 0x002949 thru 0x004fc8 (0x2680 bytes) 000301 1788471585.777242 read 0x00098b thru 0x0038ee (0x2f64 bytes) 000302 1788471585.785237 mapread 0x003811 thru 0x004fc8 (0x17b8 bytes) 000303 1788471585.786694 punch from 0x0043fc to 0x004fc8 (0xbcd bytes) 000304 1788471585.832675 mapread 0x0029a9 thru 0x004fc8 (0x2620 bytes) 000305 1788471585.844498 mapwrite 0x002fb7 thru 0x007cee (0x4d38 bytes) 000306 1788471585.909228 write 0x01dde4 thru 0x0266e8 (0x8905 bytes) 000307 1788471585.914274 mapwrite 0x004be6 thru 0x0114c9 (0xc8e4 bytes) 000308 1788471585.968453 read 0x00366b thru 0x005f49 (0x28df bytes) 000309 1788471585.969303 fallocate 0x025f57 thru 0x0275c2 (0x166c bytes) 000310 1788471585.976105 trunc from 0x007c92 to 0x014130 (0xc49f bytes) 000311 1788471585.987976 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x266e9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266e9): Mirror 2: 0xa507595c (stale) CRC-32 checksum value for chunk [0, 0x266e9): Mirror 1: 0x64e00929 Mirror 3: 0x64e00929 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266e9: skipped 000312 1788471586.934413 mapread 0x014aa9 thru 0x01c6a7 (0x7bff bytes) 000313 1788471586.969696 fallocate 0x011c3a thru 0x021a7f (0xfe46 bytes) 000314 1788471587.046349 read 0x000915 thru 0x00bd5e (0xb44a bytes) 000315 1788471587.059467 mapread 0x006260 thru 0x012d04 (0xcaa5 bytes) 000316 1788471587.094223 trunc from 0x017466 to 0x019a98 (0x2633 bytes) 000317 1788471587.105135 mapread 0x0240a7 thru 0x025b4d (0x1aa7 bytes) 000318 1788471587.150995 trunc from 0x010224 to 0x01caf0 (0xc8cd bytes) 000319 1788471587.168097 fallocate 0x017ffc thru 0x019f8f (0x1f94 bytes) 000320 1788471587.198170 write 0x011f73 thru 0x016858 (0x48e6 bytes) 000321 1788471587.280422 MIRROR_RESYNC 3 mirrors 000322 1788471588.711960 trunc from 0x02473d to 0x0266e8 (0x1fac bytes) 000323 1788471588.757931 mapwrite 0x0301c4 thru 0x0319f8 (0x1835 bytes) 000324 1788471588.846911 read 0x02cb7c thru 0x030706 (0x3b8b bytes) 000325 1788471588.863358 fallocate 0x0159c7 thru 0x01828c (0x28c6 bytes) 000326 1788471588.879326 MIRROR_RESYNC 3 mirrors 000327 1788471590.382896 punch from 0x00e950 to 0x013ae2 (0x5193 bytes) 000328 1788471590.439770 punch from 0x015638 to 0x0252b5 (0xfc7e bytes) 000329 1788471590.450139 punch from 0x010c24 to 0x01cb12 (0xbeef bytes) 000330 1788471590.455880 read 0x026f86 thru 0x0319f8 (0xaa73 bytes) 000331 1788471590.468721 trunc from 0x0319f9 to 0x0012d1 (0x30727 bytes) 000332 1788471590.488122 mapwrite 0x01548e thru 0x01c37f (0x6ef2 bytes) 000333 1788471590.537751 read 0x01885e thru 0x01c37f (0x3b22 bytes) 000334 1788471590.538348 mapwrite 0x03128f thru 0x03e4c9 (0xd23b bytes) 000335 1788471590.598822 mapwrite 0x01caa0 thru 0x021d47 (0x52a8 bytes) 000336 1788471590.650065 MIRROR_RESYNC 3 mirrors 000337 1788471591.680573 trunc from 0x03e4ca to 0x03eb1d (0x654 bytes) 000338 1788471591.740095 write 0x03f499 thru 0x03ffff (0xb67 bytes) 000339 1788471591.741737 fallocate 0x0031ba thru 0x010a94 (0xd8db bytes) 000340 1788471591.748806 punch from 0x001f2e to 0x0078f8 (0x59cb bytes) 000341 1788471591.758624 write 0x030a92 thru 0x03ffff (0xf56e bytes) 000342 1788471591.770289 mapread 0x014c2d thru 0x01cd6d (0x8141 bytes) 000343 1788471591.812189 write 0x03826a thru 0x03ffff (0x7d96 bytes) 000344 1788471591.815204 write 0x034567 thru 0x03ea78 (0xa512 bytes) 000345 1788471591.823250 write 0x01ade8 thru 0x020bf0 (0x5e09 bytes) 000346 1788471591.835134 fallocate 0x017a60 thru 0x01da4e (0x5fef bytes) 000347 1788471591.841125 mapread 0x011afc thru 0x014175 (0x267a bytes) 000349 1788471591.843149 trunc from 0x01567f to 0x0233f8 (0xdd7a bytes) 000350 1788471591.864627 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x92c85f43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x92c85f43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1788471592.299264 trunc from 0x040000 to 0x0113c3 (0x2ec3c bytes) 000352 1788471592.319274 punch from 0x003fbd to 0x00f626 (0xb66a bytes) 000353 1788471592.328123 read 0x0030c1 thru 0x00d924 (0xa864 bytes) 000354 1788471592.341613 fallocate 0x03ed34 thru 0x03ffff (0x12cc bytes) 000355 1788471592.347132 mapread 0x006ec6 thru 0x00e524 (0x765f bytes) 000357 1788471592.358159 MIRROR_RESYNC 2 mirrors 000358 1788471592.891942 fallocate 0x038d27 thru 0x03ffff (0x72d9 bytes) 000359 1788471592.997847 trunc from 0x040000 to 0x002f14 (0x3d0eb bytes) 000360 1788471593.085443 read 0x000537 thru 0x001dda (0x18a4 bytes) 000361 1788471593.094314 read 0x0015dd thru 0x002f14 (0x1938 bytes) 000362 1788471593.100845 mapwrite 0x032b30 thru 0x03f8be (0xcd8f bytes) 000363 1788471593.172866 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8bf): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8bf): Mirror 3: 0xf9417064 (stale) CRC-32 checksum value for chunk [0, 0x3f8bf): Mirror 1: 0x576fe2a0 Mirror 4: 0x576fe2a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8bf: skipped 000364 1788471594.141716 trunc from 0x039a99 to 0x03f8be (0x5e26 bytes) 000365 1788471594.202845 write 0x032944 thru 0x038463 (0x5b20 bytes) 000366 1788471594.228384 read 0x026370 thru 0x033847 (0xd4d8 bytes) 000368 1788471594.244385 trunc from 0x016ea7 to 0x01de0c (0x6f66 bytes) 000369 1788471594.265319 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8bf): Mirror 4: 0x576fe2a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8bf): Mirror 4: 0x576fe2a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8bf: skipped 000370 1788471594.630821 mapwrite 0x0029e3 thru 0x004332 (0x1950 bytes) 000371 1788471594.700896 write 0x0117d3 thru 0x01e33c (0xcb6a bytes) 000372 1788471594.719736 mapread 0x002665 thru 0x00c627 (0x9fc3 bytes) 000373 1788471594.720998 fallocate 0x0081e7 thru 0x017edd (0xfcf7 bytes) 000374 1788471594.727781 trunc from 0x03f8bf to 0x03988e (0x6030 bytes) 000375 1788471594.793605 trunc from 0x016c38 to 0x01e6bd (0x7a86 bytes) 000376 1788471594.809956 trunc from 0x03988f to 0x0095aa (0x302e4 bytes) 000377 1788471594.828118 write 0x02f88d thru 0x03a690 (0xae04 bytes) 000379 1788471594.832001 trunc from 0x03a691 to 0x026ee4 (0x137ac bytes) 000380 1788471594.849607 punch from 0x026649 to 0x026ee4 (0x89c bytes) 000381 1788471594.854142 punch from 0x00db26 to 0x01a763 (0xcc3e bytes) 000382 1788471594.860684 read 0x013042 thru 0x020e82 (0xde41 bytes) 000383 1788471594.871240 fallocate 0x02a6ef thru 0x0315ac (0x6ebe bytes) 000384 1788471594.876200 read 0x02deee thru 0x0315ac (0x36bf bytes) 000385 1788471594.891930 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x315ad): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315ad): Mirror 4: 0xd0d4a4e8 (stale) CRC-32 checksum value for chunk [0, 0x315ad): Mirror 1: 0xefcbe79b Mirror 5: 0xefcbe79b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315ad: skipped 000386 1788471595.998871 trunc from 0x0239de to 0x0315ac (0xdbcf bytes) 000387 1788471596.102648 punch from 0x030c1d to 0x0315ac (0x990 bytes) 000388 1788471596.112381 mapread 0x02142c thru 0x02d710 (0xc2e5 bytes) 000389 1788471596.188137 trunc from 0x0315ad to 0x02137e (0x1022e bytes) 000390 1788471596.256763 mapwrite 0x006b67 thru 0x006d3c (0x1d6 bytes) 000391 1788471596.297053 fallocate 0x02d1b7 thru 0x039bd1 (0xca1b bytes) 000392 1788471596.312615 mapwrite 0x0353f5 thru 0x03ac4e (0x585a bytes) 000393 1788471596.418567 read 0x018e77 thru 0x01fd86 (0x6f10 bytes) 000394 1788471596.432791 mapread 0x036bd4 thru 0x03ac4e (0x407b bytes) 000395 1788471596.437038 read 0x033a06 thru 0x03ac4e (0x7249 bytes) 000396 1788471596.442670 fallocate 0x0038a6 thru 0x01294b (0xf0a6 bytes) 000398 1788471596.450077 trunc from 0x03ac4f to 0x02d447 (0xd807 bytes) 000399 1788471596.471882 mapread 0x006e00 thru 0x007bfe (0xdff bytes) 000400 1788471596.473879 punch from 0x00735a to 0x007dd7 (0xa7e bytes) 000401 1788471596.486887 fallocate 0x03bcdf thru 0x03ffff (0x4321 bytes) 000402 1788471596.498674 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d448): Mirror 5: 0xd0c45db4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d448): Mirror 5: 0xd0c45db4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d448: skipped 000403 1788471596.872400 punch from 0x0281e0 to 0x02a4eb (0x230c bytes) 000404 1788471596.878598 trunc from 0x02d448 to 0x03db48 (0x10701 bytes) 000406 1788471596.912134 write 0x012d2b thru 0x01b93e (0x8c14 bytes) 000407 1788471596.928385 trunc from 0x014b4b to 0x01f4a1 (0xa957 bytes) 000408 1788471596.954642 read 0x0219d3 thru 0x026612 (0x4c40 bytes) 000409 1788471596.964012 punch from 0x006754 to 0x01114f (0xa9fc bytes) 000410 1788471596.969666 punch from 0x0255f3 to 0x02a7cb (0x51d9 bytes) 000411 1788471596.977647 write 0x00c0dc thru 0x018b6d (0xca92 bytes) 000412 1788471596.991944 mapwrite 0x0170aa thru 0x01bccb (0x4c22 bytes) 000414 1788471597.043087 read 0x02d405 thru 0x0327d7 (0x53d3 bytes) 000415 1788471597.054147 fallocate 0x005d2e thru 0x00b147 (0x541a bytes) 000416 1788471597.060278 punch from 0x0007a0 to 0x00a3ab (0x9c0c bytes) 000417 1788471597.066300 mapwrite 0x000271 thru 0x00d3e7 (0xd177 bytes) 000419 1788471597.158783 write 0x0107bf thru 0x01cbbc (0xc3fe bytes) 000420 1788471597.162846 write 0x00763c thru 0x011e97 (0xa85c bytes) 000421 1788471597.164264 read 0x00ed84 thru 0x016d5d (0x7fda bytes) 000422 1788471597.172848 MIRROR_RESYNC 2 mirrors 000423 1788471597.797894 write 0x019655 thru 0x022ff1 (0x999d bytes) 000424 1788471597.835444 punch from 0x000389 to 0x00e489 (0xe101 bytes) 000425 1788471597.840632 punch from 0x02a3fb to 0x02d023 (0x2c29 bytes) 000426 1788471597.847726 read 0x0395df thru 0x03db48 (0x456a bytes) 000427 1788471597.858049 trunc from 0x024e2d to 0x02e781 (0x9955 bytes) 000428 1788471597.862846 write 0x03fd1e thru 0x03ffff (0x2e2 bytes) 000429 1788471597.865124 write 0x010667 thru 0x012cb3 (0x264d bytes) 000430 1788471597.881033 mapread 0x01d1a7 thru 0x01ef65 (0x1dbf bytes) 000431 1788471597.889610 mapwrite 0x0126d2 thru 0x0132bf (0xbee bytes) 000432 1788471597.938044 punch from 0x029fc2 to 0x034ddb (0xae1a bytes) 000434 1788471597.954940 mapwrite 0x00ef9a thru 0x016c59 (0x7cc0 bytes) 000435 1788471598.011230 MIRROR_RESYNC 2 mirrors 000436 1788471598.774871 fallocate 0x03f596 thru 0x03ffff (0xa6a bytes) 000438 1788471598.810782 mapread 0x0378c3 thru 0x0384a0 (0xbde bytes) 000439 1788471598.822236 write 0x009eb5 thru 0x00c25d (0x23a9 bytes) 000441 1788471598.833528 trunc from 0x040000 to 0x00869d (0x37962 bytes) 000442 1788471598.863742 mapread 0x0059ef thru 0x0064a1 (0xab3 bytes) 000443 1788471598.874904 read 0x0047d5 thru 0x00869d (0x3ec9 bytes) 000444 1788471598.875720 write 0x018c8f thru 0x0214f5 (0x8867 bytes) 000446 1788471598.878022 mapwrite 0x030995 thru 0x036f8b (0x65f7 bytes) 000447 1788471598.927868 mapread 0x007b57 thru 0x00ce7b (0x5325 bytes) 000448 1788471598.941181 trunc from 0x036f8c to 0x01e53c (0x18a4f bytes) 000449 1788471598.962671 mapread 0x00b4c3 thru 0x00d5eb (0x2129 bytes) 000450 1788471598.963604 write 0x018deb thru 0x025c19 (0xce2f bytes) 000451 1788471598.969271 trunc from 0x025c1a to 0x023684 (0x2595 bytes) 000452 1788471598.990254 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000454 1788471599.173396 write 0x006733 thru 0x00b5a5 (0x4e73 bytes) 000455 1788471599.198927 trunc from 0x023685 to 0x03783d (0x141b9 bytes) 000456 1788471599.231965 trunc from 0x03783e to 0x001b34 (0x35d09 bytes) 000457 1788471599.255958 punch from 0x00006f to 0x001314 (0x12a6 bytes) 000458 1788471599.260760 fallocate 0x03715f thru 0x03a393 (0x3235 bytes) 000460 1788471599.268674 fallocate 0x03c42a thru 0x03ffff (0x3bd6 bytes) 000461 1788471599.275643 trunc from 0x001b35 to 0x00c59f (0xaa6b bytes) 000462 1788471599.297582 trunc from 0x00c5a0 to 0x03c9ad (0x3040e bytes) 000463 1788471599.317201 fallocate 0x020acd thru 0x02f7d3 (0xed07 bytes) 000465 1788471599.323877 write 0x01058f thru 0x013824 (0x3296 bytes) 000466 1788471599.335450 trunc from 0x03c9ae to 0x02611c (0x16891 bytes) 000468 1788471599.357096 trunc from 0x02611d to 0x014ea5 (0x11277 bytes) 000469 1788471599.373297 trunc from 0x014ea6 to 0x029837 (0x14992 bytes) 000470 1788471599.394240 punch from 0x028702 to 0x029837 (0x1136 bytes) 000472 1788471599.400921 punch from 0x00f1a4 to 0x0121ec (0x3049 bytes) 000474 1788471599.414033 mapread 0x027a5d thru 0x029837 (0x1ddb bytes) 000475 1788471599.435221 trunc from 0x029838 to 0x009a35 (0x1fe02 bytes) 000476 1788471599.452986 trunc from 0x009a36 to 0x00af38 (0x1503 bytes) 000477 1788471599.470455 fallocate 0x01b7cd thru 0x0298a8 (0xe0dc bytes) 000478 1788471599.477021 read 0x02172f thru 0x027513 (0x5de5 bytes) 000479 1788471599.487505 trunc from 0x00bf7e to 0x00e01d (0x20a0 bytes) 000480 1788471599.492508 read 0x009df6 thru 0x00bcf6 (0x1f01 bytes) 000481 1788471599.503856 trunc from 0x026411 to 0x0298a8 (0x3498 bytes) 000482 1788471599.509011 mapwrite 0x01d025 thru 0x02a3da (0xd3b6 bytes) 000484 1788471599.583935 read 0x004262 thru 0x004bb6 (0x955 bytes) 000485 1788471599.584525 punch from 0x012d38 to 0x01dfe7 (0xb2b0 bytes) 000486 1788471599.591214 punch from 0x018d38 to 0x01b0d7 (0x23a0 bytes) 000487 1788471599.599336 mapwrite 0x01ce66 thru 0x01d54a (0x6e5 bytes) 000488 1788471599.632662 punch from 0x00896d to 0x0160ce (0xd762 bytes) 000489 1788471599.641726 trunc from 0x02a3db to 0x03c6f6 (0x1231c bytes) 000490 1788471599.663975 write 0x038517 thru 0x03ffff (0x7ae9 bytes) 000491 1788471599.684032 punch from 0x005a44 to 0x00dfbc (0x8579 bytes) 000492 1788471599.694225 trunc from 0x040000 to 0x006514 (0x39aeb bytes) 000493 1788471599.728220 mapread 0x001f39 thru 0x006514 (0x45dc bytes) 000494 1788471599.741760 punch from 0x001f12 to 0x00273b (0x82a bytes) 000495 1788471599.755109 read 0x0025de thru 0x006514 (0x3f37 bytes) 000496 1788471599.762211 punch from 0x0042d4 to 0x006514 (0x2241 bytes) 000497 1788471599.773914 read 0x004f98 thru 0x006514 (0x157d bytes) 000498 1788471599.782137 fallocate 0x0040f8 thru 0x00be4d (0x7d56 bytes) 000499 1788471599.790018 read 0x003eb5 thru 0x006514 (0x2660 bytes) 000500 1788471599.792040 write 0x002261 thru 0x005a84 (0x3824 bytes) 000501 1788471599.793192 punch from 0x00404f to 0x006514 (0x24c6 bytes) 000502 1788471599.811425 trunc from 0x006515 to 0x01814c (0x11c38 bytes) 000503 1788471599.831333 punch from 0x017716 to 0x0177b0 (0x9b bytes) 000504 1788471599.836145 mapwrite 0x00480c thru 0x006b6e (0x2363 bytes) 000505 1788471599.862523 fallocate 0x011fcc thru 0x01e609 (0xc63e bytes) 000506 1788471599.868572 fallocate 0x014264 thru 0x01edeb (0xab88 bytes) 000507 1788471599.874671 trunc from 0x00e4ed to 0x018bc6 (0xa6da bytes) 000508 1788471599.880471 fallocate 0x00f797 thru 0x0161e9 (0x6a53 bytes) 000509 1788471599.886525 mapwrite 0x03b507 thru 0x03ffff (0x4af9 bytes) 000510 1788471599.931824 read 0x018ec1 thru 0x01c67a (0x37ba bytes) 000512 1788471599.942902 punch from 0x036396 to 0x03efa0 (0x8c0b bytes) 000513 1788471599.948358 read 0x02712d thru 0x03184d (0xa721 bytes) 000514 1788471599.949444 trunc from 0x02b7fd to 0x030c11 (0x5415 bytes) 000515 1788471599.954023 mapwrite 0x03a64f thru 0x03ffff (0x59b1 bytes) 000516 1788471599.980504 write 0x01a03e thru 0x029aba (0xfa7d bytes) 000517 1788471599.988748 punch from 0x0289ff to 0x033f7c (0xb57e bytes) 000518 1788471600.018971 fallocate 0x004562 thru 0x0122c3 (0xdd62 bytes) 000520 1788471600.035886 trunc from 0x040000 to 0x02f5f1 (0x10a0e bytes) 000521 1788471600.066204 mapwrite 0x01bf51 thru 0x02902a (0xd0da bytes) 000522 1788471600.123372 read 0x002ff4 thru 0x003032 (0x3f bytes) 000523 1788471600.124040 read 0x026737 thru 0x02f5f1 (0x8ebb bytes) 000524 1788471600.138302 fallocate 0x039838 thru 0x03ffff (0x67c8 bytes) 000525 1788471600.158719 mapwrite 0x00a55c thru 0x011a46 (0x74eb bytes) 000526 1788471600.192076 trunc from 0x02f5f2 to 0x03bf1a (0xc929 bytes) 000527 1788471600.224994 read 0x00ffcb thru 0x01680f (0x6845 bytes) 000528 1788471600.225830 write 0x017ddd thru 0x0223af (0xa5d3 bytes) 000530 1788471600.228359 write 0x024b88 thru 0x03174d (0xcbc6 bytes) 000531 1788471600.234784 trunc from 0x03bf1b to 0x00bb77 (0x303a3 bytes) 000532 1788471600.254123 read 0x008543 thru 0x00bb77 (0x3635 bytes) 000534 1788471600.255931 mapwrite 0x0252a0 thru 0x02fdc9 (0xab2a bytes) 000535 1788471600.361255 write 0x02aaea thru 0x02c0b8 (0x15cf bytes) 000537 1788471600.362708 mapread 0x00ea4d thru 0x010611 (0x1bc5 bytes) 000538 1788471600.364098 punch from 0x0200b7 to 0x028c8a (0x8bd4 bytes) 000539 1788471600.370343 trunc from 0x020b59 to 0x026c87 (0x612f bytes) 000540 1788471600.375695 read 0x0293d4 thru 0x02dcf1 (0x491e bytes) 000541 1788471600.376530 punch from 0x02f177 to 0x02fdc9 (0xc53 bytes) 000542 1788471600.381941 mapread 0x01bdd0 thru 0x027cea (0xbf1b bytes) 000546 1788471600.394842 punch from 0x001a1e to 0x00b6c9 (0x9cac bytes) 000547 1788471600.399400 punch from 0x01fee5 to 0x02a72f (0xa84b bytes) 000548 1788471600.410800 fallocate 0x023d21 thru 0x025da1 (0x2081 bytes) 000549 1788471600.415405 punch from 0x013169 to 0x0139b8 (0x850 bytes) 000550 1788471600.423161 trunc from 0x02fdca to 0x039528 (0x975f bytes) 000552 1788471600.438935 write 0x03d215 thru 0x03ffff (0x2deb bytes) 000553 1788471600.440702 read 0x03afef thru 0x03ffff (0x5011 bytes) 000555 1788471600.462994 fallocate 0x02349b thru 0x030499 (0xcfff bytes) 000556 1788471600.468910 trunc from 0x0032ee to 0x003394 (0xa7 bytes) 000557 1788471600.473645 punch from 0x0109c2 to 0x01ba6d (0xb0ac bytes) 000558 1788471600.478681 punch from 0x014f74 to 0x02263a (0xd6c7 bytes) 000559 1788471600.483430 fallocate 0x00f85e thru 0x019295 (0x9a38 bytes) 000560 1788471600.487848 mapwrite 0x0054ed thru 0x009bfb (0x470f bytes) 000561 1788471600.525094 read 0x03c6ac thru 0x03ffff (0x3954 bytes) 000562 1788471600.525713 fallocate 0x013f68 thru 0x01dc70 (0x9d09 bytes) 000563 1788471600.536364 trunc from 0x040000 to 0x016261 (0x29d9e bytes) 000564 1788471600.572967 MIRROR_EXTEND to 2 mirrors 000565 1788471601.577174 punch from 0x015377 to 0x016261 (0xeeb bytes) 000566 1788471601.653879 MIRROR_RESYNC 2 mirrors 000567 1788471602.563099 trunc from 0x012180 to 0x016261 (0x40e2 bytes) 000568 1788471602.601455 read 0x00ca8b thru 0x016261 (0x97d7 bytes) 000570 1788471602.611792 read 0x010033 thru 0x016261 (0x622f bytes) 000571 1788471602.612797 punch from 0x006b79 to 0x00f161 (0x85e9 bytes) 000572 1788471602.618073 punch from 0x00c08b to 0x013d11 (0x7c87 bytes) 000574 1788471602.624281 read 0x00e406 thru 0x014379 (0x5f74 bytes) 000575 1788471602.634732 mapread 0x0049b9 thru 0x00538f (0x9d7 bytes) 000576 1788471602.636133 fallocate 0x007e67 thru 0x00e22c (0x63c6 bytes) 000577 1788471602.641746 read 0x007626 thru 0x007e07 (0x7e2 bytes) 000578 1788471602.642606 read 0x005c55 thru 0x010452 (0xa7fe bytes) 000579 1788471602.643423 trunc from 0x016262 to 0x027611 (0x113b0 bytes) 000580 1788471602.667858 mapread 0x00c9d8 thru 0x0139d9 (0x7002 bytes) 000581 1788471602.674336 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000582 1788471602.820146 mapwrite 0x03dc21 thru 0x03ffff (0x23df bytes) 000583 1788471602.884345 trunc from 0x040000 to 0x0053e3 (0x3ac1c bytes) 000584 1788471602.912066 punch from 0x000e20 to 0x0053e3 (0x45c4 bytes) 000585 1788471602.918884 trunc from 0x00426d to 0x0053e3 (0x1177 bytes) 000586 1788471602.924281 mapread 0x003caa thru 0x0053e3 (0x173a bytes) 000587 1788471602.941349 fallocate 0x02ec84 thru 0x0312f8 (0x2675 bytes) 000589 1788471602.949273 mapwrite 0x0275a1 thru 0x02dddc (0x683c bytes) 000590 1788471602.994709 mapread 0x00054a thru 0x00c4bb (0xbf72 bytes) 000591 1788471603.018804 write 0x02679b thru 0x0315b2 (0xae18 bytes) 000592 1788471603.029640 fallocate 0x006dff thru 0x014f36 (0xe138 bytes) 000593 1788471603.038283 trunc from 0x0315b3 to 0x027a8a (0x9b28 bytes) 000594 1788471603.060969 read 0x02564e thru 0x027a8a (0x243d bytes) 000596 1788471603.063571 write 0x037b6a thru 0x03eb29 (0x6fc0 bytes) 000597 1788471603.069272 read 0x029d8c thru 0x02a1b1 (0x426 bytes) 000598 1788471603.079865 write 0x002cc6 thru 0x009c08 (0x6f43 bytes) 000599 1788471603.081722 fallocate 0x03dd70 thru 0x03ffff (0x2290 bytes) 000600 1788471603.090399 mapread 0x0064b0 thru 0x011a23 (0xb574 bytes) 000601 1788471603.092640 trunc from 0x03eb2a to 0x0140e1 (0x2aa48 bytes) 000602 1788471603.126394 MIRROR_EXTEND to 2 mirrors 000603 1788471603.860267 punch from 0x005510 to 0x00e353 (0x8e44 bytes) 000605 1788471603.937437 trunc from 0x000db1 to 0x00b109 (0xa359 bytes) 000607 1788471603.953712 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x140e2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x140e2): Mirror 2: 0x58bc4fe0 (stale) CRC-32 checksum value for chunk [0, 0x140e2): Mirror 1: 0xded005ae Mirror 3: 0xded005ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x140e2: skipped 000608 1788471604.845981 write 0x014985 thru 0x01dc8d (0x9309 bytes) 000609 1788471604.897892 fallocate 0x03e6f3 thru 0x03f77d (0x108b bytes) 000610 1788471604.905023 write 0x039238 thru 0x03ffff (0x6dc8 bytes) 000612 1788471604.914787 trunc from 0x024f75 to 0x02b4a4 (0x6530 bytes) 000613 1788471604.920792 trunc from 0x0333b8 to 0x03ffff (0xcc48 bytes) 000614 1788471604.939436 punch from 0x017935 to 0x020ac8 (0x9194 bytes) 000615 1788471604.945164 fallocate 0x02d00f thru 0x02f643 (0x2635 bytes) 000616 1788471604.951722 read 0x001112 thru 0x0042aa (0x3199 bytes) 000617 1788471604.959873 fallocate 0x0097c9 thru 0x018bd7 (0xf40f bytes) 000618 1788471604.966717 mapread 0x03abf0 thru 0x03ffff (0x5410 bytes) 000619 1788471604.993799 punch from 0x03b435 to 0x03ffff (0x4bcb bytes) 000620 1788471604.998235 trunc from 0x040000 to 0x0164ed (0x29b12 bytes) 000621 1788471605.017018 mapwrite 0x00da0a thru 0x01a0f6 (0xc6ed bytes) 000622 1788471605.062460 trunc from 0x001f55 to 0x0068fa (0x49a6 bytes) 000623 1788471605.067938 write 0x0046d2 thru 0x00e12d (0x9a5c bytes) 000624 1788471605.074419 mapread 0x005221 thru 0x008d7d (0x3b5d bytes) 000625 1788471605.079347 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a0f7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0f7): Mirror 2: 0xd3c355a6 (stale) Mirror 3: 0xcbb01d5 (stale) CRC-32 checksum value for chunk [0, 0x1a0f7): Mirror 1: 0x9bee1b55 Mirror 4: 0x9bee1b55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0f7: skipped 000626 1788471605.922595 mapread 0x013e5d thru 0x01847e (0x4622 bytes) 000627 1788471605.982397 punch from 0x0137b3 to 0x01a0f6 (0x6944 bytes) 000628 1788471606.029308 trunc from 0x012f0a to 0x012f63 (0x5a bytes) 000629 1788471606.036622 punch from 0x004a78 to 0x009bbe (0x5147 bytes) 000630 1788471606.053504 trunc from 0x01a0f7 to 0x0191c7 (0xf2f bytes) 000631 1788471606.123678 trunc from 0x0191c8 to 0x02ee51 (0x15c8a bytes) 000632 1788471606.139075 mapread 0x001778 thru 0x00f281 (0xdb0a bytes) 000633 1788471606.150288 read 0x017401 thru 0x02677d (0xf37d bytes) 000636 1788471606.160224 punch from 0x020247 to 0x022a4b (0x2805 bytes) 000638 1788471606.165851 punch from 0x00d655 to 0x00eb84 (0x1530 bytes) 000639 1788471606.174246 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee52): Mirror 3: 0xe077608f (stale) Mirror 4: 0xc226d545 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee52): Mirror 3: 0xe077608f (stale) Mirror 4: 0xc226d545 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee52: skipped 000640 1788471606.562252 read 0x001b88 thru 0x00f0a4 (0xd51d bytes) 000641 1788471606.579218 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee52): Mirror 3: 0xe077608f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee52): Mirror 3: 0xe077608f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee52: skipped 000642 1788471606.880231 write 0x039643 thru 0x03c6fa (0x30b8 bytes) 000643 1788471606.885346 trunc from 0x03c6fb to 0x01056a (0x2c190 bytes) 000644 1788471606.908219 mapread 0x001496 thru 0x00b68f (0xa1fa bytes) 000645 1788471606.925804 mapwrite 0x00f1ee thru 0x01217e (0x2f91 bytes) 000647 1788471606.961233 punch from 0x005b91 to 0x01217e (0xc5ee bytes) 000648 1788471606.967266 mapwrite 0x03c22c thru 0x03ffff (0x3dd4 bytes) 000649 1788471607.006825 read 0x023108 thru 0x02b4cc (0x83c5 bytes) 000650 1788471607.016051 mapread 0x02901c thru 0x038417 (0xf3fc bytes) 000651 1788471607.024874 mapwrite 0x02b5a7 thru 0x03877c (0xd1d6 bytes) 000652 1788471607.047086 fallocate 0x03521d thru 0x03d986 (0x876a bytes) 000654 1788471607.052723 punch from 0x00bf89 to 0x0179fe (0xba76 bytes) 000656 1788471607.058715 trunc from 0x040000 to 0x00a1d0 (0x35e2f bytes) 000657 1788471607.077781 punch from 0x00a114 to 0x00a1d0 (0xbd bytes) 000658 1788471607.082772 punch from 0x002daf to 0x00a1d0 (0x7422 bytes) 000659 1788471607.091856 MIRROR_RESYNC 2 mirrors 000660 1788471607.536239 trunc from 0x00a1d1 to 0x0297cf (0x1f5ff bytes) 000662 1788471607.596208 punch from 0x01ac62 to 0x029328 (0xe6c7 bytes) 000663 1788471607.605181 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000664 1788471607.746990 MIRROR_EXTEND to 2 mirrors 000666 1788471608.411183 trunc from 0x0297d0 to 0x03ba62 (0x12293 bytes) 000667 1788471608.463416 trunc from 0x01387b to 0x017b7a (0x4300 bytes) 000668 1788471608.469817 mapread 0x02fa8d thru 0x037a87 (0x7ffb bytes) 000669 1788471608.489452 mapread 0x01f08f thru 0x028b26 (0x9a98 bytes) 000670 1788471608.504803 mapread 0x02e37f thru 0x034bd7 (0x6859 bytes) 000671 1788471608.506328 mapwrite 0x00166b thru 0x0066df (0x5075 bytes) 000672 1788471608.538679 trunc from 0x0178e9 to 0x018601 (0xd19 bytes) 000673 1788471608.543811 mapwrite 0x01705c thru 0x01e7de (0x7783 bytes) 000674 1788471608.578365 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 2: 0x56a32ebe (stale) CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 1: 0x76a35102 Mirror 3: 0x76a35102 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba63: skipped 000675 1788471609.394990 fallocate 0x005e1e thru 0x00f16c (0x934f bytes) 000676 1788471609.444335 trunc from 0x03ba63 to 0x0088fe (0x33164 bytes) 000677 1788471609.509755 punch from 0x0065d2 to 0x0088fe (0x232d bytes) 000678 1788471609.517825 read 0x00466b thru 0x005064 (0x9fa bytes) 000679 1788471609.529935 punch from 0x0052a8 to 0x0088fe (0x3657 bytes) 000680 1788471609.536954 trunc from 0x0088ff to 0x0274b7 (0x1ebb9 bytes) 000681 1788471609.556610 mapwrite 0x01fc2a thru 0x02d8ec (0xdcc3 bytes) 000682 1788471609.644901 trunc from 0x0067e9 to 0x00d6ac (0x6ec4 bytes) 000683 1788471609.662327 trunc from 0x02d8ed to 0x01956f (0x1437d bytes) 000684 1788471609.683107 mapread 0x01912a thru 0x01956f (0x446 bytes) 000685 1788471609.702079 trunc from 0x019570 to 0x013600 (0x5f6f bytes) 000686 1788471609.727708 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13601): Mirror 3: 0x5592d905 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13601): Mirror 3: 0x5592d905 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13601: skipped 000687 1788471610.076013 punch from 0x009088 to 0x013600 (0xa579 bytes) 000688 1788471610.083455 punch from 0x0091fa to 0x013600 (0xa407 bytes) 000689 1788471610.090418 mapwrite 0x02740b thru 0x02d67e (0x6274 bytes) 000690 1788471610.150443 read 0x00e05d thru 0x01b898 (0xd83c bytes) 000691 1788471610.157615 mapwrite 0x0256d1 thru 0x03330e (0xdc3e bytes) 000694 1788471610.206462 trunc from 0x028bb4 to 0x02e84d (0x5c9a bytes) 000695 1788471610.212746 punch from 0x02dc6c to 0x03330e (0x56a3 bytes) 000696 1788471610.218718 fallocate 0x009b77 thru 0x00b380 (0x180a bytes) 000697 1788471610.224781 write 0x02167d thru 0x02e783 (0xd107 bytes) 000698 1788471610.234569 trunc from 0x01c493 to 0x02919b (0xcd09 bytes) 000699 1788471610.248273 mapread 0x003ca6 thru 0x00aa95 (0x6df0 bytes) 000700 1788471610.258419 punch from 0x01fda6 to 0x026a7c (0x6cd7 bytes) 000701 1788471610.263454 fallocate 0x025db8 thru 0x0294a2 (0x36eb bytes) 000703 1788471610.274418 trunc from 0x0061c1 to 0x0136d8 (0xd518 bytes) 000704 1788471610.284158 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3330f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3330f): Mirror 3: 0x9644f2fe (stale) CRC-32 checksum value for chunk [0, 0x3330f): Mirror 1: 0xd988946b Mirror 4: 0xd988946b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3330f: skipped 000705 1788471611.169409 MIRROR_RESYNC 3 mirrors 000706 1788471611.984660 trunc from 0x03330f to 0x00b5d0 (0x27d3e bytes) 000707 1788471612.054473 write 0x00b359 thru 0x019bfe (0xe8a6 bytes) 000708 1788471612.061421 punch from 0x014678 to 0x019bfe (0x5587 bytes) 000709 1788471612.092146 read 0x00c019 thru 0x00cd24 (0xd0c bytes) 000710 1788471612.093193 trunc from 0x000618 to 0x002c78 (0x2661 bytes) 000711 1788471612.104920 read 0x017e12 thru 0x019bfe (0x1ded bytes) 000712 1788471612.127113 MIRROR_RESYNC 3 mirrors 000714 1788471613.081020 trunc from 0x008bed to 0x00e2d3 (0x56e7 bytes) 000715 1788471613.119300 read 0x008a98 thru 0x00961d (0xb86 bytes) 000716 1788471613.127680 read 0x0008bc thru 0x00809e (0x77e3 bytes) 000717 1788471613.141614 mapread 0x01506e thru 0x019bfe (0x4b91 bytes) 000718 1788471613.142855 punch from 0x003694 to 0x01103d (0xd9aa bytes) 000719 1788471613.150656 trunc from 0x0013f8 to 0x008438 (0x7041 bytes) 000720 1788471613.156766 read 0x0007cb thru 0x001cee (0x1524 bytes) 000721 1788471613.173965 read 0x01270c thru 0x014385 (0x1c7a bytes) 000722 1788471613.175570 write 0x026110 thru 0x02a3fb (0x42ec bytes) 000723 1788471613.178069 trunc from 0x02a3fc to 0x00c1be (0x1e23d bytes) 000724 1788471613.227473 read 0x003531 thru 0x00c1be (0x8c8e bytes) 000725 1788471613.228430 read 0x005c29 thru 0x00c1be (0x6596 bytes) 000726 1788471613.230544 write 0x01c364 thru 0x01c46a (0x107 bytes) 000727 1788471613.232251 fallocate 0x018456 thru 0x0207c7 (0x8372 bytes) 000728 1788471613.237862 mapread 0x01c3ec thru 0x0207c7 (0x43dc bytes) 000729 1788471613.247585 trunc from 0x0207c8 to 0x013b99 (0xcc2e bytes) 000730 1788471613.263381 trunc from 0x013b9a to 0x025466 (0x118cd bytes) 000731 1788471613.283260 mapwrite 0x03c421 thru 0x03ec59 (0x2839 bytes) 000732 1788471613.320316 fallocate 0x01559c thru 0x024911 (0xf376 bytes) 000733 1788471613.325704 write 0x03e4ea thru 0x03ffff (0x1b16 bytes) 000734 1788471613.327415 fallocate 0x0073f3 thru 0x01165b (0xa269 bytes) 000735 1788471613.332456 trunc from 0x016b08 to 0x01af1d (0x4416 bytes) 000736 1788471613.337373 trunc from 0x040000 to 0x013804 (0x2c7fb bytes) 000737 1788471613.355604 read 0x00048c thru 0x00429e (0x3e13 bytes) 000738 1788471613.356208 mapwrite 0x025c75 thru 0x029cf5 (0x4081 bytes) 000739 1788471613.417639 trunc from 0x029cf6 to 0x028d0f (0xfe6 bytes) 000740 1788471613.450352 MIRROR_RESYNC 3 mirrors 000741 1788471614.195013 write 0x02257f thru 0x02e82e (0xc2b0 bytes) 000742 1788471614.246041 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e82f): Mirror 3: 0x1b322924 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e82f): Mirror 3: 0x1b322924 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e82f: skipped 000743 1788471614.522135 trunc from 0x01833f to 0x021915 (0x95d7 bytes) 000744 1788471614.527349 fallocate 0x039a05 thru 0x03d2c3 (0x38bf bytes) 000745 1788471614.532722 fallocate 0x004d39 thru 0x00958d (0x4855 bytes) 000746 1788471614.538315 read 0x01101e thru 0x016475 (0x5458 bytes) 000747 1788471614.548143 mapwrite 0x02cf2b thru 0x037b47 (0xac1d bytes) 000748 1788471614.588034 trunc from 0x039adc to 0x03a855 (0xd7a bytes) 000749 1788471614.592518 mapread 0x03b3d4 thru 0x03d2c3 (0x1ef0 bytes) 000750 1788471614.597374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000751 1788471614.748108 mapread 0x027e99 thru 0x02db2f (0x5c97 bytes) 000753 1788471614.777968 fallocate 0x01020c thru 0x01b63d (0xb432 bytes) 000754 1788471614.792062 read 0x00ca0b thru 0x019b11 (0xd107 bytes) 000755 1788471614.809546 write 0x00ab3c thru 0x018a31 (0xdef6 bytes) 000756 1788471614.823196 mapread 0x01d621 thru 0x02256b (0x4f4b bytes) 000757 1788471614.832417 mapwrite 0x02f8da thru 0x03d1f6 (0xd91d bytes) 000761 1788471614.929660 punch from 0x029d51 to 0x039cf4 (0xffa4 bytes) 000763 1788471614.936227 fallocate 0x00c6e7 thru 0x01b346 (0xec60 bytes) 000764 1788471614.951601 MIRROR_EXTEND to 2 mirrors 000765 1788471615.793367 write 0x0034f2 thru 0x00c0f1 (0x8c00 bytes) 000766 1788471615.845829 mapread 0x016479 thru 0x026105 (0xfc8d bytes) 000767 1788471615.877019 read 0x02259f thru 0x02f4ea (0xcf4c bytes) 000768 1788471615.889179 mapwrite 0x001ac2 thru 0x00c712 (0xac51 bytes) 000769 1788471615.933047 mapread 0x012146 thru 0x01afcc (0x8e87 bytes) 000770 1788471615.933964 mapwrite 0x023c53 thru 0x025415 (0x17c3 bytes) 000771 1788471615.977770 punch from 0x027d31 to 0x02d3ff (0x56cf bytes) 000772 1788471615.998642 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000773 1788471616.237371 mapwrite 0x01da73 thru 0x01e51b (0xaa9 bytes) 000774 1788471616.268942 write 0x0172b2 thru 0x02231b (0xb06a bytes) 000775 1788471616.309075 MIRROR_EXTEND to 2 mirrors 000776 1788471617.426625 read 0x029b76 thru 0x036094 (0xc51f bytes) 000777 1788471617.501234 MIRROR_RESYNC 2 mirrors 000778 1788471617.983282 MIRROR_RESYNC 2 mirrors 000779 1788471618.366300 trunc from 0x035b27 to 0x03d2c3 (0x779d bytes) 000780 1788471618.404366 fallocate 0x00693a thru 0x006afb (0x1c2 bytes) 000781 1788471618.409221 fallocate 0x03d088 thru 0x03ffff (0x2f78 bytes) 000782 1788471618.414644 mapread 0x003610 thru 0x010b1e (0xd50f bytes) 000783 1788471618.445333 mapwrite 0x0312bd thru 0x03ee21 (0xdb65 bytes) 000784 1788471618.495359 write 0x036c99 thru 0x03ffff (0x9367 bytes) 000785 1788471618.502805 write 0x023e6d thru 0x02bdbb (0x7f4f bytes) 000786 1788471618.509609 trunc from 0x03147a to 0x03de5b (0xc9e2 bytes) 000787 1788471618.530295 mapread 0x027eca thru 0x034c30 (0xcd67 bytes) 000788 1788471618.539611 mapwrite 0x034833 thru 0x039a59 (0x5227 bytes) 000789 1788471618.555644 fallocate 0x0294c8 thru 0x0368d4 (0xd40d bytes) 000792 1788471618.561657 fallocate 0x00c4d8 thru 0x00c8d0 (0x3f9 bytes) 000793 1788471618.566830 mapwrite 0x034d5a thru 0x039663 (0x490a bytes) 000794 1788471618.583881 mapread 0x00793a thru 0x008d34 (0x13fb bytes) 000795 1788471618.584798 mapwrite 0x019b92 thru 0x01ee9e (0x530d bytes) 000796 1788471618.606731 mapread 0x0104cc thru 0x01eabf (0xe5f4 bytes) 000797 1788471618.612464 trunc from 0x040000 to 0x032ed1 (0xd12e bytes) 000798 1788471618.637444 write 0x00d02a thru 0x00fdcc (0x2da3 bytes) 000799 1788471618.638609 read 0x018581 thru 0x023253 (0xacd3 bytes) 000800 1788471618.645995 read 0x01b407 thru 0x0278fd (0xc4f7 bytes) 000801 1788471618.646723 fallocate 0x032839 thru 0x034987 (0x214f bytes) 000802 1788471618.652000 mapread 0x0313a3 thru 0x032ed1 (0x1b2f bytes) 000803 1788471618.652923 trunc from 0x032ed2 to 0x00556f (0x2d962 bytes) 000804 1788471618.675267 mapwrite 0x01ec3e thru 0x02b084 (0xc447 bytes) 000805 1788471618.738086 fallocate 0x00c4fb thru 0x018d5f (0xc865 bytes) 000806 1788471618.743675 trunc from 0x02b085 to 0x033575 (0x84f1 bytes) 000807 1788471618.774180 mapwrite 0x032a8d thru 0x03c90b (0x9e7f bytes) 000808 1788471618.839823 fallocate 0x0293e3 thru 0x0312ad (0x7ecb bytes) 000809 1788471618.846942 trunc from 0x03c90c to 0x032721 (0xa1ea bytes) 000810 1788471618.862145 write 0x01372b thru 0x0211a4 (0xda7a bytes) 000811 1788471618.865142 punch from 0x02836e to 0x032721 (0xa3b4 bytes) 000812 1788471618.871094 read 0x006a73 thru 0x006b6e (0xfc bytes) 000813 1788471618.872395 trunc from 0x00f73d to 0x018d47 (0x960b bytes) 000814 1788471618.886023 trunc from 0x032722 to 0x0348f2 (0x21d1 bytes) 000815 1788471618.899620 fallocate 0x00d8a7 thru 0x01327d (0x59d7 bytes) 000816 1788471618.904393 fallocate 0x03e631 thru 0x03f4ec (0xebc bytes) 000817 1788471618.910070 read 0x00ecb3 thru 0x01ba43 (0xcd91 bytes) 000819 1788471618.927437 write 0x03828e thru 0x03ffff (0x7d72 bytes) 000820 1788471618.934567 read 0x0358c5 thru 0x03cebf (0x75fb bytes) 000821 1788471618.942751 trunc from 0x040000 to 0x02e4f1 (0x11b0e bytes) 000822 1788471618.984253 write 0x007f01 thru 0x017168 (0xf268 bytes) 000823 1788471618.997794 read 0x02210a thru 0x02e4f1 (0xc3e8 bytes) 000824 1788471618.998829 punch from 0x008eb4 to 0x014e24 (0xbf71 bytes) 000825 1788471619.047409 trunc from 0x02b924 to 0x02e4f1 (0x2bce bytes) 000826 1788471619.063835 MIRROR_RESYNC 2 mirrors 000827 1788471620.054363 trunc from 0x02e4f2 to 0x03541f (0x6f2e bytes) 000828 1788471620.115429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000829 1788471620.255081 punch from 0x02568b to 0x03232d (0xcca3 bytes) 000830 1788471620.263138 mapread 0x02b997 thru 0x03541f (0x9a89 bytes) 000831 1788471620.290705 trunc from 0x035420 to 0x00305c (0x323c3 bytes) 000832 1788471620.312268 mapread 0x000dfb thru 0x0014ab (0x6b1 bytes) 000835 1788471620.326956 mapread 0x000176 thru 0x00305c (0x2ee7 bytes) 000836 1788471620.328050 read 0x002c72 thru 0x00305c (0x3eb bytes) 000837 1788471620.329117 trunc from 0x00305d to 0x01adee (0x17d92 bytes) 000838 1788471620.345145 write 0x01a1b5 thru 0x026f3a (0xcd86 bytes) 000839 1788471620.352077 read 0x022d2e thru 0x026f3a (0x420d bytes) 000840 1788471620.353251 trunc from 0x026f3b to 0x01b717 (0xb823 bytes) 000842 1788471620.371279 mapread 0x0026dc thru 0x0113f7 (0xed1c bytes) 000843 1788471620.390756 mapread 0x008840 thru 0x0165f0 (0xddb1 bytes) 000845 1788471620.392718 mapread 0x013ae9 thru 0x01b206 (0x771e bytes) 000846 1788471620.394249 fallocate 0x03a17c thru 0x03ffff (0x5e84 bytes) 000847 1788471620.400023 punch from 0x02cb0c to 0x02d45e (0x953 bytes) 000848 1788471620.404430 read 0x03893e thru 0x039d9c (0x145f bytes) 000849 1788471620.430029 trunc from 0x028aae to 0x02d497 (0x49ea bytes) 000850 1788471620.435307 punch from 0x019eb9 to 0x024edd (0xb025 bytes) 000851 1788471620.444774 trunc from 0x040000 to 0x007971 (0x3868e bytes) 000852 1788471620.463186 read 0x0045ad thru 0x007971 (0x33c5 bytes) 000853 1788471620.463841 read 0x000a55 thru 0x007971 (0x6f1d bytes) 000854 1788471620.464338 write 0x0300e7 thru 0x03d7b3 (0xd6cd bytes) 000855 1788471620.468192 read 0x0227ed thru 0x02c1d1 (0x99e5 bytes) 000856 1788471620.478194 trunc from 0x0185fc to 0x02592b (0xd330 bytes) 000857 1788471620.484067 trunc from 0x02deca to 0x039f7e (0xc0b5 bytes) 000859 1788471620.509755 trunc from 0x02d143 to 0x03c359 (0xf217 bytes) 000860 1788471620.517096 trunc from 0x03cba3 to 0x03d7b3 (0xc11 bytes) 000861 1788471620.521971 mapwrite 0x00fe45 thru 0x01e83a (0xe9f6 bytes) 000862 1788471620.593170 write 0x03cd77 thru 0x03ffff (0x3289 bytes) 000863 1788471620.599523 mapwrite 0x03d467 thru 0x03ffff (0x2b99 bytes) 000864 1788471620.614099 write 0x024db6 thru 0x032a2e (0xdc79 bytes) 000865 1788471620.626654 fallocate 0x0255af thru 0x031e11 (0xc863 bytes) 000866 1788471620.632206 fallocate 0x023acf thru 0x02a3a3 (0x68d5 bytes) 000867 1788471620.639102 write 0x008f7b thru 0x018c90 (0xfd16 bytes) 000868 1788471620.643810 trunc from 0x03cd53 to 0x03ffff (0x32ad bytes) 000870 1788471620.656358 write 0x00bde1 thru 0x00ed7b (0x2f9b bytes) 000871 1788471620.658109 trunc from 0x0284ef to 0x036139 (0xdc4b bytes) 000873 1788471620.679139 read 0x02da74 thru 0x035dd9 (0x8366 bytes) 000874 1788471620.694151 read 0x03164e thru 0x035afb (0x44ae bytes) 000875 1788471620.695571 write 0x02f768 thru 0x03e5d8 (0xee71 bytes) 000876 1788471620.698183 mapwrite 0x0124c6 thru 0x01a5b5 (0x80f0 bytes) 000878 1788471620.720279 write 0x0067bc thru 0x014885 (0xe0ca bytes) 000879 1788471620.721878 read 0x010d2d thru 0x0201bf (0xf493 bytes) 000880 1788471620.723211 fallocate 0x01ccbc thru 0x025947 (0x8c8c bytes) 000881 1788471620.729706 trunc from 0x040000 to 0x02ddcf (0x12230 bytes) 000882 1788471620.751526 mapread 0x01ab49 thru 0x0215b8 (0x6a70 bytes) 000885 1788471620.755064 punch from 0x011a10 to 0x016767 (0x4d58 bytes) 000886 1788471620.770039 write 0x0349f1 thru 0x03a5bb (0x5bcb bytes) 000887 1788471620.772015 trunc from 0x039fd4 to 0x03a5bb (0x5e8 bytes) 000888 1788471620.798408 MIRROR_EXTEND to 2 mirrors 000889 1788471621.562427 fallocate 0x01d41c thru 0x029c34 (0xc819 bytes) 000890 1788471621.627226 mapread 0x01f6e1 thru 0x01fe70 (0x790 bytes) 000891 1788471621.634298 fallocate 0x03608f thru 0x03ebc1 (0x8b33 bytes) 000892 1788471621.640687 read 0x015bb2 thru 0x02486a (0xecb9 bytes) 000894 1788471621.654215 fallocate 0x0389d6 thru 0x03ffff (0x762a bytes) 000895 1788471621.659909 trunc from 0x03a5bc to 0x01f298 (0x1b323 bytes) 000896 1788471621.697520 mapread 0x008c1c thru 0x009f35 (0x131a bytes) 000897 1788471621.713807 mapwrite 0x0301fc thru 0x0319c2 (0x17c7 bytes) 000898 1788471621.769215 write 0x03683f thru 0x036cd9 (0x49b bytes) 000899 1788471621.773464 trunc from 0x002c0d to 0x0038b0 (0xca4 bytes) 000900 1788471621.785134 trunc from 0x018299 to 0x020637 (0x839f bytes) 000901 1788471621.793258 read 0x016498 thru 0x01d9f9 (0x7562 bytes) 000902 1788471621.850940 read 0x003c7d thru 0x00e2f8 (0xa67c bytes) 000903 1788471621.866237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36cda): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cda): Mirror 2: 0xe51139c2 (stale) CRC-32 checksum value for chunk [0, 0x36cda): Mirror 1: 0x75f6b78d Mirror 3: 0x75f6b78d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cda: skipped 000904 1788471623.285898 fallocate 0x02f5a6 thru 0x032b7b (0x35d6 bytes) 000905 1788471623.390773 fallocate 0x016b45 thru 0x01dbfd (0x70b9 bytes) 000906 1788471623.402576 mapread 0x013442 thru 0x01b381 (0x7f40 bytes) 000907 1788471623.438208 trunc from 0x032ae3 to 0x036694 (0x3bb2 bytes) 000908 1788471623.444457 trunc from 0x036cda to 0x00a190 (0x2cb49 bytes) 000909 1788471623.470896 mapread 0x008ada thru 0x009730 (0xc57 bytes) 000910 1788471623.479263 mapread 0x001ffd thru 0x00a190 (0x8194 bytes) 000912 1788471623.483271 write 0x01dd7d thru 0x0220e3 (0x4367 bytes) 000913 1788471623.486122 write 0x03b9df thru 0x03ffff (0x4621 bytes) 000914 1788471623.487454 trunc from 0x040000 to 0x0273f4 (0x18c0b bytes) 000915 1788471623.508365 trunc from 0x0273f5 to 0x03f8b0 (0x184bc bytes) 000916 1788471623.526069 mapread 0x01a493 thru 0x01adb8 (0x926 bytes) 000917 1788471623.539908 punch from 0x03e6f7 to 0x03f8b0 (0x11ba bytes) 000918 1788471623.544239 trunc from 0x0078ce to 0x0081a6 (0x8d9 bytes) 000919 1788471623.555055 trunc from 0x018eb4 to 0x0211e1 (0x832e bytes) 000920 1788471623.578725 trunc from 0x03f8b1 to 0x01c8e9 (0x22fc7 bytes) 000922 1788471623.605000 mapread 0x00bd8c thru 0x015f9b (0xa210 bytes) 000923 1788471623.620198 MIRROR_RESYNC 3 mirrors 000924 1788471624.676196 fallocate 0x028ef3 thru 0x02b374 (0x2482 bytes) 000925 1788471624.716191 write 0x00ce6a thru 0x014314 (0x74ab bytes) 000927 1788471624.726098 trunc from 0x01f0f6 to 0x0273e6 (0x82f1 bytes) 000928 1788471624.730940 fallocate 0x01f9b5 thru 0x02a4b8 (0xab04 bytes) 000929 1788471624.735509 write 0x031ffb thru 0x03dc8a (0xbc90 bytes) 000930 1788471624.738093 write 0x006bb0 thru 0x00f0a7 (0x84f8 bytes) 000931 1788471624.744011 write 0x03dd7c thru 0x03ffff (0x2284 bytes) 000933 1788471624.746281 trunc from 0x040000 to 0x021a51 (0x1e5ae bytes) 000934 1788471624.767283 trunc from 0x01c35e to 0x021a51 (0x56f4 bytes) 000935 1788471624.772417 trunc from 0x021a52 to 0x035a5b (0x1400a bytes) 000936 1788471624.787236 mapwrite 0x033f8e thru 0x03ffff (0xc072 bytes) 000937 1788471624.857912 fallocate 0x036ef9 thru 0x03ffff (0x9107 bytes) 000938 1788471624.865414 punch from 0x004462 to 0x00d776 (0x9315 bytes) 000939 1788471624.870995 fallocate 0x034ae3 thru 0x03fbee (0xb10c bytes) 000941 1788471624.876388 fallocate 0x004f4b thru 0x00610f (0x11c5 bytes) 000942 1788471624.887447 MIRROR_RESYNC 3 mirrors 000943 1788471625.633078 fallocate 0x0242af thru 0x02e742 (0xa494 bytes) 000944 1788471625.667206 fallocate 0x030f0c thru 0x03ffff (0xf0f4 bytes) 000945 1788471625.673257 read 0x00cdde thru 0x018032 (0xb255 bytes) 000947 1788471625.683409 trunc from 0x004ea8 to 0x0078ef (0x2a48 bytes) 000948 1788471625.687977 write 0x027ae1 thru 0x02d694 (0x5bb4 bytes) 000949 1788471625.696881 mapwrite 0x024d4c thru 0x028bb1 (0x3e66 bytes) 000952 1788471625.745407 mapread 0x0288ee thru 0x0378cd (0xefe0 bytes) 000954 1788471625.756133 trunc from 0x01abb7 to 0x02308e (0x84d8 bytes) 000955 1788471625.760474 trunc from 0x0297dc to 0x0324fb (0x8d20 bytes) 000957 1788471625.765525 mapwrite 0x0289c8 thru 0x0338c6 (0xaeff bytes) 000958 1788471625.812225 mapwrite 0x01cc8f thru 0x01ce8e (0x200 bytes) 000959 1788471625.837558 mapwrite 0x035b4f thru 0x03ffff (0xa4b1 bytes) 000960 1788471625.890442 trunc from 0x01b475 to 0x020c13 (0x579f bytes) 000961 1788471625.899933 mapread 0x002d94 thru 0x00aa1c (0x7c89 bytes) 000962 1788471625.919541 punch from 0x004f8e to 0x011783 (0xc7f6 bytes) 000963 1788471625.948110 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1788471626.306937 trunc from 0x040000 to 0x01ec4c (0x213b3 bytes) 000965 1788471626.327153 mapwrite 0x02f01f thru 0x03bb48 (0xcb2a bytes) 000966 1788471626.400201 write 0x00f6a9 thru 0x0193f9 (0x9d51 bytes) 000967 1788471626.412119 punch from 0x02f9d7 to 0x036f42 (0x756c bytes) 000968 1788471626.418368 fallocate 0x0173cf thru 0x0212cb (0x9efd bytes) 000969 1788471626.424576 write 0x021caa thru 0x031798 (0xfaef bytes) 000970 1788471626.443045 mapread 0x029785 thru 0x030012 (0x688e bytes) 000973 1788471626.446836 mapwrite 0x01954f thru 0x01cbb3 (0x3665 bytes) 000974 1788471626.486331 mapwrite 0x035ae8 thru 0x03ffff (0xa518 bytes) 000975 1788471626.537216 trunc from 0x040000 to 0x025701 (0x1a8fe bytes) 000976 1788471626.555100 write 0x02b461 thru 0x02e809 (0x33a9 bytes) 000978 1788471626.558084 mapwrite 0x037aa2 thru 0x03e2cc (0x682b bytes) 000979 1788471626.598036 fallocate 0x018922 thru 0x01ae75 (0x2554 bytes) 000980 1788471626.603112 mapread 0x00ccbc thru 0x01ca95 (0xfdda bytes) 000981 1788471626.611369 mapread 0x03861e thru 0x03e2c6 (0x5ca9 bytes) 000982 1788471626.612793 fallocate 0x0222b7 thru 0x027398 (0x50e2 bytes) 000983 1788471626.618033 fallocate 0x0396bc thru 0x03ea44 (0x5389 bytes) 000984 1788471626.623232 mapwrite 0x0331b2 thru 0x03ffff (0xce4e bytes) 000985 1788471626.660919 fallocate 0x02ae14 thru 0x02f11a (0x4307 bytes) 000987 1788471626.668001 trunc from 0x0294da to 0x02c6dd (0x3204 bytes) 000988 1788471626.673327 write 0x014298 thru 0x01a2d4 (0x603d bytes) 000989 1788471626.675142 write 0x03e282 thru 0x03ffff (0x1d7e bytes) 000990 1788471626.676771 write 0x03b7ce thru 0x03d6f5 (0x1f28 bytes) 000991 1788471626.681847 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66011c6a Mirror 4: 0x66011c6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1788471627.705987 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1788471627.987427 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b70a45 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66011c6a Mirror 4: 0x66011c6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000994 1788471629.396178 trunc from 0x040000 to 0x035d34 (0xa2cb bytes) 000995 1788471629.489380 mapwrite 0x02fea3 thru 0x03a2a7 (0xa405 bytes) 000996 1788471629.634217 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a2a8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2a8): Mirror 3: 0xf6af3c75 (stale) Mirror 4: 0xea6bd7ed (stale) CRC-32 checksum value for chunk [0, 0x3a2a8): Mirror 1: 0x8fb9cf0b Mirror 5: 0x8fb9cf0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2a8: skipped 000997 1788471631.575361 MIRROR_RESYNC 4 mirrors 000998 1788471633.235847 mapwrite 0x02e471 thru 0x03e0ff (0xfc8f bytes) 000999 1788471633.404156 read 0x035a55 thru 0x03e0ff (0x86ab bytes) 001000 1788471633.407157 trunc from 0x00afd4 to 0x0186f4 (0xd721 bytes) All operations completed A-OK! PASS 70a (68s) == sanity-flr test 71: check mirror extend parent fid ==== 17:40:40 (1788471640) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1bb:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 17:40:51 (1788471651) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280965 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125581 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 17:41:00 (1788471660) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5803360 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6187360 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6823168 .. failed Extending file size to 6792480 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8148672 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6988224 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9213760 .. done Extending file size to 8244704 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8697472 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8707776 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61281 61283 61284 61286 61287 done 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:42:20 (1788471740) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 5624704 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2588800 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8406720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8219456 .. Extending file size to 8704192 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62839 62841 62842 62844 62845 done 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:43:41 (1788471821) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1269364 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24496 3582524 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17280 3589740 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41776 7172264 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1940 1269364 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24496 3582524 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17280 3589740 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41776 7172264 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110574 s, 19.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225179 s, 9.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6b45848 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:44:03 (1788471843) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174347 s, 12.0 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 17:44:13 (1788471853) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ca:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x301:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2dc:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:44:21 (1788471861) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150845 s, 13.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x42:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x42:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x43:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:44:30 (1788471870) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112796 s, 18.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x44:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x44:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x45:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:44:38 (1788471878) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122729 s, 17.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:45:12 (1788471912) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893129 s, 11.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x45:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x45:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x46:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:45:58 (1788471958) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144488 s, 14.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:46:53 (1788472013) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg450-server: Pool lustre.test_204f created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161079 s, 13.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg450-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:48:14 (1788472094) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:48:23 (1788472103) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:48:32 (1788472112) oleg450-server: Pool lustre.test_206 created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg450-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:49:03 (1788472143) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189544 s, 5.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894685 s, 11.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1addc356 Mirror 3: 0x1addc356 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:49:18 (1788472158) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:49:22 (1788472162) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:49:26 (1788472166) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.11381 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:50:03 (1788472203) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0254218 s, 322 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:50:09 (1788472209) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466371 s, 22.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e9 lmm_fid: [0x200000405:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e9 lmm_fid: [0x200000405:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611625 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 17:50:11.000000000 -0400 Modify: 2026-09-03 17:50:11.000000000 -0400 Change: 2026-09-03 17:50:12.000000000 -0400 Birth: 2026-09-03 17:50:11.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:50:17 (1788472217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680899 s, 15.4 MB/s oleg450-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg450-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg450-server: O/280000401/d12/2ec: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:50:42 (1788472242) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.273426 s, 150 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0302101 s, 136 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 3 17:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ee:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 3 17:50 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:50:53 (1788472253) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:51:01 (1788472261) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1951 sec =========== 17:51:11 (1788472271) === sanity-flr: start cleanup 17:51:13 (1788472273) === === sanity-flr: finish cleanup 17:51:23 (1788472283) ===