-----============= acceptance-small: sanity-flr ============----- Wed Sep 2 10:54:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg357-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:54:26 (1788360866) === oleg357-client.virtnet: executing check_config_client /mnt/lustre oleg357-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg357-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff916848365800.idle_timeout=debug osc.lustre-OST0001-osc-ffff916848365800.idle_timeout=debug disable quota as required oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:54:54 (1788360894) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7588] == sanity-flr test 0a: lfs mirror create with -N option == 10:54:55 (1788360895) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:55:09 (1788360909) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:55:11 (1788360911) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:55:14 (1788360914) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:55:36 (1788360936) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:55:39 (1788360939) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:55:42 (1788360942) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff916848365800.stats=clear osc.lustre-OST0001-osc-ffff916848365800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:55:49 (1788360949) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:55:59 (1788360959) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:56:07 (1788360967) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:56:10 (1788360970) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:56:18 (1788360978) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:56:27 (1788360987) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:56:33 (1788360993) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:56:36 (1788360996) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:56:43 (1788361003) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:56:52 (1788361012) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:56:58 (1788361018) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.751177 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161286 s, 6.5 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:57:07 (1788361027) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.327495 s, 38.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:57:14 (1788361034) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:57:22 (1788361042) Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (52s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:58:13 (1788361093) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351147 s, 29.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (28s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:58:41 (1788361121) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (59s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:59:40 (1788361180) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.88449 s, 13.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 2 10:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.922s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.034s user 0m0.001s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg357-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg357-server oleg357-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.917s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg357-server: oleg357-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg357-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff916848365800.ost_server_uuid in FULL state after 0 sec PASS 33b (69s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:00:49 (1788361249) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:00:51 (1788361251) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:00:54 (1788361254) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 11:00:55 (1788361255) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 11:01:00 (1788361260) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.395898 s, 31.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:01:10 (1788361270) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333746 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293167 s, 35.8 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:01:15 (1788361275) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:01:20 (1788361280) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010032 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00264884 s, 387 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103531 s, 989 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000997371 s, 1.0 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 11:01:28 (1788361288) create test files with size 15758234 .. af8c32b6b46aa7bd438517089e575b79 - 6815ce6df3eb2966c778e4df2dac1e25 - a00de833ca60fed7c07a9f0388f1bab0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre PASS 37 (15s) == sanity-flr test 38: resync ============================ 11:01:43 (1788361303) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:01:52 (1788361312) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:01:57 (1788361317) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129292 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243501 s, 43.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 11:02:02 (1788361322) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104649 s, 20.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.254453 s, 16.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186189 s, 11.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 11:02:13 (1788361333) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 11:02:15 (1788361335) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:02:17 (1788361337) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325217 s, 32.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 11:02:23 (1788361343) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:02:25 (1788361345) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:02:32 (1788361352) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.175882 s, 59.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:02:40 (1788361360) llite.lustre-ffff916870486800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.193461 s, 54.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff916870531000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:02:49 (1788361369) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (23s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:03:12 (1788361392) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:03:34 (1788361414) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 11:03:36 (1788361416) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.197582 s, 15.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.12889 s, 24.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x667e82c2 (nosync,stale) Mirror 2: 0x667e82c2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8c573e84ee21551f21a13fc44ba4eb11 - mirror 1 checksum: 57fb334eb3d483afd06e4ee58fd8b623 - mirror 2 checksum: 8c573e84ee21551f21a13fc44ba4eb11 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T11:03:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T11:03:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8c573e84ee21551f21a13fc44ba4eb11 - mirror 1 checksum: 57fb334eb3d483afd06e4ee58fd8b623 - mirror 2 checksum: 57fb334eb3d483afd06e4ee58fd8b623 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T11:03:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T11:03:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:03:43 (1788361423) Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server 11:03:47 (1788361427) shut down facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server Failover mds1 to oleg357-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 11:04:05 (1788361445) targets are mounted 11:04:05 (1788361445) facet_failover done oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238154 s, 44.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (32s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:04:15 (1788361455) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:04:17 (1788361457) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:04:19 (1788361459) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00625996 s, 2.6 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:04:23 (1788361463) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:04:25 (1788361465) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00668379 s, 613 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0598293 s, 68.5 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:04:30 (1788361470) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (16s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:04:46 (1788361486) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:05:10 (1788361510) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:05:30 (1788361530) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:05:51 (1788361551) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 11:05:55 (1788361555) mirror create pid 54180 mirror split pid 54182 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54180 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54182 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 11:06:59 (1788361619) Seed set to 1620 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004 1788361620.328197 write 0x0201e6 thru 0x02ff0e (0xfd29 bytes) 000005 1788361620.330181 mapread 0x016510 thru 0x0242d4 (0xddc5 bytes) 000008 1788361620.350556 read 0x019c23 thru 0x0284ff (0xe8dd bytes) 000010 1788361620.352182 mapread 0x0105c5 thru 0x0156f8 (0x5134 bytes) 000012 1788361620.361956 read 0x011258 thru 0x012337 (0x10e0 bytes) 000017 1788361620.364921 write 0x0038b8 thru 0x005f95 (0x26de bytes) 000020 1788361620.374498 mapread 0x0078f1 thru 0x00b630 (0x3d40 bytes) 000022 1788361620.376306 mapread 0x02bb83 thru 0x02c904 (0xd82 bytes) 000024 1788361620.377800 mapwrite 0x015ab5 thru 0x017ce8 (0x2234 bytes) 000026 1788361620.420097 mapread 0x008644 thru 0x016e13 (0xe7d0 bytes) 000028 1788361620.422304 write 0x008129 thru 0x00f69a (0x7572 bytes) truncating to largest ever: 0xde9c 000029 1788361620.423912 trunc from 0x02ff0f to 0x00de9b (0x22073 bytes) 000032 1788361620.440427 write 0x014517 thru 0x01afe5 (0x6acf bytes) truncating to largest ever: 0x3a406 000036 1788361620.445129 trunc from 0x01afe6 to 0x03a405 (0x1f420 bytes) 000038 1788361620.466898 write 0x01e906 thru 0x0252c1 (0x69bc bytes) 000039 1788361620.481964 mapread 0x00f5d6 thru 0x01f579 (0xffa4 bytes) 000040 1788361620.524555 read 0x01e796 thru 0x0206aa (0x1f15 bytes) 000041 1788361620.525130 mapread 0x005de7 thru 0x00f97a (0x9b94 bytes) 000042 1788361620.534869 mapwrite 0x028997 thru 0x0356a1 (0xcd0b bytes) 000045 1788361620.581292 trunc from 0x03a406 to 0x0381dd (0x2228 bytes) 000046 1788361620.601771 write 0x02c94d thru 0x039007 (0xc6bb bytes) 000047 1788361620.603272 mapread 0x015de7 thru 0x023f7f (0xe199 bytes) 000049 1788361620.605799 write 0x028b00 thru 0x036470 (0xd971 bytes) 000050 1788361620.607492 trunc from 0x039008 to 0x03552c (0x3adb bytes) 000052 1788361620.621886 MIRROR_EXTEND to 2 mirrors 000053 1788361621.057329 mapwrite 0x01b3be thru 0x022380 (0x6fc3 bytes) truncating to largest ever: 0x3c3d8 000054 1788361621.178327 trunc from 0x03552d to 0x03c3d7 (0x6eab bytes) 000055 1788361621.219638 mapread 0x03a420 thru 0x03c3d7 (0x1fb8 bytes) 000057 1788361621.233541 MIRROR_RESYNC 2 mirrors 000060 1788361621.678187 write 0x03ccf5 thru 0x03ffff (0x330b bytes) 000062 1788361621.744567 MIRROR_RESYNC 2 mirrors 000064 1788361622.145605 MIRROR_EXTEND to 3 mirrors 000066 1788361622.649938 write 0x00a1e2 thru 0x0176d0 (0xd4ef bytes) 000067 1788361622.715111 trunc from 0x040000 to 0x006d4a (0x392b5 bytes) 000068 1788361622.735676 mapwrite 0x011492 thru 0x01dec2 (0xca31 bytes) 000069 1788361622.831733 mapwrite 0x01fcb2 thru 0x0226d0 (0x2a1f bytes) 000070 1788361622.891198 write 0x00ef9e thru 0x0117cf (0x2832 bytes) 000071 1788361622.892792 mapwrite 0x03a048 thru 0x03c231 (0x21ea bytes) 000076 1788361622.958442 mapread 0x02d468 thru 0x038a38 (0xb5d1 bytes) 000077 1788361622.965329 write 0x0023b0 thru 0x00c549 (0xa19a bytes) 000080 1788361622.971499 mapread 0x00084f thru 0x005d52 (0x5504 bytes) 000081 1788361622.975781 mapwrite 0x03cacb thru 0x03ffff (0x3535 bytes) 000082 1788361623.053492 mapwrite 0x022ab5 thru 0x02a3a7 (0x78f3 bytes) 000083 1788361623.085215 read 0x00601e thru 0x015287 (0xf26a bytes) 000085 1788361623.086492 mapread 0x003e8c thru 0x00bf52 (0x80c7 bytes) 000086 1788361623.087362 trunc from 0x040000 to 0x01ffbc (0x20043 bytes) 000087 1788361623.102784 write 0x0284b4 thru 0x036508 (0xe055 bytes) 000090 1788361623.105367 mapwrite 0x01e50f thru 0x02cc48 (0xe73a bytes) 000091 1788361623.149083 trunc from 0x036509 to 0x000c31 (0x358d7 bytes) 000092 1788361623.164099 read 0x000b59 thru 0x000c31 (0xd9 bytes) 000093 1788361623.164756 write 0x0210e9 thru 0x02fced (0xec05 bytes) 000094 1788361623.167332 mapwrite 0x03f0bc thru 0x03ffff (0xf44 bytes) 000095 1788361623.226109 read 0x000951 thru 0x005fb3 (0x5663 bytes) 000097 1788361623.227369 write 0x018136 thru 0x01ee12 (0x6cdd bytes) truncating to largest ever: 0x3fd06 000101 1788361623.229952 trunc from 0x040000 to 0x03fd05 (0x2fa bytes) 000102 1788361623.249026 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: 0xa62f5ddc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: 0xa62f5ddc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd06: skipped 000103 1788361623.490611 write 0x010878 thru 0x0123b2 (0x1b3b bytes) 000104 1788361623.501150 mapread 0x02de8a thru 0x033788 (0x58ff bytes) 000105 1788361623.517037 write 0x0004f5 thru 0x009f16 (0x9a22 bytes) 000109 1788361623.530478 mapread 0x03d3af thru 0x03fd05 (0x2957 bytes) 000113 1788361623.539170 read 0x00cf6c thru 0x013354 (0x63e9 bytes) 000114 1788361623.545568 write 0x016e3b thru 0x021784 (0xa94a bytes) 000116 1788361623.560342 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: 0xa62f5ddc (stale) CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 1: 0x87d2c98c Mirror 4: 0x87d2c98c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd06: skipped 000118 1788361624.125704 mapwrite 0x03ba19 thru 0x03ffff (0x45e7 bytes) 000119 1788361624.275379 read 0x0112ed thru 0x0195c5 (0x82d9 bytes) 000120 1788361624.283415 write 0x006cdc thru 0x00c351 (0x5676 bytes) 000121 1788361624.296632 trunc from 0x040000 to 0x01e37c (0x21c83 bytes) 000122 1788361624.313075 read 0x000c4c thru 0x003fc8 (0x337d bytes) 000124 1788361624.320640 write 0x01e771 thru 0x02b478 (0xcd08 bytes) 000127 1788361624.327889 mapread 0x01cfbc thru 0x028cb2 (0xbcf7 bytes) 000128 1788361624.329260 read 0x01ff66 thru 0x020a4a (0xae5 bytes) 000131 1788361624.335111 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b479): Mirror 3: 0x830e4336 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b479): Mirror 3: 0x830e4336 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b479: skipped 000133 1788361624.576302 trunc from 0x02b479 to 0x02de8c (0x2a14 bytes) 000134 1788361624.592662 MIRROR_RESYNC 2 mirrors 000135 1788361624.983479 read 0x02d29b thru 0x02de8c (0xbf2 bytes) 000136 1788361624.988901 mapwrite 0x03be54 thru 0x03ffff (0x41ac bytes) 000139 1788361625.117477 trunc from 0x040000 to 0x0114b2 (0x2eb4d bytes) 000143 1788361625.136522 mapread 0x00b83e thru 0x0114b2 (0x5c75 bytes) 000144 1788361625.143490 mapwrite 0x03d4e0 thru 0x03e15a (0xc7b bytes) 000145 1788361625.186565 trunc from 0x03e15b to 0x00e0a5 (0x300b5 bytes) 000147 1788361625.200863 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe0a6): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe0a6): Mirror 3: 0x321ca03a (stale) CRC-32 checksum value for chunk [0, 0xe0a6): Mirror 1: 0x321ca03a Mirror 4: 0x321ca03a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe0a6: skipped 000148 1788361625.625949 mapread 0x005817 thru 0x00e0a5 (0x888f bytes) 000149 1788361625.659365 trunc from 0x00e0a6 to 0x025128 (0x17083 bytes) 000151 1788361625.738104 read 0x001707 thru 0x008c50 (0x754a bytes) 000153 1788361625.746600 mapread 0x0226c3 thru 0x025128 (0x2a66 bytes) 000154 1788361625.756289 mapwrite 0x024176 thru 0x02c8b3 (0x873e bytes) 000156 1788361625.809160 mapwrite 0x0195de thru 0x021a36 (0x8459 bytes) 000157 1788361625.859288 write 0x02c527 thru 0x03348b (0x6f65 bytes) 000159 1788361625.864574 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3348c): Mirror 3: 0x3ec8075c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3348c): Mirror 3: 0x3ec8075c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3348c: skipped 000161 1788361626.049157 write 0x00dd56 thru 0x01c417 (0xe6c2 bytes) 000163 1788361626.059742 mapread 0x0220bf thru 0x026dc8 (0x4d0a bytes) 000166 1788361626.078967 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000168 1788361626.165128 mapwrite 0x0067f1 thru 0x015cbe (0xf4ce bytes) 000170 1788361626.220983 mapwrite 0x03c6c3 thru 0x03ffff (0x393d bytes) 000171 1788361626.265534 trunc from 0x040000 to 0x038778 (0x7887 bytes) 000172 1788361626.275944 mapread 0x0254aa thru 0x026197 (0xcee bytes) 000173 1788361626.282882 write 0x035244 thru 0x0387a8 (0x3565 bytes) 000174 1788361626.283972 read 0x033f45 thru 0x0387a8 (0x4864 bytes) 000175 1788361626.288496 trunc from 0x0387a9 to 0x0233ca (0x153de bytes) 000176 1788361626.304973 trunc from 0x0233cb to 0x02acde (0x7914 bytes) 000177 1788361626.318132 mapwrite 0x037ef9 thru 0x03ffff (0x8107 bytes) 000178 1788361626.390677 trunc from 0x040000 to 0x02fe75 (0x1018a bytes) 000180 1788361626.403430 mapwrite 0x03e975 thru 0x03ffff (0x168b bytes) 000182 1788361626.455930 mapwrite 0x02fed8 thru 0x03ab3f (0xac68 bytes) 000183 1788361626.498642 mapwrite 0x0148da thru 0x01f585 (0xacac bytes) 000184 1788361626.540248 mapwrite 0x019253 thru 0x01f906 (0x66b4 bytes) 000187 1788361626.579402 trunc from 0x040000 to 0x01aca7 (0x25358 bytes) 000189 1788361626.601639 MIRROR_EXTEND to 2 mirrors 000194 1788361626.977056 mapread 0x018d95 thru 0x01aca7 (0x1f13 bytes) 000196 1788361626.993143 mapwrite 0x01bac2 thru 0x021b96 (0x60d5 bytes) 000197 1788361627.111307 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000198 1788361627.230960 trunc from 0x021b97 to 0x0147a3 (0xd3f3 bytes) 000199 1788361627.241556 mapread 0x006fa1 thru 0x008fc0 (0x2020 bytes) 000200 1788361627.260695 MIRROR_EXTEND to 2 mirrors 000201 1788361627.687022 mapread 0x014050 thru 0x0147a3 (0x754 bytes) 000202 1788361627.692277 read 0x0027a4 thru 0x00cc52 (0xa4af bytes) 000207 1788361627.700115 write 0x0174e1 thru 0x01e572 (0x7092 bytes) 000208 1788361627.750634 write 0x007f8a thru 0x00800c (0x83 bytes) 000210 1788361627.760678 mapwrite 0x0084e3 thru 0x014047 (0xbb65 bytes) 000212 1788361627.834687 mapread 0x0009b8 thru 0x00b29b (0xa8e4 bytes) 000213 1788361627.842626 trunc from 0x01e573 to 0x02fe0f (0x1189d bytes) 000214 1788361627.890557 write 0x011366 thru 0x01c5d4 (0xb26f bytes) 000215 1788361627.892776 mapwrite 0x01ac0e thru 0x02a59c (0xf98f bytes) 000216 1788361627.983590 write 0x022f4e thru 0x029925 (0x69d8 bytes) 000217 1788361627.985006 mapwrite 0x035d53 thru 0x036604 (0x8b2 bytes) 000218 1788361628.038970 write 0x02b09b thru 0x031d75 (0x6cdb bytes) 000220 1788361628.046567 MIRROR_RESYNC 2 mirrors 000222 1788361628.569591 read 0x032778 thru 0x036604 (0x3e8d bytes) 000224 1788361628.579475 mapwrite 0x001152 thru 0x01006a (0xef19 bytes) 000226 1788361628.742282 read 0x001553 thru 0x01005c (0xeb0a bytes) 000228 1788361628.745597 trunc from 0x036605 to 0x0254fb (0x11109 bytes) 000233 1788361628.773336 MIRROR_RESYNC 2 mirrors 000234 1788361629.272728 MIRROR_RESYNC 2 mirrors 000235 1788361629.492599 trunc from 0x0254fc to 0x0166cd (0xee2e bytes) 000236 1788361629.554923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000237 1788361629.664887 read 0x00137c thru 0x00a9b7 (0x963c bytes) 000238 1788361629.677200 trunc from 0x0166ce to 0x01e53e (0x7e71 bytes) 000239 1788361629.692133 write 0x027d4f thru 0x0290bd (0x136f bytes) 000241 1788361629.693962 write 0x0254f9 thru 0x032c5b (0xd763 bytes) 000242 1788361629.699936 trunc from 0x032c5c to 0x01add7 (0x17e84 bytes) 000244 1788361629.715395 read 0x01aa86 thru 0x01add7 (0x352 bytes) 000247 1788361629.725871 write 0x0024c7 thru 0x0097ed (0x7327 bytes) 000254 1788361629.732103 mapread 0x005b54 thru 0x015971 (0xfe1e bytes) 000257 1788361629.737670 mapwrite 0x001289 thru 0x00f7af (0xe527 bytes) 000258 1788361629.780560 write 0x021c30 thru 0x0312a1 (0xf672 bytes) 000260 1788361629.783963 read 0x00d183 thru 0x0106e9 (0x3567 bytes) 000261 1788361629.784747 mapread 0x01a204 thru 0x02864e (0xe44b bytes) 000262 1788361629.792757 mapread 0x021ea6 thru 0x02edcf (0xcf2a bytes) 000263 1788361629.793959 trunc from 0x0312a2 to 0x03ecdc (0xda3b bytes) 000264 1788361629.811679 mapread 0x011009 thru 0x015faa (0x4fa2 bytes) 000266 1788361629.813273 mapwrite 0x013604 thru 0x021efe (0xe8fb bytes) 000269 1788361629.867006 trunc from 0x03ecdd to 0x01c1f0 (0x22aec bytes) 000270 1788361629.886939 write 0x03e30a thru 0x03ffff (0x1cf6 bytes) 000271 1788361629.889065 trunc from 0x040000 to 0x017c9d (0x28362 bytes) 000272 1788361629.905127 trunc from 0x017c9e to 0x01edd0 (0x7133 bytes) 000274 1788361629.923591 mapwrite 0x015e78 thru 0x024c4a (0xedd3 bytes) 000276 1788361630.042053 mapwrite 0x01d32c thru 0x02ab9c (0xd871 bytes) 000277 1788361630.147476 mapwrite 0x0363e3 thru 0x036863 (0x481 bytes) 000278 1788361630.207070 trunc from 0x036864 to 0x01a99c (0x1bec7 bytes) 000282 1788361630.229306 mapread 0x014a17 thru 0x019cce (0x52b8 bytes) 000283 1788361630.230227 read 0x0023f4 thru 0x010c8b (0xe898 bytes) 000284 1788361630.230733 mapwrite 0x010464 thru 0x0138c8 (0x3465 bytes) 000285 1788361630.298476 read 0x006ac7 thru 0x00c803 (0x5d3d bytes) 000288 1788361630.305515 MIRROR_EXTEND to 2 mirrors 000289 1788361630.697370 mapwrite 0x0016b3 thru 0x00befd (0xa84b bytes) 000290 1788361630.830201 write 0x0189e0 thru 0x01d5e2 (0x4c03 bytes) 000291 1788361630.837543 read 0x00aca7 thru 0x0124bf (0x7819 bytes) 000292 1788361630.842635 read 0x012bf3 thru 0x01988c (0x6c9a bytes) 000293 1788361630.856165 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d5e3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d5e3): Mirror 2: 0xa0a45393 (stale) CRC-32 checksum value for chunk [0, 0x1d5e3): Mirror 1: 0x46dfe525 Mirror 3: 0x46dfe525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d5e3: skipped 000294 1788361631.483708 mapwrite 0x000dab thru 0x00ee72 (0xe0c8 bytes) 000296 1788361631.614470 mapread 0x01b670 thru 0x01d5e2 (0x1f73 bytes) 000299 1788361631.627916 read 0x00a855 thru 0x012b23 (0x82cf bytes) 000300 1788361631.639395 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d5e3): Mirror 2: 0xa0a45393 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d5e3): Mirror 2: 0xa0a45393 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d5e3: skipped 000301 1788361631.818083 trunc from 0x01d5e3 to 0x02e994 (0x113b2 bytes) 000302 1788361631.832880 write 0x03a0ac thru 0x03ffff (0x5f54 bytes) 000303 1788361631.834883 trunc from 0x040000 to 0x0222f2 (0x1dd0d bytes) 000305 1788361631.852456 write 0x025385 thru 0x032ec2 (0xdb3e bytes) 000308 1788361631.856090 read 0x0078d8 thru 0x008581 (0xcaa bytes) 000312 1788361631.864367 mapread 0x01900e thru 0x01ef4e (0x5f41 bytes) 000313 1788361631.886250 mapread 0x00f36c thru 0x01e51e (0xf1b3 bytes) 000315 1788361631.895883 mapread 0x029d88 thru 0x02a015 (0x28e bytes) 000316 1788361631.897282 write 0x01137e thru 0x0127a8 (0x142b bytes) 000317 1788361631.898685 mapread 0x005227 thru 0x014710 (0xf4ea bytes) 000321 1788361631.910116 trunc from 0x032ec3 to 0x01258e (0x20934 bytes) 000323 1788361631.933266 MIRROR_RESYNC 2 mirrors 000325 1788361632.384680 mapread 0x00c621 thru 0x012579 (0x5f59 bytes) 000326 1788361632.401561 mapread 0x006ee7 thru 0x00d5dc (0x66f6 bytes) 000327 1788361632.416932 MIRROR_RESYNC 2 mirrors 000329 1788361632.637061 mapread 0x00b4bf thru 0x01258e (0x70d0 bytes) 000330 1788361632.638201 mapwrite 0x03ac61 thru 0x03ffff (0x539f bytes) 000332 1788361632.781654 trunc from 0x040000 to 0x03a034 (0x5fcb bytes) 000335 1788361632.800076 trunc from 0x03a035 to 0x00b27f (0x2edb5 bytes) 000339 1788361632.825417 read 0x0031ef thru 0x00b27f (0x8091 bytes) 000341 1788361632.842168 write 0x020dbe thru 0x02d087 (0xc2ca bytes) 000343 1788361632.846867 read 0x02b8ed thru 0x02d087 (0x179b bytes) 000344 1788361632.847716 write 0x023395 thru 0x025f01 (0x2b6d bytes) 000345 1788361632.849362 trunc from 0x02d088 to 0x022d67 (0xa320 bytes) 000347 1788361632.869478 MIRROR_RESYNC 2 mirrors 000348 1788361633.433751 read 0x00118d thru 0x00c2db (0xb14f bytes) 000349 1788361633.441975 mapwrite 0x03f20d thru 0x03ffff (0xdf3 bytes) 000350 1788361633.547874 read 0x02bb03 thru 0x0388ca (0xcdc8 bytes) 000352 1788361633.554946 mapread 0x013ecb thru 0x01dd13 (0x9e49 bytes) 000355 1788361633.564058 MIRROR_RESYNC 2 mirrors 000357 1788361634.039042 mapwrite 0x030728 thru 0x038f69 (0x8842 bytes) 000359 1788361634.148750 mapwrite 0x033f5c thru 0x036f8c (0x3031 bytes) 000364 1788361634.183710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000368 1788361634.297267 mapread 0x00407d thru 0x00e6b4 (0xa638 bytes) 000369 1788361634.318402 MIRROR_EXTEND to 2 mirrors 000370 1788361634.854771 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000376 1788361634.957785 read 0x0387e4 thru 0x03d5ce (0x4deb bytes) 000377 1788361634.968639 read 0x00b78c thru 0x00fb40 (0x43b5 bytes) 000378 1788361634.979696 read 0x002217 thru 0x005f66 (0x3d50 bytes) 000381 1788361634.981923 write 0x037c2e thru 0x03ffff (0x83d2 bytes) 000382 1788361635.005216 read 0x01e2c3 thru 0x029424 (0xb162 bytes) 000383 1788361635.005963 mapwrite 0x0276aa thru 0x02dd68 (0x66bf bytes) 000386 1788361635.053282 trunc from 0x040000 to 0x0107a6 (0x2f859 bytes) 000388 1788361635.065192 mapwrite 0x03d98d thru 0x03ffff (0x2673 bytes) 000390 1788361635.107593 mapwrite 0x00de65 thru 0x01de5d (0xfff9 bytes) 000392 1788361635.147836 mapread 0x036d84 thru 0x03f99d (0x8c1a bytes) 000394 1788361635.150223 mapwrite 0x036d0a thru 0x03ffff (0x92f6 bytes) 000395 1788361635.179657 read 0x0194d1 thru 0x0207db (0x730b bytes) 000399 1788361635.181989 read 0x00b92d thru 0x016626 (0xacfa bytes) 000400 1788361635.183836 write 0x0290e5 thru 0x037712 (0xe62e bytes) 000402 1788361635.187119 mapwrite 0x03c798 thru 0x03ffff (0x3868 bytes) 000403 1788361635.224782 read 0x0380ea thru 0x03ffff (0x7f16 bytes) 000405 1788361635.226456 write 0x02f295 thru 0x03e5a7 (0xf313 bytes) 000406 1788361635.228062 trunc from 0x040000 to 0x006f69 (0x39096 bytes) 000407 1788361635.244856 mapwrite 0x0228a0 thru 0x02fe5d (0xd5be bytes) 000408 1788361635.307730 trunc from 0x02fe5e to 0x00f667 (0x207f6 bytes) 000413 1788361635.325125 mapwrite 0x0237fe thru 0x024cfb (0x14fe bytes) 000414 1788361635.375181 read 0x0145d6 thru 0x01df5b (0x9986 bytes) 000416 1788361635.383410 mapwrite 0x034d56 thru 0x03ffff (0xb2aa bytes) 000417 1788361635.435602 trunc from 0x040000 to 0x02dfd9 (0x12026 bytes) 000418 1788361635.453627 MIRROR_EXTEND to 2 mirrors 000420 1788361635.903020 mapwrite 0x03f3a6 thru 0x03ffff (0xc5a bytes) 000421 1788361635.997007 trunc from 0x040000 to 0x033056 (0xcfa9 bytes) 000426 1788361636.014151 write 0x02886b thru 0x0313fc (0x8b92 bytes) 000427 1788361636.026101 write 0x03596e thru 0x03a14e (0x47e1 bytes) 000428 1788361636.027337 trunc from 0x03a14f to 0x030dea (0x9364 bytes) 000430 1788361636.040290 mapwrite 0x01da29 thru 0x02582d (0x7e05 bytes) 000431 1788361636.095223 write 0x02931a thru 0x029ba3 (0x88a bytes) 000432 1788361636.100394 mapwrite 0x035aee thru 0x03a345 (0x4858 bytes) 000433 1788361636.154222 mapread 0x023bb7 thru 0x02a30e (0x6758 bytes) 000434 1788361636.158461 write 0x02e997 thru 0x03c499 (0xdb03 bytes) 000436 1788361636.160670 write 0x00db59 thru 0x00db71 (0x19 bytes) 000437 1788361636.165264 write 0x01a1a6 thru 0x01dd8e (0x3be9 bytes) 000439 1788361636.166800 trunc from 0x03c49a to 0x02e8d4 (0xdbc5 bytes) 000442 1788361636.181489 trunc from 0x02e8d5 to 0x026ece (0x7a06 bytes) 000444 1788361636.193478 mapwrite 0x028fe3 thru 0x035000 (0xc01e bytes) 000445 1788361636.258520 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000446 1788361636.359418 mapread 0x0001d3 thru 0x004188 (0x3fb6 bytes) 000448 1788361636.380241 write 0x017d6b thru 0x021de7 (0xa07d bytes) 000451 1788361636.389137 trunc from 0x035001 to 0x01e8df (0x16721 bytes) 000452 1788361636.404796 read 0x0029d3 thru 0x00e0a3 (0xb6d1 bytes) 000453 1788361636.405543 trunc from 0x01e8e0 to 0x024de1 (0x6502 bytes) 000455 1788361636.418057 write 0x034497 thru 0x03ddbe (0x9928 bytes) 000458 1788361636.420424 mapread 0x0179a6 thru 0x0182bb (0x916 bytes) 000460 1788361636.424944 MIRROR_EXTEND to 2 mirrors 000461 1788361636.866530 MIRROR_EXTEND to 3 mirrors 000464 1788361637.348032 write 0x01df14 thru 0x02d4c3 (0xf5b0 bytes) 000468 1788361637.404143 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ddbf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddbf): Mirror 2: 0x63511251 (stale) Mirror 3: 0x63511251 (stale) CRC-32 checksum value for chunk [0, 0x3ddbf): Mirror 1: 0x11b1e22e Mirror 4: 0x11b1e22e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddbf: skipped 000469 1788361637.997458 mapwrite 0x01077c thru 0x016650 (0x5ed5 bytes) 000470 1788361638.101462 trunc from 0x03ddbf to 0x02e8ec (0xf4d2 bytes) 000473 1788361638.116401 mapwrite 0x0149a9 thru 0x0237a5 (0xedfd bytes) 000474 1788361638.165160 write 0x005edb thru 0x008814 (0x293a bytes) 000475 1788361638.174759 trunc from 0x02e8ed to 0x010025 (0x1e8c7 bytes) 000478 1788361638.188005 mapread 0x00bbe7 thru 0x010025 (0x443f bytes) 000479 1788361638.189514 write 0x03b618 thru 0x03ffff (0x49e8 bytes) 000481 1788361638.194930 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x84571832 (stale) Mirror 3: 0x84571832 (stale) Mirror 4: 0x98502dc1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ce70089 Mirror 5: 0x6ce70089 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000482 1788361638.728743 mapread 0x00e108 thru 0x010335 (0x222e bytes) 000484 1788361638.755368 write 0x00f2e9 thru 0x011274 (0x1f8c bytes) 000485 1788361638.803147 mapwrite 0x021aa1 thru 0x02d80f (0xbd6f bytes) 000486 1788361638.853878 mapread 0x02d718 thru 0x034038 (0x6921 bytes) 000487 1788361638.865366 MIRROR_RESYNC 5 mirrors 000488 1788361639.472535 read 0x01048a thru 0x013773 (0x32ea bytes) 000491 1788361639.479495 mapwrite 0x01d7bc thru 0x020999 (0x31de bytes) 000492 1788361639.603141 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7be4db38 (stale) Mirror 4: 0x7be4db38 (stale) Mirror 5: 0x7be4db38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7be4db38 (stale) Mirror 4: 0x7be4db38 (stale) Mirror 5: 0x7be4db38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1788361639.846568 mapread 0x016cc6 thru 0x01bbdd (0x4f18 bytes) 000494 1788361639.866170 read 0x03b526 thru 0x03d640 (0x211b bytes) 000495 1788361639.879482 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7be4db38 (stale) Mirror 5: 0x7be4db38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7be4db38 (stale) Mirror 5: 0x7be4db38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000496 1788361640.115879 read 0x027672 thru 0x028ad5 (0x1464 bytes) 000498 1788361640.136777 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7be4db38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7be4db38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000500 1788361640.333169 write 0x004281 thru 0x00a097 (0x5e17 bytes) 000501 1788361640.347098 trunc from 0x040000 to 0x00a798 (0x35867 bytes) 000502 1788361640.362902 trunc from 0x00a799 to 0x033d7f (0x295e7 bytes) 000504 1788361640.378458 trunc from 0x033d80 to 0x03172a (0x2655 bytes) 000505 1788361640.394392 write 0x027960 thru 0x030603 (0x8ca4 bytes) 000507 1788361640.407896 mapwrite 0x004ab4 thru 0x01143f (0xc98c bytes) 000508 1788361640.461695 mapwrite 0x02f10f thru 0x03c41f (0xd311 bytes) 000509 1788361640.522431 trunc from 0x03c420 to 0x001fa4 (0x3a47b bytes) 000510 1788361640.543635 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000512 1788361640.643897 trunc from 0x001fa5 to 0x00b5e2 (0x963e bytes) 000516 1788361640.664533 mapread 0x0097a1 thru 0x00b5e2 (0x1e42 bytes) 000520 1788361640.685574 mapread 0x00afb7 thru 0x00b5e2 (0x62c bytes) 000523 1788361640.687950 mapread 0x00417f thru 0x00a690 (0x6512 bytes) 000524 1788361640.689143 trunc from 0x00b5e3 to 0x00420c (0x73d6 bytes) 000525 1788361640.701628 mapread 0x003d60 thru 0x00420c (0x4ad bytes) 000527 1788361640.710462 write 0x03f99d thru 0x03ffff (0x663 bytes) 000528 1788361640.711632 read 0x03ccfe thru 0x03ffff (0x3302 bytes) 000529 1788361640.717921 read 0x03f189 thru 0x03ffff (0xe77 bytes) 000534 1788361640.722328 trunc from 0x040000 to 0x0206c9 (0x1f936 bytes) 000536 1788361640.733626 mapread 0x00c92b thru 0x016148 (0x981e bytes) 000537 1788361640.743576 mapwrite 0x02072c thru 0x02f9fb (0xf2d0 bytes) 000538 1788361640.803140 read 0x02d45d thru 0x02f9fb (0x259f bytes) 000539 1788361640.803915 mapread 0x016a60 thru 0x01a0fb (0x369c bytes) 000541 1788361640.805711 mapread 0x020c2e thru 0x023900 (0x2cd3 bytes) 000544 1788361640.807929 mapread 0x01f07c thru 0x020e8c (0x1e11 bytes) 000545 1788361640.809146 write 0x026f37 thru 0x0294df (0x25a9 bytes) 000546 1788361640.810261 trunc from 0x02f9fc to 0x039ddb (0xa3e0 bytes) 000548 1788361640.823096 mapwrite 0x033c51 thru 0x03ffff (0xc3af bytes) 000549 1788361640.876616 trunc from 0x040000 to 0x036f8b (0x9074 bytes) 000551 1788361640.897881 trunc from 0x036f8c to 0x025a18 (0x11573 bytes) 000556 1788361640.912091 write 0x011c59 thru 0x015e23 (0x41cb bytes) 000558 1788361640.916151 MIRROR_EXTEND to 2 mirrors 000559 1788361641.343218 MIRROR_EXTEND to 3 mirrors 000562 1788361641.812712 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000563 1788361642.091473 trunc from 0x025a19 to 0x023a86 (0x1f92 bytes) 000564 1788361642.160379 write 0x031a06 thru 0x0332a4 (0x189f bytes) 000568 1788361642.164869 read 0x01d43e thru 0x02b452 (0xe015 bytes) 000569 1788361642.172398 read 0x00ce44 thru 0x010d7b (0x3f38 bytes) 000570 1788361642.182080 trunc from 0x0332a5 to 0x0189da (0x1a8ca bytes) 000571 1788361642.204423 write 0x014a32 thru 0x01c760 (0x7d2f bytes) 000572 1788361642.207069 mapwrite 0x0333dc thru 0x03ff37 (0xcb5c bytes) 000573 1788361642.261155 mapwrite 0x00378e thru 0x00d3fe (0x9c71 bytes) 000578 1788361642.296468 mapread 0x008fab thru 0x014f2c (0xbf82 bytes) 000579 1788361642.297854 write 0x01b329 thru 0x026458 (0xb130 bytes) 000580 1788361642.302928 trunc from 0x03ff38 to 0x00de24 (0x32113 bytes) 000583 1788361642.315871 trunc from 0x00de25 to 0x01619d (0x8379 bytes) 000584 1788361642.339179 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1619e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1619e): Mirror 3: 0x43e8cf9c (stale) CRC-32 checksum value for chunk [0, 0x1619e): Mirror 1: 0x8d2ffb26 Mirror 4: 0x8d2ffb26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1619e: skipped 000587 1788361642.932507 MIRROR_RESYNC 3 mirrors 000588 1788361643.319091 trunc from 0x01619e to 0x02b6b0 (0x15513 bytes) 000590 1788361643.374000 read 0x0292d7 thru 0x029c3e (0x968 bytes) 000591 1788361643.378596 mapread 0x0120b0 thru 0x015e8e (0x3ddf bytes) 000592 1788361643.405999 write 0x018c70 thru 0x024ddc (0xc16d bytes) 000593 1788361643.412409 mapread 0x00b741 thru 0x00d7ae (0x206e bytes) 000594 1788361643.428377 MIRROR_RESYNC 3 mirrors 000595 1788361643.840195 trunc from 0x02b6b1 to 0x035f89 (0xa8d9 bytes) 000597 1788361643.907185 mapread 0x0274b9 thru 0x032120 (0xac68 bytes) 000598 1788361643.929102 trunc from 0x035f8a to 0x026a43 (0xf546 bytes) 000599 1788361643.945403 write 0x00f8e8 thru 0x0100f4 (0x80d bytes) 000600 1788361643.955098 write 0x003116 thru 0x004049 (0xf34 bytes) 000601 1788361643.964095 trunc from 0x026a44 to 0x0156d4 (0x1136f bytes) 000602 1788361643.973607 mapread 0x009e6e thru 0x01005e (0x61f1 bytes) 000603 1788361643.978379 mapread 0x000c21 thru 0x00f101 (0xe4e1 bytes) 000610 1788361643.987288 trunc from 0x0156d5 to 0x02b647 (0x15f73 bytes) 000612 1788361643.998495 mapwrite 0x00c90c thru 0x00df4f (0x1644 bytes) 000613 1788361644.023414 write 0x00dd04 thru 0x00f4d9 (0x17d6 bytes) 000615 1788361644.025072 trunc from 0x02b648 to 0x009798 (0x21eaf bytes) 000616 1788361644.039449 mapwrite 0x004cdc thru 0x00b594 (0x68b9 bytes) 000617 1788361644.097793 MIRROR_RESYNC 3 mirrors 000619 1788361644.532228 read 0x000c25 thru 0x00b594 (0xa970 bytes) 000620 1788361644.541966 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000621 1788361644.741559 write 0x02227d thru 0x030d88 (0xeb0c bytes) 000622 1788361644.789203 read 0x023871 thru 0x026cb9 (0x3449 bytes) 000623 1788361644.790023 read 0x003b2c thru 0x00bf7f (0x8454 bytes) 000625 1788361644.801783 write 0x023b2a thru 0x031b61 (0xe038 bytes) 000626 1788361644.803797 write 0x004148 thru 0x007d8b (0x3c44 bytes) 000627 1788361644.804775 trunc from 0x031b62 to 0x027c49 (0x9f18 bytes) 000629 1788361644.839154 trunc from 0x027c4a to 0x00dc9e (0x19fab bytes) 000630 1788361644.851574 trunc from 0x00dc9f to 0x03e0c8 (0x3042a bytes) 000632 1788361644.863747 mapread 0x03625f thru 0x03cc1e (0x69c0 bytes) 000635 1788361644.877755 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e0c9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0c9): Mirror 3: 0x9dbff06e (stale) CRC-32 checksum value for chunk [0, 0x3e0c9): Mirror 1: 0x5f890519 Mirror 4: 0x5f890519 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0c9: skipped 000639 1788361645.491484 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000641 1788361645.748276 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000642 1788361645.853155 trunc from 0x03e0c9 to 0x02b599 (0x12b2f bytes) 000643 1788361645.879915 write 0x0081cd thru 0x00b5b5 (0x33e9 bytes) 000644 1788361645.887403 mapwrite 0x015f55 thru 0x022220 (0xc2cc bytes) 000645 1788361645.954113 mapread 0x01c5e8 thru 0x026af3 (0xa50c bytes) 000646 1788361645.956920 trunc from 0x02b59a to 0x02aa79 (0xb20 bytes) 000649 1788361645.978418 trunc from 0x02aa7a to 0x02d1f9 (0x2780 bytes) 000650 1788361645.990726 mapread 0x00d90a thru 0x00f092 (0x1789 bytes) 000652 1788361646.000564 trunc from 0x02d1fa to 0x029d41 (0x34b8 bytes) 000658 1788361646.025787 mapwrite 0x02a221 thru 0x035e7a (0xbc5a bytes) 000664 1788361646.087587 mapwrite 0x01162d thru 0x01847c (0x6e50 bytes) 000667 1788361646.119658 write 0x03911a thru 0x039ab8 (0x99f bytes) 000668 1788361646.120744 mapread 0x016955 thru 0x017827 (0xed3 bytes) 000670 1788361646.122256 mapread 0x01e75d thru 0x02b91a (0xd1be bytes) 000671 1788361646.125234 mapwrite 0x03f260 thru 0x03ffff (0xda0 bytes) 000673 1788361646.189791 read 0x02d33b thru 0x0339a0 (0x6666 bytes) 000675 1788361646.191262 mapwrite 0x0353cc thru 0x03ffff (0xac34 bytes) 000676 1788361646.227132 write 0x0019f1 thru 0x010b9f (0xf1af bytes) 000682 1788361646.235612 mapwrite 0x01a2ea thru 0x01e8d6 (0x45ed bytes) 000684 1788361646.263358 mapwrite 0x02cb90 thru 0x0372cd (0xa73e bytes) 000685 1788361646.287308 trunc from 0x040000 to 0x010aa7 (0x2f558 bytes) 000688 1788361646.301551 trunc from 0x010aa8 to 0x02119c (0x106f5 bytes) 000689 1788361646.315083 mapread 0x01230d thru 0x01fe20 (0xdb14 bytes) 000690 1788361646.330040 mapwrite 0x00ef55 thru 0x013bc6 (0x4c72 bytes) 000692 1788361646.358964 read 0x00f897 thru 0x01266f (0x2dd9 bytes) 000693 1788361646.359599 write 0x01648c thru 0x025ab0 (0xf625 bytes) 000696 1788361646.365750 read 0x02314d thru 0x025ab0 (0x2964 bytes) 000697 1788361646.366854 write 0x01c15d thru 0x0221f3 (0x6097 bytes) 000699 1788361646.369061 mapread 0x007900 thru 0x00cd30 (0x5431 bytes) 000704 1788361646.373232 trunc from 0x025ab1 to 0x02e667 (0x8bb7 bytes) 000705 1788361646.385842 mapread 0x00c2ae thru 0x0154bb (0x920e bytes) 000712 1788361646.390452 write 0x037fdc thru 0x03f5f7 (0x761c bytes) 000713 1788361646.391985 read 0x031307 thru 0x03f5f7 (0xe2f1 bytes) 000716 1788361646.401957 read 0x034c3f thru 0x03cf9f (0x8361 bytes) 000719 1788361646.404007 write 0x027363 thru 0x02a34f (0x2fed bytes) 000724 1788361646.408170 mapread 0x02b0fe thru 0x038b73 (0xda76 bytes) 000728 1788361646.411262 trunc from 0x03f5f8 to 0x02073c (0x1eebb bytes) 000729 1788361646.424674 mapread 0x01806a thru 0x018e5d (0xdf4 bytes) 000730 1788361646.425710 read 0x010ebd thru 0x01eb8f (0xdcd3 bytes) 000731 1788361646.426427 read 0x01836e thru 0x020577 (0x820a bytes) 000734 1788361646.428350 write 0x03ef0b thru 0x03ffff (0x10f5 bytes) 000738 1788361646.431443 read 0x01feb5 thru 0x026b10 (0x6c5c bytes) 000739 1788361646.441261 trunc from 0x040000 to 0x008a1e (0x375e1 bytes) 000741 1788361646.453996 mapread 0x004b93 thru 0x008a1e (0x3e8c bytes) 000742 1788361646.455128 trunc from 0x008a1f to 0x0236d5 (0x1acb7 bytes) 000743 1788361646.466757 write 0x01c4a3 thru 0x023dcf (0x792d bytes) 000744 1788361646.475958 trunc from 0x023dd0 to 0x028bc4 (0x4df5 bytes) 000745 1788361646.486849 mapread 0x01dad1 thru 0x020a9a (0x2fca bytes) 000746 1788361646.487796 mapread 0x00190a thru 0x00198c (0x83 bytes) 000749 1788361646.489938 read 0x0057f4 thru 0x011c29 (0xc436 bytes) 000750 1788361646.496044 read 0x021326 thru 0x028bc4 (0x789f bytes) 000752 1788361646.497728 mapread 0x014177 thru 0x018c18 (0x4aa2 bytes) 000753 1788361646.498943 read 0x00e082 thru 0x017b84 (0x9b03 bytes) 000756 1788361646.500291 read 0x01580f thru 0x02414f (0xe941 bytes) 000759 1788361646.501593 read 0x01826e thru 0x0272c9 (0xf05c bytes) 000760 1788361646.502220 mapwrite 0x027974 thru 0x02eeaa (0x7537 bytes) 000762 1788361646.562452 mapwrite 0x007fc6 thru 0x00b7b9 (0x37f4 bytes) 000765 1788361646.607713 trunc from 0x02eeab to 0x00760f (0x2789b bytes) 000766 1788361646.623519 trunc from 0x007610 to 0x0184bc (0x10ead bytes) 000768 1788361646.641233 trunc from 0x0184bd to 0x03498f (0x1c4d3 bytes) 000769 1788361646.651736 mapread 0x017b78 thru 0x024f93 (0xd41c bytes) 000771 1788361646.713954 write 0x02aae1 thru 0x034368 (0x9888 bytes) 000772 1788361646.718177 mapwrite 0x000db6 thru 0x00355f (0x27aa bytes) 000773 1788361646.751302 read 0x00dc56 thru 0x00e6f7 (0xaa2 bytes) 000783 1788361646.763205 mapread 0x00e168 thru 0x00e35c (0x1f5 bytes) 000784 1788361646.764270 write 0x0291da thru 0x02fa96 (0x68bd bytes) 000787 1788361646.766487 mapread 0x0006b9 thru 0x008913 (0x825b bytes) 000791 1788361646.769096 write 0x00a7d7 thru 0x00c415 (0x1c3f bytes) 000792 1788361646.769929 mapread 0x02e4bc thru 0x02fd54 (0x1899 bytes) 000793 1788361646.770764 read 0x0232c4 thru 0x028912 (0x564f bytes) 000800 1788361646.774605 read 0x01e4af thru 0x023e55 (0x59a7 bytes) 000801 1788361646.777582 MIRROR_EXTEND to 2 mirrors 000802 1788361647.249369 read 0x008411 thru 0x0088b5 (0x4a5 bytes) 000803 1788361647.253138 trunc from 0x034990 to 0x019077 (0x1b918 bytes) 000805 1788361647.290101 mapread 0x00986d thru 0x01669f (0xce33 bytes) 000806 1788361647.308404 mapwrite 0x00e760 thru 0x00ef27 (0x7c8 bytes) 000807 1788361647.342507 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) 000809 1788361647.448861 read 0x007bca thru 0x00cce8 (0x511f bytes) 000811 1788361647.456543 read 0x0013fb thru 0x0113c3 (0xffc9 bytes) 000813 1788361647.465404 write 0x009886 thru 0x01796c (0xe0e7 bytes) 000814 1788361647.466800 mapwrite 0x02ecdb thru 0x032ef4 (0x421a bytes) 000815 1788361647.519542 trunc from 0x032ef5 to 0x011aaf (0x21445 bytes) 000816 1788361647.529177 write 0x01a284 thru 0x0270db (0xce58 bytes) 000817 1788361647.530751 mapwrite 0x01a266 thru 0x026ded (0xcb88 bytes) 000818 1788361647.568796 read 0x0053de thru 0x013358 (0xdf7b bytes) 000819 1788361647.573490 write 0x02ca56 thru 0x02e751 (0x1cfc bytes) 000823 1788361647.576617 mapread 0x0206c8 thru 0x022051 (0x198a bytes) 000824 1788361647.577699 read 0x016191 thru 0x01e7a1 (0x8611 bytes) 000826 1788361647.579150 mapread 0x00ceeb thru 0x016dac (0x9ec2 bytes) 000827 1788361647.580265 mapwrite 0x003e8b thru 0x00d1fb (0x9371 bytes) 000828 1788361647.608793 trunc from 0x02e752 to 0x002cee (0x2ba63 bytes) 000829 1788361647.618230 mapread 0x00210c thru 0x002cee (0xbe3 bytes) 000830 1788361647.621942 MIRROR_EXTEND to 2 mirrors 000833 1788361647.927401 write 0x0376a6 thru 0x037a99 (0x3f4 bytes) 000834 1788361647.969466 write 0x03108b thru 0x0343a0 (0x3316 bytes) 000835 1788361647.979987 mapread 0x0097be thru 0x00984f (0x92 bytes) 000836 1788361647.984768 mapread 0x02563a thru 0x025e60 (0x827 bytes) 000837 1788361647.989652 write 0x00ad37 thru 0x01aa8e (0xfd58 bytes) 000838 1788361647.998257 read 0x018614 thru 0x01a164 (0x1b51 bytes) 000840 1788361647.999587 trunc from 0x037a9a to 0x02533d (0x1275c bytes) 000844 1788361648.035719 MIRROR_RESYNC 2 mirrors 000846 1788361648.427404 MIRROR_EXTEND to 3 mirrors 000847 1788361649.055098 read 0x019673 thru 0x02533d (0xbccb bytes) 000849 1788361649.077158 mapread 0x00c796 thru 0x01208d (0x58f8 bytes) 000852 1788361649.116580 trunc from 0x02533e to 0x01569a (0xfca3 bytes) 000853 1788361649.164518 trunc from 0x01569b to 0x0245e3 (0xef49 bytes) 000859 1788361649.185473 trunc from 0x0245e4 to 0x036cb9 (0x126d6 bytes) 000860 1788361649.203241 read 0x02ea27 thru 0x036cb9 (0x8293 bytes) 000861 1788361649.225607 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36cba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cba): Mirror 2: 0x16139f1e (stale) Mirror 3: 0x16139f1e (stale) CRC-32 checksum value for chunk [0, 0x36cba): Mirror 1: 0xc0b42392 Mirror 4: 0xc0b42392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cba: skipped 000862 1788361649.913427 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36cba): Mirror 2: 0x16139f1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cba): Mirror 2: 0x16139f1e (stale) CRC-32 checksum value for chunk [0, 0x36cba): Mirror 1: 0xc0b42392 Mirror 4: 0xc0b42392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cba: skipped 000864 1788361650.352040 trunc from 0x036cba to 0x02f20a (0x7aaf bytes) 000865 1788361650.423441 read 0x00e1f2 thru 0x01565d (0x746c bytes) 000866 1788361650.429830 write 0x0000a0 thru 0x0034c3 (0x3424 bytes) 000868 1788361650.449393 trunc from 0x02f20b to 0x00497d (0x2a88d bytes) 000871 1788361650.464800 read 0x0017eb thru 0x0041d7 (0x29ed bytes) 000873 1788361650.471779 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x497e): Mirror 4: 0x24e4a598 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x497e): Mirror 4: 0x24e4a598 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x497e: skipped 000874 1788361650.665334 read 0x0010e5 thru 0x00497d (0x3899 bytes) 000875 1788361650.669196 read 0x0011c6 thru 0x00497d (0x37b8 bytes) 000877 1788361650.670355 read 0x0041e0 thru 0x00497d (0x79e bytes) 000878 1788361650.670767 mapwrite 0x0341d7 thru 0x03ffff (0xbe29 bytes) 000880 1788361650.708607 write 0x004b41 thru 0x0071c6 (0x2686 bytes) 000882 1788361650.710152 mapread 0x010b80 thru 0x01c72e (0xbbaf bytes) 000885 1788361650.718717 MIRROR_RESYNC 2 mirrors 000886 1788361651.165789 mapwrite 0x000c92 thru 0x0102d7 (0xf646 bytes) 000889 1788361651.319035 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x473f46ae (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9e736f3 Mirror 5: 0xf9e736f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000890 1788361651.966997 write 0x009d01 thru 0x015133 (0xb433 bytes) 000891 1788361652.046543 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x473f46ae (stale) Mirror 5: 0xf9e736f3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f44c3e0 Mirror 6: 0x5f44c3e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1788361652.711568 mapwrite 0x010404 thru 0x01ac71 (0xa86e bytes) 000893 1788361652.840308 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x473f46ae (stale) Mirror 5: 0xf9e736f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x473f46ae (stale) Mirror 5: 0xf9e736f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000894 1788361653.107068 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x473f46ae (stale) Mirror 5: 0xf9e736f3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5c533730 Mirror 6: 0x5c533730 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000895 1788361653.694484 write 0x030f0f thru 0x0364d6 (0x55c8 bytes) 000896 1788361653.743316 read 0x03de5a thru 0x03ffff (0x21a6 bytes) 000898 1788361653.748084 read 0x02514f thru 0x025e51 (0xd03 bytes) 000899 1788361653.755931 read 0x032b5f thru 0x038548 (0x59ea bytes) 000901 1788361653.756953 write 0x0018f6 thru 0x00ee66 (0xd571 bytes) 000902 1788361653.758023 trunc from 0x040000 to 0x02c4a5 (0x13b5a bytes) 000903 1788361653.781773 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c4a6): Mirror 4: 0x6f936c70 (stale) Mirror 6: 0x83d76be7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4a6): Mirror 4: 0x6f936c70 (stale) Mirror 6: 0x83d76be7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4a6: skipped 000904 1788361653.997971 write 0x033d5c thru 0x035fce (0x2273 bytes) 000906 1788361654.000027 mapwrite 0x010db2 thru 0x01a750 (0x999f bytes) 000910 1788361654.048935 mapwrite 0x0096df thru 0x017b72 (0xe494 bytes) 000911 1788361654.092187 read 0x0133e9 thru 0x020044 (0xcc5c bytes) 000912 1788361654.104894 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35fcf): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fcf): Mirror 4: 0x422db7d3 (stale) Mirror 6: 0x7778e8e4 (stale) CRC-32 checksum value for chunk [0, 0x35fcf): Mirror 1: 0x6d43ad99 Mirror 7: 0x6d43ad99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fcf: skipped 000913 1788361654.652160 read 0x0193a2 thru 0x0289d5 (0xf634 bytes) 000914 1788361654.668216 MIRROR_RESYNC 4 mirrors 000915 1788361655.254764 mapread 0x01c707 thru 0x025ab8 (0x93b2 bytes) 000916 1788361655.268047 trunc from 0x035fcf to 0x03532a (0xca4 bytes) 000917 1788361655.338549 mapwrite 0x016192 thru 0x022065 (0xbed4 bytes) 000919 1788361655.393491 mapwrite 0x0246b2 thru 0x02ece0 (0xa62f bytes) 000921 1788361655.453328 mapread 0x00c56a thru 0x0103ac (0x3e43 bytes) 000924 1788361655.471523 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3532b): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3532b): Mirror 4: 0xda90fed4 (stale) Mirror 6: 0xda90fed4 (stale) Mirror 7: 0xda90fed4 (stale) CRC-32 checksum value for chunk [0, 0x3532b): Mirror 1: 0xe415393f Mirror 8: 0xe415393f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3532b: skipped 000926 1788361655.947960 MIRROR_RESYNC 5 mirrors 000927 1788361656.479268 read 0x00a97b thru 0x00cfe5 (0x266b bytes) 000930 1788361656.486971 mapread 0x031899 thru 0x03532a (0x3a92 bytes) 000931 1788361656.505035 read 0x0335bd thru 0x03532a (0x1d6e bytes) 000932 1788361656.505411 write 0x000cde thru 0x009ba6 (0x8ec9 bytes) 000933 1788361656.570903 write 0x0329c4 thru 0x03990f (0x6f4c bytes) 000934 1788361656.572889 read 0x005770 thru 0x00fa4f (0xa2e0 bytes) 000935 1788361656.573717 mapwrite 0x004efe thru 0x011e29 (0xcf2c bytes) 000936 1788361656.619108 mapwrite 0x032373 thru 0x034693 (0x2321 bytes) 000937 1788361656.646672 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39910): Mirror 6: 0xd6679979 (stale) Mirror 7: 0xd6679979 (stale) Mirror 8: 0xd6679979 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39910): Mirror 6: 0xd6679979 (stale) Mirror 7: 0xd6679979 (stale) Mirror 8: 0xd6679979 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39910: skipped 000938 1788361656.880593 mapwrite 0x02fc08 thru 0x0331be (0x35b7 bytes) 000940 1788361656.940559 read 0x00aa26 thru 0x00fbc8 (0x51a3 bytes) 000943 1788361656.956619 mapread 0x02a960 thru 0x0387c0 (0xde61 bytes) 000945 1788361656.958914 trunc from 0x039910 to 0x035913 (0x3ffc bytes) 000947 1788361657.001467 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35914): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35914): Mirror 6: 0x6895783 (stale) Mirror 7: 0x6895783 (stale) Mirror 8: 0x6895783 (stale) CRC-32 checksum value for chunk [0, 0x35914): Mirror 1: 0x223b089e Mirror 9: 0x223b089e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35914: skipped 000948 1788361657.567496 mapwrite 0x00aba1 thru 0x00ae1f (0x27f bytes) 000949 1788361657.692276 read 0x016b63 thru 0x026aae (0xff4c bytes) 000950 1788361657.704346 MIRROR_RESYNC 5 mirrors 000951 1788361658.385715 read 0x01e72a thru 0x02d2dd (0xebb4 bytes) 000959 1788361658.394304 write 0x02240e thru 0x02a91b (0x850e bytes) 000960 1788361658.447570 trunc from 0x035914 to 0x018694 (0x1d27f bytes) 000963 1788361658.462450 mapwrite 0x0168d9 thru 0x021875 (0xaf9d bytes) 000964 1788361658.542360 MIRROR_RESYNC 5 mirrors 000965 1788361659.118012 mapwrite 0x01d94f thru 0x02d516 (0xfbc8 bytes) 000966 1788361659.261683 trunc from 0x02d517 to 0x0260c0 (0x7456 bytes) 000968 1788361659.282565 read 0x02162d thru 0x0260c0 (0x4a94 bytes) 000969 1788361659.283265 mapwrite 0x001ee1 thru 0x00ead4 (0xcbf4 bytes) 000970 1788361659.334056 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x260c1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x260c1): Mirror 6: 0xb5ea22b8 (stale) Mirror 7: 0xb5ea22b8 (stale) Mirror 8: 0xb5ea22b8 (stale) Mirror 9: 0xb5ea22b8 (stale) CRC-32 checksum value for chunk [0, 0x260c1): Mirror 1: 0xbea4497b Mirror 10: 0xbea4497b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x260c1: skipped 000971 1788361659.820584 mapread 0x0051b5 thru 0x0138d9 (0xe725 bytes) 000972 1788361659.846960 write 0x01bcc2 thru 0x026588 (0xa8c7 bytes) 000974 1788361659.909605 mapread 0x00e13f thru 0x01baed (0xd9af bytes) 000976 1788361659.923181 trunc from 0x026589 to 0x0226c8 (0x3ec0 bytes) 000977 1788361659.950597 write 0x01cebb thru 0x020c5d (0x3da3 bytes) 000980 1788361659.953101 write 0x0251cc thru 0x02e1e8 (0x901d bytes) 000981 1788361659.954368 read 0x005d2c thru 0x010dca (0xb09f bytes) 000982 1788361659.964713 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e1e9): Mirror 7: 0x9db1c546 (stale) Mirror 8: 0x9db1c546 (stale) Mirror 9: 0x9db1c546 (stale) Mirror 10: 0x91d14160 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1e9): Mirror 7: 0x9db1c546 (stale) Mirror 8: 0x9db1c546 (stale) Mirror 9: 0x9db1c546 (stale) Mirror 10: 0x91d14160 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1e9: skipped 000983 1788361660.231025 read 0x00ae5c thru 0x012feb (0x8190 bytes) 000984 1788361660.246289 write 0x012abc thru 0x01eb0c (0xc051 bytes) 000985 1788361660.252728 write 0x00ff9d thru 0x01d7b1 (0xd815 bytes) 000986 1788361660.254723 mapwrite 0x0369bc thru 0x03ffff (0x9644 bytes) 000988 1788361660.305628 mapwrite 0x01431a thru 0x01c001 (0x7ce8 bytes) 000989 1788361660.339106 read 0x02e25e thru 0x03d2e8 (0xf08b bytes) 000990 1788361660.348701 mapwrite 0x03475b thru 0x0368c0 (0x2166 bytes) 000991 1788361660.379335 mapread 0x0311a3 thru 0x03ffff (0xee5d bytes) 000994 1788361660.387994 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb69bfcdc (stale) Mirror 8: 0xb69bfcdc (stale) Mirror 9: 0xb69bfcdc (stale) Mirror 10: 0x7eb15d97 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd8636e2 Mirror 11: 0xdd8636e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000995 1788361660.944849 mapwrite 0x00e4d3 thru 0x01a22b (0xbd59 bytes) 000996 1788361661.060572 mapwrite 0x02d5eb thru 0x03a743 (0xd159 bytes) All operations completed A-OK! PASS 70a (44s) == sanity-flr test 71: check mirror extend parent fid ==== 11:07:43 (1788361663) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 11:07:45 (1788361665) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124604 s, 16.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850223 s, 12.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x42f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x430:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 11:07:50 (1788361670) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6942272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2294336 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4177696 .. Extending file size to 5308448 .. Extending file size to 8095520 .. Extending file size to 8740416 .. Extending file size to 6595552 .. Extending file size to 8368928 .. Extending file size to 8938912 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9104192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8827168 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9280672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57814 57815 57817 57820 57821 failed 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) 192.168.203.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:08:57 (1788361737) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6908064 .. Extending file size to 8208864 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9237184 .. Extending file size to 7761920 .. Extending file size to 8319200 .. Extending file size to 9071296 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9197696 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9245280 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59917 59918 59920 59922 59924 failed 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) 192.168.203.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:10:03 (1788361803) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0616623 s, 34.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147512 s, 14.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1f6878bb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:10:12 (1788361812) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0572473 s, 36.6 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 11:10:16 (1788361816) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2f4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x441:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x441:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x442:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x445:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:10:19 (1788361819) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0985551 s, 21.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x443:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x446:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x447:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:10:23 (1788361823) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0866536 s, 24.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x444:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x445:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x448:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x444:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x445:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x448:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x449:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:10:28 (1788361828) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129692 s, 16.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x446:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x446:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44b:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:10:54 (1788361854) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414655 s, 25.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x447:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x447:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x448:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:11:41 (1788361901) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x449:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x44a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0937745 s, 22.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x449:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x44a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x449:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x44a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:12:27 (1788361947) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. Creating new pool test_204f oleg357-server: Pool lustre.test_204f created Adding targets to pool oleg357-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg357-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790789 s, 26.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x44b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x44b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x44f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg357-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg357-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg357-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:13:28 (1788362008) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:13:32 (1788362012) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:13:35 (1788362015) oleg357-server: Pool lustre.test_206 created Adding targets to pool oleg357-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg357-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg357-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg357-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg357-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:13:52 (1788362032) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358211 s, 29.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288331 s, 36.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x35b05804 Mirror 3: 0x35b05804 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:13:56 (1788362036) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:13:58 (1788362038) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:14:00 (1788362040) Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.055002 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2160000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xaf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xaf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xaf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:14:20 (1788362060) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00546843 s, 1.5 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:14:24 (1788362064) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251155 s, 41.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x319 lmm_fid: [0x200000404:0x319:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x319 lmm_fid: [0x200000404:0x319:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x45a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834713 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 11:14:24.000000000 -0400 Modify: 2026-09-02 11:14:24.000000000 -0400 Change: 2026-09-02 11:14:25.000000000 -0400 Birth: 2026-09-02 11:14:24.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:14:27 (1788362067) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:14:29 (1788362069) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.287934 s, 142 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.024041 s, 170 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 2 11:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x31b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 2 11:14 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:14:33 (1788362073) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:14:37 (1788362077) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1229 sec =========== 11:14:41 (1788362081) === sanity-flr: start cleanup 11:14:41 (1788362081) === === sanity-flr: finish cleanup 11:14:44 (1788362084) ===