-----============= acceptance-small: sanity-flr ============----- Mon Jul 6 10:55:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg437-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:55:38 (1783349738) === oleg437-client.virtnet: executing check_config_client /mnt/lustre oleg437-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg437-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ec04963c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ec04963c800.idle_timeout=debug disable quota as required oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:55:59 (1783349759) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7592] == sanity-flr test 0a: lfs mirror create with -N option == 10:56:01 (1783349761) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:56:11 (1783349771) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:56:13 (1783349773) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:56:16 (1783349776) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:56:31 (1783349791) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:56:33 (1783349793) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:56:35 (1783349795) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ec04963c800.stats=clear osc.lustre-OST0001-osc-ffff9ec04963c800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:56:40 (1783349800) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:56:47 (1783349807) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:56:53 (1783349813) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:56:55 (1783349815) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:57:00 (1783349820) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 10:57:06 (1783349826) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:57:11 (1783349831) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:57:13 (1783349833) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:57:18 (1783349838) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:57:24 (1783349844) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:57:29 (1783349849) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.21039 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100044 s, 10.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:57:37 (1783349857) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471077 s, 22.3 MB/s PASS 22 (5s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:57:42 (1783349862) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:57:50 (1783349870) Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (50s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:58:41 (1783349921) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.515571 s, 14.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (65s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:59:45 (1783349985) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (98s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:01:23 (1783350083) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.81424 s, 7.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 6 11:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.924s user 0m0.000s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.540s user 0m0.001s sys 0m0.202s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.207s user 0m0.001s sys 0m0.196s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ec04963c800.ost_server_uuid in FULL state after 0 sec PASS 33b (109s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:03:13 (1783350193) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:03:16 (1783350196) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:03:20 (1783350200) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 11:03:24 (1783350204) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 11:03:32 (1783350212) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.671027 s, 18.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:03:46 (1783350226) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384872 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529582 s, 19.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:03:53 (1783350233) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:04:02 (1783350242) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00206612 s, 496 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116694 s, 878 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103509 s, 989 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00289145 s, 354 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 11:04:14 (1783350254) create test files with size 15737928 .. 639a59a87dc0807d24de6595a1a1a444 - 04d543782dae198d3b28b26a7778a4ad - e1b90a5e8bb747c993ad08da56c3d4d8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 11:04:44 (1783350284) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:05:02 (1783350302) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:05:11 (1783350311) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202566 s, 5.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311768 s, 33.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 11:05:23 (1783350323) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161811 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.368425 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259432 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 11:05:41 (1783350341) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 11:05:45 (1783350345) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:05:48 (1783350348) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320195 s, 32.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 11:05:56 (1783350356) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:05:59 (1783350359) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:06:09 (1783350369) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241987 s, 43.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:06:20 (1783350380) llite.lustre-ffff9ec060438800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.318435 s, 32.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ec05166a000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:06:34 (1783350394) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:07:10 (1783350430) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:07:46 (1783350466) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 11:07:49 (1783350469) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.216467 s, 14.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213433 s, 14.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x65d7b7f8 (nosync,stale) Mirror 2: 0x65d7b7f8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 97c6d8a6a294071d720f39b3e49ed35b - mirror 1 checksum: b776bc2733074b991c646e3dd25a75a1 - mirror 2 checksum: 97c6d8a6a294071d720f39b3e49ed35b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-06T11:07:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-06T11:07:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 97c6d8a6a294071d720f39b3e49ed35b - mirror 1 checksum: b776bc2733074b991c646e3dd25a75a1 - mirror 2 checksum: b776bc2733074b991c646e3dd25a75a1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-06T11:07:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-06T11:07:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:07:59 (1783350479) Failing mds1 on oleg437-server Stopping /mnt/lustre-mds1 (opts:) on oleg437-server 11:08:06 (1783350486) shut down facet: mds1 facet_host: oleg437-server facet_failover_host: oleg437-server Failover mds1 to oleg437-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 11:08:27 (1783350507) targets are mounted 11:08:27 (1783350507) facet_failover done oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296234 s, 35.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:08:43 (1783350523) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:08:46 (1783350526) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:08:49 (1783350529) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0152315 s, 1.1 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:08:54 (1783350534) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:08:57 (1783350537) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:09:00 (1783350540) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:09:24 (1783350564) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:09:52 (1783350592) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:10:15 (1783350615) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:10:38 (1783350638) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 11:10:43 (1783350643) mirror create pid 53734 mirror split pid 53735 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53734 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53735 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 11:11:50 (1783350710) Seed set to 711 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1783350712.134559 mapwrite 0x015fab thru 0x018121 (0x2177 bytes) 000005 1783350712.221862 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x1246 000009 1783350712.844469 trunc from 0x018122 to 0x001245 (0x16edc bytes) truncating to largest ever: 0x183f0 000011 1783350712.981632 trunc from 0x001246 to 0x0183ef (0x171aa bytes) truncating to largest ever: 0x295dd 000013 1783350713.021465 trunc from 0x0183f0 to 0x0295dc (0x111ed bytes) 000014 1783350713.045501 write 0x02c3aa thru 0x037039 (0xac90 bytes) 000015 1783350713.049270 mapwrite 0x01fea9 thru 0x02a92e (0xaa86 bytes) 000016 1783350713.183732 trunc from 0x03703a to 0x02428f (0x12daa bytes) 000017 1783350713.198423 read 0x00d37d thru 0x018924 (0xb5a8 bytes) 000018 1783350713.211415 mapread 0x02415f thru 0x02428f (0x131 bytes) 000020 1783350713.217551 mapread 0x01efb9 thru 0x02428f (0x52d7 bytes) 000021 1783350713.219983 mapwrite 0x015b71 thru 0x015e4b (0x2db bytes) 000023 1783350713.274966 read 0x00a732 thru 0x012f9c (0x886b bytes) 000024 1783350713.286979 mapread 0x0052ed thru 0x007479 (0x218d bytes) 000025 1783350713.292422 write 0x01a123 thru 0x020853 (0x6731 bytes) 000026 1783350713.294043 write 0x005158 thru 0x00daf3 (0x899c bytes) 000028 1783350713.297176 write 0x002521 thru 0x00aa9d (0x857d bytes) truncating to largest ever: 0x362dd 000029 1783350713.301110 trunc from 0x024290 to 0x0362dc (0x1204d bytes) 000032 1783350713.323562 read 0x01261e thru 0x02063d (0xe020 bytes) 000037 1783350713.328050 write 0x01cee1 thru 0x0254bd (0x85dd bytes) 000038 1783350713.335041 mapread 0x011064 thru 0x01a45c (0x93f9 bytes) 000040 1783350713.336959 mapwrite 0x014ca4 thru 0x0208a7 (0xbc04 bytes) 000042 1783350713.396941 mapwrite 0x0245e0 thru 0x031f54 (0xd975 bytes) 000043 1783350713.488219 mapwrite 0x03bb27 thru 0x03ffff (0x44d9 bytes) 000045 1783350713.601517 write 0x01e306 thru 0x02dad8 (0xf7d3 bytes) 000047 1783350713.615849 MIRROR_RESYNC 2 mirrors 000048 1783350714.298562 write 0x001ec9 thru 0x01117b (0xf2b3 bytes) 000050 1783350714.396409 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf4bc670e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe1047fb4 Mirror 3: 0xe1047fb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000052 1783350715.271133 mapread 0x0018f8 thru 0x01078b (0xee94 bytes) 000053 1783350715.311757 write 0x016078 thru 0x021890 (0xb819 bytes) truncating to largest ever: 0x37574 000056 1783350715.421267 trunc from 0x040000 to 0x037573 (0x8a8c bytes) 000057 1783350715.447694 write 0x00d0a6 thru 0x0182e4 (0xb23f bytes) 000058 1783350715.457284 read 0x008569 thru 0x00ead4 (0x656c bytes) 000059 1783350715.478653 write 0x0300ab thru 0x035d19 (0x5c6f bytes) 000062 1783350715.513216 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37574): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37574): Mirror 2: 0xfbdd5a10 (stale) Mirror 3: 0x51fce1f0 (stale) CRC-32 checksum value for chunk [0, 0x37574): Mirror 1: 0x3e8d32ed Mirror 4: 0x3e8d32ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37574: skipped 000063 1783350716.387279 mapread 0x00b2f6 thru 0x00d65e (0x2369 bytes) 000065 1783350716.438686 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37574): Mirror 2: 0xfbdd5a10 (stale) Mirror 3: 0x51fce1f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37574): Mirror 2: 0xfbdd5a10 (stale) Mirror 3: 0x51fce1f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37574: skipped 000069 1783350716.822025 read 0x00a00c thru 0x012e6d (0x8e62 bytes) 000075 1783350716.837883 write 0x02f070 thru 0x03dd0b (0xec9c bytes) 000078 1783350716.855546 trunc from 0x03dd0c to 0x0179ef (0x2631c bytes) 000079 1783350716.896395 write 0x01096e thru 0x01f21d (0xe8b0 bytes) 000080 1783350716.902911 trunc from 0x01f21e to 0x02cdb7 (0xdb9a bytes) 000081 1783350716.929246 write 0x0364f4 thru 0x03ffff (0x9b0c bytes) 000082 1783350716.938947 write 0x00a48a thru 0x00a94c (0x4c3 bytes) 000083 1783350716.954391 mapread 0x01af92 thru 0x02a6fa (0xf769 bytes) 000084 1783350716.985833 mapwrite 0x01e962 thru 0x022fcd (0x466c bytes) 000086 1783350717.087472 read 0x01f355 thru 0x02a3ce (0xb07a bytes) 000087 1783350717.088431 write 0x0361d1 thru 0x03ffff (0x9e2f bytes) 000088 1783350717.102432 MIRROR_RESYNC 3 mirrors 000089 1783350718.415015 MIRROR_RESYNC 3 mirrors 000091 1783350718.791479 mapwrite 0x016aea thru 0x025dd0 (0xf2e7 bytes) 000093 1783350719.050274 trunc from 0x040000 to 0x00ac19 (0x353e6 bytes) 000095 1783350719.094305 mapwrite 0x035bc4 thru 0x03ffff (0xa43c bytes) 000096 1783350719.185378 write 0x013e79 thru 0x02243b (0xe5c3 bytes) 000097 1783350719.197196 mapwrite 0x03a9c7 thru 0x03ffff (0x5639 bytes) 000098 1783350719.246974 trunc from 0x040000 to 0x034b27 (0xb4d8 bytes) 000099 1783350719.263628 mapwrite 0x02f7a7 thru 0x039367 (0x9bc1 bytes) 000100 1783350719.343519 mapread 0x023072 thru 0x03209e (0xf02d bytes) 000101 1783350719.363888 trunc from 0x039368 to 0x02c583 (0xcde4 bytes) 000102 1783350719.394846 trunc from 0x02c584 to 0x018b24 (0x13a5f bytes) 000105 1783350719.417174 write 0x012121 thru 0x019fa2 (0x7e82 bytes) 000107 1783350719.432236 MIRROR_RESYNC 3 mirrors 000109 1783350720.592664 mapwrite 0x0336bb thru 0x039fa2 (0x68e8 bytes) 000110 1783350720.749918 mapread 0x011f9e thru 0x01320b (0x126e bytes) 000112 1783350720.763411 trunc from 0x039fa3 to 0x019537 (0x20a6b bytes) truncating to largest ever: 0x39f8f 000113 1783350720.788278 trunc from 0x019538 to 0x039f8e (0x20a57 bytes) 000114 1783350720.811725 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f8f): Mirror 2: 0xa3530080 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f8f): Mirror 2: 0xa3530080 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f8f: skipped 000115 1783350721.103630 mapread 0x01cf61 thru 0x02583e (0x88de bytes) 000116 1783350721.122152 write 0x0315b3 thru 0x038530 (0x6f7e bytes) 000117 1783350721.132173 mapread 0x0021a9 thru 0x00af47 (0x8d9f bytes) 000119 1783350721.141139 trunc from 0x039f8f to 0x02cfa3 (0xcfeb bytes) 000122 1783350721.165509 mapread 0x013fda thru 0x021e25 (0xde4c bytes) 000123 1783350721.187382 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cfa4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cfa4): Mirror 2: 0xb79b34c8 (stale) CRC-32 checksum value for chunk [0, 0x2cfa4): Mirror 1: 0x7f5e3a53 Mirror 3: 0x7f5e3a53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cfa4: skipped 000124 1783350721.971183 trunc from 0x02cfa4 to 0x005230 (0x27d73 bytes) 000126 1783350722.123136 mapwrite 0x0162d4 thru 0x01db46 (0x7873 bytes) 000129 1783350722.291710 mapwrite 0x02d3fb thru 0x03bdc3 (0xe9c9 bytes) 000130 1783350722.470067 write 0x0287f6 thru 0x03361a (0xae25 bytes) 000131 1783350722.472019 write 0x001c5e thru 0x00abfa (0x8f9d bytes) 000133 1783350722.485531 mapread 0x03bd17 thru 0x03bdc3 (0xad bytes) 000134 1783350722.486737 read 0x0216a4 thru 0x0254f8 (0x3e55 bytes) 000135 1783350722.503402 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 2: 0xa16d231c (stale) Mirror 3: 0xb20a2fa4 (stale) CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 1: 0xccf95eb1 Mirror 4: 0xccf95eb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdc4: skipped 000136 1783350723.489986 mapread 0x03af78 thru 0x03bdc3 (0xe4c bytes) 000138 1783350723.534820 mapread 0x01b1d4 thru 0x024fe1 (0x9e0e bytes) 000139 1783350723.573779 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 2: 0xa16d231c (stale) Mirror 3: 0xb20a2fa4 (stale) CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 1: 0xccf95eb1 Mirror 4: 0xccf95eb1 Mirror 5: 0xccf95eb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdc4: skipped 000140 1783350724.596280 write 0x00a5a9 thru 0x0138f3 (0x934b bytes) 000141 1783350724.760714 MIRROR_RESYNC 5 mirrors 000142 1783350726.007981 mapwrite 0x034aac thru 0x03ffff (0xb554 bytes) 000143 1783350726.191614 MIRROR_RESYNC 5 mirrors 000145 1783350727.304093 read 0x001efc thru 0x003e3a (0x1f3f bytes) 000146 1783350727.311548 mapwrite 0x024782 thru 0x02f4c7 (0xad46 bytes) 000147 1783350727.524090 read 0x0178af thru 0x01e55c (0x6cae bytes) 000148 1783350727.536857 mapwrite 0x000f52 thru 0x003a71 (0x2b20 bytes) 000149 1783350727.590430 trunc from 0x040000 to 0x01bbf6 (0x24409 bytes) 000150 1783350727.634745 write 0x02be78 thru 0x038f7c (0xd105 bytes) 000151 1783350727.637914 write 0x007f56 thru 0x016a71 (0xeb1c bytes) 000152 1783350727.641969 mapread 0x0014ee thru 0x00ec04 (0xd717 bytes) 000153 1783350727.649535 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f7d): Mirror 2: 0xb0e63ce3 (stale) Mirror 4: 0xb0e63ce3 (stale) Mirror 5: 0xb0e63ce3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f7d): Mirror 2: 0xb0e63ce3 (stale) Mirror 4: 0xb0e63ce3 (stale) Mirror 5: 0xb0e63ce3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f7d: skipped 000154 1783350728.171619 mapwrite 0x022e16 thru 0x026de1 (0x3fcc bytes) 000155 1783350728.284767 write 0x027113 thru 0x02b3c5 (0x42b3 bytes) 000158 1783350728.288352 read 0x01de24 thru 0x0239f7 (0x5bd4 bytes) 000161 1783350728.290308 read 0x01976b thru 0x01c66a (0x2f00 bytes) 000162 1783350728.311013 read 0x02953f thru 0x02ffee (0x6ab0 bytes) 000163 1783350728.312945 write 0x027d5c thru 0x02d183 (0x5428 bytes) 000164 1783350728.317390 write 0x00cd0f thru 0x01870f (0xba01 bytes) 000165 1783350728.344021 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38f7d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f7d): Mirror 2: 0xb0e63ce3 (stale) Mirror 4: 0xb0e63ce3 (stale) Mirror 5: 0xb0e63ce3 (stale) CRC-32 checksum value for chunk [0, 0x38f7d): Mirror 1: 0xf18ba8df Mirror 6: 0xf18ba8df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f7d: skipped 000166 1783350729.162828 trunc from 0x038f7d to 0x039cc3 (0xd47 bytes) 000167 1783350729.293727 write 0x039003 thru 0x03e330 (0x532e bytes) 000169 1783350729.310021 read 0x00937d thru 0x01535b (0xbfdf bytes) 000171 1783350729.327585 read 0x02d486 thru 0x030b89 (0x3704 bytes) 000172 1783350729.342615 write 0x023856 thru 0x026e7d (0x3628 bytes) 000173 1783350729.348416 mapwrite 0x03b003 thru 0x03ffff (0x4ffd bytes) 000174 1783350729.424033 mapwrite 0x034fc3 thru 0x037392 (0x23d0 bytes) 000175 1783350729.463211 read 0x022a7d thru 0x027364 (0x48e8 bytes) 000176 1783350729.464039 mapwrite 0x025563 thru 0x02bdc4 (0x6862 bytes) 000179 1783350729.509349 read 0x023890 thru 0x0309b7 (0xd128 bytes) 000180 1783350729.510304 mapread 0x02af94 thru 0x032565 (0x75d2 bytes) 000184 1783350729.513443 write 0x00e574 thru 0x0127e2 (0x426f bytes) 000186 1783350729.522895 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 4: 0x4b96bed (stale) Mirror 5: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ab17ab9 Mirror 7: 0x3ab17ab9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000187 1783350730.315102 write 0x0178d5 thru 0x01eca7 (0x73d3 bytes) 000188 1783350730.412177 mapread 0x01f05d thru 0x02afbd (0xbf61 bytes) 000189 1783350730.444654 trunc from 0x040000 to 0x00e265 (0x31d9a bytes) 000190 1783350730.495918 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xe266): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe266): Mirror 2: 0x5a7425b (stale) Mirror 4: 0x5a7425b (stale) Mirror 5: 0x5a7425b (stale) Mirror 6: 0xef297270 (stale) Mirror 7: 0xef297270 (stale) CRC-32 checksum value for chunk [0, 0xe266): Mirror 1: 0xef297270 Mirror 8: 0xef297270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe266: skipped 000191 1783350731.401457 mapwrite 0x016db9 thru 0x0201f4 (0x943c bytes) 000192 1783350731.594663 read 0x0182eb thru 0x01f111 (0x6e27 bytes) 000193 1783350731.596010 mapwrite 0x024c60 thru 0x02a21c (0x55bd bytes) 000195 1783350731.692127 write 0x00a51c thru 0x00a8ef (0x3d4 bytes) 000196 1783350731.704319 mapread 0x014109 thru 0x017511 (0x3409 bytes) 000200 1783350731.716390 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2a21d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a21d): Mirror 2: 0x3ad6398c (stale) Mirror 4: 0x3ad6398c (stale) Mirror 5: 0x3ad6398c (stale) Mirror 6: 0x9d1c5110 (stale) Mirror 7: 0xf58fc38 (stale) Mirror 8: 0x675743b6 (stale) CRC-32 checksum value for chunk [0, 0x2a21d): Mirror 1: 0x93a0ac9b Mirror 9: 0x93a0ac9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a21d: skipped 000201 1783350732.840648 read 0x0009ad thru 0x0034c6 (0x2b1a bytes) truncating to largest ever: 0x3e77d 000202 1783350732.864335 trunc from 0x02a21d to 0x03e77c (0x14560 bytes) 000203 1783350732.962448 mapwrite 0x020b69 thru 0x02ee73 (0xe30b bytes) 000207 1783350733.103528 mapwrite 0x014432 thru 0x016030 (0x1bff bytes) 000210 1783350733.183717 mapwrite 0x0345f4 thru 0x03ea4b (0xa458 bytes) 000211 1783350733.296695 write 0x02da70 thru 0x03d839 (0xfdca bytes) 000212 1783350733.299554 read 0x01ab1c thru 0x01efb0 (0x4495 bytes) 000213 1783350733.300651 trunc from 0x03ea4c to 0x03c6f3 (0x2358 bytes) 000215 1783350733.328244 mapread 0x0123bf thru 0x01b976 (0x95b8 bytes) 000216 1783350733.329653 read 0x02bfd0 thru 0x0364a4 (0xa4d5 bytes) 000218 1783350733.335700 read 0x011110 thru 0x011dda (0xccb bytes) 000219 1783350733.339814 trunc from 0x03c6f4 to 0x0310e3 (0xb610 bytes) 000220 1783350733.369110 read 0x001e34 thru 0x01092d (0xeafa bytes) 000221 1783350733.384032 read 0x0250db thru 0x027549 (0x246f bytes) 000222 1783350733.384997 mapread 0x017682 thru 0x01c27b (0x4bfa bytes) 000225 1783350733.390405 read 0x028685 thru 0x0310e3 (0x8a5f bytes) 000226 1783350733.391379 write 0x039f40 thru 0x03ffff (0x60c0 bytes) 000231 1783350733.397874 mapwrite 0x028c1c thru 0x02d86e (0x4c53 bytes) 000232 1783350733.452903 trunc from 0x040000 to 0x036e83 (0x917c bytes) 000234 1783350733.472605 mapread 0x00dd90 thru 0x010b8c (0x2dfd bytes) 000237 1783350733.476351 mapread 0x02fa05 thru 0x036e83 (0x747f bytes) 000238 1783350733.486595 mapwrite 0x03c99d thru 0x03ffff (0x3663 bytes) 000240 1783350733.569617 mapread 0x03a00e thru 0x03ffff (0x5ff2 bytes) 000242 1783350733.571332 trunc from 0x040000 to 0x00f6bb (0x30944 bytes) 000243 1783350733.590635 trunc from 0x00f6bc to 0x03453c (0x24e81 bytes) 000244 1783350733.626102 mapread 0x0200e9 thru 0x02dfa8 (0xdec0 bytes) 000246 1783350733.719711 write 0x02e26a thru 0x032749 (0x44e0 bytes) 000247 1783350733.732325 mapwrite 0x03e2de thru 0x03ffff (0x1d22 bytes) 000251 1783350733.859535 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 4: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) Mirror 7: 0x3ab17ab9 (stale) Mirror 8: 0x74cbe68b (stale) Mirror 9: 0xd9f09ee9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 4: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) Mirror 7: 0x3ab17ab9 (stale) Mirror 8: 0x74cbe68b (stale) Mirror 9: 0xd9f09ee9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000252 1783350734.648388 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) Mirror 7: 0x3ab17ab9 (stale) Mirror 8: 0x74cbe68b (stale) Mirror 9: 0xd9f09ee9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) Mirror 7: 0x3ab17ab9 (stale) Mirror 8: 0x74cbe68b (stale) Mirror 9: 0xd9f09ee9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000253 1783350735.061558 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b96bed (stale) Mirror 6: 0xbf837172 (stale) Mirror 7: 0x3ab17ab9 (stale) Mirror 8: 0x74cbe68b (stale) Mirror 9: 0xd9f09ee9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8c8998c Mirror 10: 0xd8c8998c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000254 1783350736.113310 mapread 0x0389de thru 0x03fd75 (0x7398 bytes) 000255 1783350736.161260 trunc from 0x040000 to 0x022d48 (0x1d2b7 bytes) 000256 1783350736.282492 mapread 0x021c4c thru 0x022d48 (0x10fd bytes) 000257 1783350736.296087 MIRROR_RESYNC 7 mirrors 000258 1783350737.677532 read 0x022850 thru 0x022d48 (0x4f9 bytes) 000260 1783350737.684919 write 0x02920d thru 0x037524 (0xe318 bytes) 000262 1783350737.758638 mapwrite 0x036782 thru 0x03afec (0x486b bytes) 000263 1783350737.861613 mapread 0x01c371 thru 0x027592 (0xb222 bytes) 000265 1783350737.886291 mapread 0x00ba1d thru 0x012e7a (0x745e bytes) 000266 1783350737.899180 write 0x035680 thru 0x03ffff (0xa980 bytes) 000267 1783350737.900891 write 0x02655e thru 0x02cd29 (0x67cc bytes) 000268 1783350737.907559 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 8: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) Mirror 10: 0xac31b67b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 8: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) Mirror 10: 0xac31b67b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1783350738.462888 mapread 0x012ef3 thru 0x01f7ce (0xc8dc bytes) 000270 1783350738.491474 read 0x01a0f8 thru 0x025558 (0xb461 bytes) 000273 1783350738.493610 write 0x0087ae thru 0x011dd4 (0x9627 bytes) 000277 1783350738.503189 mapwrite 0x020f3e thru 0x027953 (0x6a16 bytes) 000278 1783350738.592291 mapwrite 0x012d4b thru 0x01dd76 (0xb02c bytes) 000280 1783350738.707776 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) Mirror 10: 0xac31b67b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) Mirror 10: 0xac31b67b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000281 1783350739.232597 write 0x035a9f thru 0x036dbf (0x1321 bytes) 000282 1783350739.247227 mapread 0x002f47 thru 0x00a739 (0x77f3 bytes) 000283 1783350739.293518 read 0x02191c thru 0x02500c (0x36f1 bytes) 000284 1783350739.318000 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac31b67b (stale) Mirror 7: 0xac31b67b (stale) Mirror 9: 0xac31b67b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000285 1783350739.736921 mapread 0x003e7d thru 0x00777f (0x3903 bytes) 000286 1783350739.775533 trunc from 0x040000 to 0x03ae41 (0x51be bytes) 000288 1783350739.804740 read 0x03811e thru 0x03ae41 (0x2d24 bytes) 000289 1783350739.825685 write 0x00d50d thru 0x0104cb (0x2fbf bytes) 000291 1783350739.837052 MIRROR_RESYNC 4 mirrors 000292 1783350740.747697 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000294 1783350741.120042 mapread 0x01daf6 thru 0x02aee0 (0xd3eb bytes) 000296 1783350741.159578 read 0x0366ed thru 0x03ae41 (0x4755 bytes) 000297 1783350741.174160 write 0x0111e4 thru 0x011382 (0x19f bytes) 000300 1783350741.242820 mapwrite 0x0067a4 thru 0x011e99 (0xb6f6 bytes) 000301 1783350741.325490 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae42): Mirror 7: 0xbc66bf4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae42): Mirror 7: 0xbc66bf4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae42: skipped 000306 1783350741.663412 write 0x036499 thru 0x03b87e (0x53e6 bytes) 000307 1783350741.688438 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000312 1783350741.903319 MIRROR_EXTEND to 2 mirrors 000314 1783350742.708388 MIRROR_EXTEND to 3 mirrors 000316 1783350743.486248 trunc from 0x03b87f to 0x01a03f (0x2183f bytes) 000317 1783350743.590327 trunc from 0x01a040 to 0x01c068 (0x2029 bytes) 000318 1783350743.625282 mapwrite 0x020c96 thru 0x02babb (0xae26 bytes) 000320 1783350743.794279 MIRROR_RESYNC 3 mirrors 000321 1783350744.548477 read 0x0140b6 thru 0x015de5 (0x1d30 bytes) 000323 1783350744.559174 mapwrite 0x00db28 thru 0x00e941 (0xe1a bytes) 000324 1783350744.836641 write 0x02afdb thru 0x02b51c (0x542 bytes) 000325 1783350744.843957 mapread 0x003bef thru 0x00bd89 (0x819b bytes) 000326 1783350744.866973 read 0x01c2e3 thru 0x02b999 (0xf6b7 bytes) 000327 1783350744.886432 read 0x008c99 thru 0x011ce5 (0x904d bytes) 000329 1783350744.896920 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2babc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2babc): Mirror 2: 0xf0a9e83 (stale) Mirror 3: 0xf0a9e83 (stale) CRC-32 checksum value for chunk [0, 0x2babc): Mirror 1: 0x9c82d629 Mirror 4: 0x9c82d629 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2babc: skipped 000330 1783350745.673208 mapwrite 0x02b6ee thru 0x02e4be (0x2dd1 bytes) 000331 1783350745.859772 mapread 0x006e27 thru 0x0147d3 (0xd9ad bytes) 000332 1783350745.897626 write 0x02cdea thru 0x03977a (0xc991 bytes) 000334 1783350745.902604 mapwrite 0x0265b0 thru 0x027fc7 (0x1a18 bytes) 000335 1783350745.983673 write 0x01cb5c thru 0x0250cf (0x8574 bytes) 000336 1783350745.998120 write 0x030410 thru 0x035cbc (0x58ad bytes) 000337 1783350745.999592 mapwrite 0x03912a thru 0x03d2cb (0x41a2 bytes) 000340 1783350746.124217 mapwrite 0x02ccf4 thru 0x036b95 (0x9ea2 bytes) 000341 1783350746.187507 mapwrite 0x0064f3 thru 0x0161d5 (0xfce3 bytes) 000344 1783350746.270848 mapwrite 0x003d27 thru 0x00a2db (0x65b5 bytes) 000345 1783350746.326381 write 0x0388d7 thru 0x03ffff (0x7729 bytes) 000347 1783350746.334672 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbfd296a8 (stale) Mirror 4: 0xbff2a7d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbfd296a8 (stale) Mirror 4: 0xbff2a7d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000348 1783350746.717032 trunc from 0x040000 to 0x032816 (0xd7e9 bytes) 000349 1783350746.741949 write 0x02bd08 thru 0x0391bb (0xd4b4 bytes) 000350 1783350746.750211 mapwrite 0x01548b thru 0x01d03e (0x7bb4 bytes) 000351 1783350746.876194 read 0x035dce thru 0x0391bb (0x33ee bytes) 000352 1783350746.886492 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x391bc): Mirror 3: 0x63f2f145 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x391bc): Mirror 3: 0x63f2f145 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x391bc: skipped 000353 1783350747.233407 mapwrite 0x021f92 thru 0x0291cb (0x723a bytes) 000354 1783350747.330316 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000356 1783350747.488288 mapwrite 0x02303f thru 0x023560 (0x522 bytes) 000357 1783350747.564601 mapwrite 0x029bd6 thru 0x02e9ab (0x4dd6 bytes) 000361 1783350747.657245 read 0x023ab6 thru 0x0292ac (0x57f7 bytes) 000362 1783350747.658375 mapread 0x01666b thru 0x023503 (0xce99 bytes) 000363 1783350747.679049 read 0x02d426 thru 0x033bd7 (0x67b2 bytes) 000364 1783350747.690903 read 0x007a3b thru 0x0158e0 (0xdea6 bytes) 000365 1783350747.692185 read 0x027355 thru 0x034454 (0xd100 bytes) 000366 1783350747.694627 mapwrite 0x019c9d thru 0x02799b (0xdcff bytes) 000369 1783350747.757448 mapwrite 0x00602e thru 0x00a972 (0x4945 bytes) 000372 1783350747.817489 trunc from 0x0391bc to 0x0185ff (0x20bbc bytes) 000373 1783350747.841493 trunc from 0x018600 to 0x002b52 (0x15aad bytes) 000374 1783350747.864166 mapread 0x0017c5 thru 0x002b52 (0x138e bytes) 000376 1783350747.873319 trunc from 0x002b53 to 0x00f0ae (0xc55c bytes) 000377 1783350747.908032 MIRROR_EXTEND to 2 mirrors 000378 1783350748.647295 trunc from 0x00f0af to 0x02f835 (0x20787 bytes) 000379 1783350748.751385 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f836): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f836): Mirror 2: 0x22ca234 (stale) CRC-32 checksum value for chunk [0, 0x2f836): Mirror 1: 0x22ca234 Mirror 3: 0x22ca234 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f836: skipped 000381 1783350749.657367 trunc from 0x02f836 to 0x0329c6 (0x3191 bytes) 000382 1783350749.798112 write 0x027c52 thru 0x02fe09 (0x81b8 bytes) 000383 1783350749.832044 trunc from 0x0329c7 to 0x00c2d2 (0x266f4 bytes) 000384 1783350749.885256 mapwrite 0x025878 thru 0x02facd (0xa256 bytes) 000385 1783350750.019351 read 0x02452d thru 0x027471 (0x2f45 bytes) 000386 1783350750.028700 MIRROR_RESYNC 3 mirrors 000387 1783350750.776247 mapwrite 0x004ea1 thru 0x00f232 (0xa392 bytes) 000388 1783350751.020947 write 0x01cfe9 thru 0x02a36a (0xd382 bytes) 000391 1783350751.040371 read 0x016335 thru 0x01cec2 (0x6b8e bytes) 000392 1783350751.052837 mapread 0x01d1fc thru 0x01d5a4 (0x3a9 bytes) 000393 1783350751.053854 read 0x0102df thru 0x01312c (0x2e4e bytes) 000395 1783350751.075401 MIRROR_RESYNC 3 mirrors 000396 1783350751.775055 MIRROR_EXTEND to 4 mirrors 000397 1783350752.453270 trunc from 0x02face to 0x019fc0 (0x15b0d bytes) 000398 1783350752.577373 write 0x03d8cc thru 0x03ffff (0x2734 bytes) 000400 1783350752.579603 trunc from 0x040000 to 0x016952 (0x296ad bytes) 000402 1783350752.613006 write 0x035d55 thru 0x03903b (0x32e7 bytes) 000403 1783350752.614264 mapread 0x0164f0 thru 0x01ea40 (0x8551 bytes) 000404 1783350752.647444 trunc from 0x03903c to 0x00fe56 (0x291e5 bytes) 000406 1783350752.675177 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xfe57): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfe57): Mirror 2: 0x55efc519 (stale) Mirror 3: 0x55efc519 (stale) Mirror 4: 0x55efc519 (stale) CRC-32 checksum value for chunk [0, 0xfe57): Mirror 1: 0x55efc519 Mirror 5: 0x55efc519 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfe57: skipped 000408 1783350753.508199 read 0x0073de thru 0x00ad7f (0x39a2 bytes) 000409 1783350753.525889 trunc from 0x00fe57 to 0x03c66a (0x2c814 bytes) 000411 1783350753.611478 mapwrite 0x00d165 thru 0x00df8c (0xe28 bytes) 000412 1783350753.674060 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c66b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c66b): Mirror 2: 0x1f8f1bbe (stale) Mirror 3: 0x1f8f1bbe (stale) Mirror 4: 0x1f8f1bbe (stale) Mirror 5: 0x39cad4 (stale) CRC-32 checksum value for chunk [0, 0x3c66b): Mirror 1: 0x1560f3ea Mirror 6: 0x1560f3ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c66b: skipped 000413 1783350754.502744 trunc from 0x03c66b to 0x027068 (0x15602 bytes) 000415 1783350754.657510 MIRROR_RESYNC 6 mirrors 000417 1783350755.723076 trunc from 0x027069 to 0x036afe (0xfa96 bytes) 000418 1783350755.798483 MIRROR_RESYNC 6 mirrors 000420 1783350756.784172 read 0x033248 thru 0x036afe (0x38b7 bytes) 000421 1783350756.801966 write 0x01e130 thru 0x022f4c (0x4e1d bytes) 000422 1783350756.879707 trunc from 0x036aff to 0x000faa (0x35b54 bytes) 000423 1783350756.907082 mapread 0x000d02 thru 0x000faa (0x2a9 bytes) 000424 1783350756.913141 read 0x000dc6 thru 0x000faa (0x1e5 bytes) 000425 1783350756.913900 write 0x019bd6 thru 0x0273ff (0xd82a bytes) 000426 1783350756.916093 read 0x01057e thru 0x017262 (0x6ce5 bytes) 000427 1783350756.924390 write 0x02a86a thru 0x02efd1 (0x4768 bytes) 000428 1783350756.928692 write 0x0287fc thru 0x02b574 (0x2d79 bytes) 000430 1783350756.937084 read 0x002e9c thru 0x00ae86 (0x7feb bytes) 000431 1783350756.943416 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2efd2): Mirror 3: 0xbed4e143 (stale) Mirror 4: 0xbed4e143 (stale) Mirror 5: 0xbed4e143 (stale) Mirror 6: 0xbed4e143 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2efd2): Mirror 3: 0xbed4e143 (stale) Mirror 4: 0xbed4e143 (stale) Mirror 5: 0xbed4e143 (stale) Mirror 6: 0xbed4e143 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2efd2: skipped 000433 1783350757.344039 mapread 0x00e678 thru 0x013621 (0x4faa bytes) 000435 1783350757.372778 mapwrite 0x020330 thru 0x029ef4 (0x9bc5 bytes) 000436 1783350757.455393 mapread 0x02cff6 thru 0x02efd1 (0x1fdc bytes) 000437 1783350757.460883 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2efd2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2efd2): Mirror 3: 0xbed4e143 (stale) Mirror 4: 0xbed4e143 (stale) Mirror 5: 0xbed4e143 (stale) Mirror 6: 0xbed4e143 (stale) CRC-32 checksum value for chunk [0, 0x2efd2): Mirror 1: 0x39571773 Mirror 7: 0x39571773 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2efd2: skipped 000439 1783350758.216229 trunc from 0x02efd2 to 0x02ad67 (0x426a bytes) 000440 1783350758.332993 mapread 0x01b730 thru 0x0246db (0x8fac bytes) 000441 1783350758.356501 trunc from 0x02ad68 to 0x03bb2d (0x10dc6 bytes) 000442 1783350758.391501 MIRROR_RESYNC 6 mirrors 000443 1783350759.433990 trunc from 0x03bb2e to 0x0371f1 (0x493c bytes) 000444 1783350759.506526 trunc from 0x0371f2 to 0x023f4e (0x132a3 bytes) 000445 1783350759.528091 trunc from 0x023f4f to 0x035319 (0x113cb bytes) 000447 1783350759.542626 read 0x00a57f thru 0x010a4d (0x64cf bytes) 000448 1783350759.557241 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3531a): Mirror 3: 0x6a0f0e5e (stale) Mirror 5: 0x6a0f0e5e (stale) Mirror 6: 0x6a0f0e5e (stale) Mirror 7: 0x6a0f0e5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3531a): Mirror 3: 0x6a0f0e5e (stale) Mirror 5: 0x6a0f0e5e (stale) Mirror 6: 0x6a0f0e5e (stale) Mirror 7: 0x6a0f0e5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3531a: skipped 000449 1783350759.910761 trunc from 0x03531a to 0x03841a (0x3101 bytes) 000450 1783350759.936525 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3841b): Mirror 3: 0x52c6441a (stale) Mirror 5: 0x52c6441a (stale) Mirror 7: 0x52c6441a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3841b): Mirror 3: 0x52c6441a (stale) Mirror 5: 0x52c6441a (stale) Mirror 7: 0x52c6441a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3841b: skipped 000451 1783350760.290841 mapwrite 0x03db88 thru 0x03ffff (0x2478 bytes) 000452 1783350760.377728 write 0x02a55f thru 0x035957 (0xb3f9 bytes) 000453 1783350760.397317 trunc from 0x040000 to 0x00fb60 (0x3049f bytes) 000454 1783350760.414542 mapwrite 0x039e70 thru 0x03a6b2 (0x843 bytes) 000456 1783350760.484312 trunc from 0x03a6b3 to 0x01ef23 (0x1b78f bytes) 000457 1783350760.516584 MIRROR_RESYNC 4 mirrors 000459 1783350761.291483 mapwrite 0x02b10d thru 0x03122e (0x6122 bytes) 000461 1783350761.442963 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3122f): Mirror 3: 0x36263051 (stale) Mirror 5: 0x36263051 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3122f): Mirror 3: 0x36263051 (stale) Mirror 5: 0x36263051 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3122f: skipped 000462 1783350761.727292 read 0x0140ba thru 0x01d586 (0x94cd bytes) 000464 1783350761.744731 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3122f): Mirror 3: 0x36263051 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3122f): Mirror 3: 0x36263051 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3122f: skipped 000465 1783350762.058902 MIRROR_RESYNC 2 mirrors 000466 1783350762.700116 MIRROR_EXTEND to 3 mirrors 000468 1783350763.282127 trunc from 0x03122f to 0x00fb31 (0x216fd bytes) 000470 1783350763.364509 mapread 0x00ec6d thru 0x00fb31 (0xec5 bytes) 000471 1783350763.381832 read 0x005fa0 thru 0x00fb31 (0x9b92 bytes) 000472 1783350763.394902 mapwrite 0x0325b8 thru 0x033a4f (0x1498 bytes) 000474 1783350763.496215 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a50): Mirror 4: 0x5d439a2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a50): Mirror 4: 0x5d439a2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a50: skipped 000476 1783350763.793384 read 0x01a38e thru 0x023f65 (0x9bd8 bytes) 000478 1783350763.810497 mapread 0x00bf51 thru 0x01932c (0xd3dc bytes) 000479 1783350763.846499 write 0x00e42e thru 0x00f626 (0x11f9 bytes) 000482 1783350763.850752 mapwrite 0x03071d thru 0x0386ba (0x7f9e bytes) 000484 1783350763.939300 write 0x0370d4 thru 0x03aee5 (0x3e12 bytes) 000485 1783350763.947097 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3aee6): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aee6): Mirror 4: 0x72843827 (stale) CRC-32 checksum value for chunk [0, 0x3aee6): Mirror 1: 0x100983a9 Mirror 5: 0x100983a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aee6: skipped 000486 1783350764.693174 trunc from 0x03aee6 to 0x01bb81 (0x1f364 bytes) 000488 1783350764.769398 trunc from 0x01bb82 to 0x00bac1 (0x100c0 bytes) 000489 1783350764.795955 mapwrite 0x00692b thru 0x009d38 (0x340e bytes) 000491 1783350764.894386 trunc from 0x00bac2 to 0x032733 (0x26c72 bytes) 000493 1783350764.912308 read 0x00be03 thru 0x015085 (0x9283 bytes) 000495 1783350764.920179 write 0x02c790 thru 0x03bf4a (0xf7bb bytes) 000496 1783350764.927739 mapread 0x01cd8b thru 0x01e392 (0x1608 bytes) 000497 1783350764.948679 mapwrite 0x011174 thru 0x01f69f (0xe52c bytes) 000499 1783350765.012949 mapread 0x025068 thru 0x027b3e (0x2ad7 bytes) 000501 1783350765.031307 write 0x015b4e thru 0x01d5dd (0x7a90 bytes) 000502 1783350765.033499 write 0x02227e thru 0x02d6b3 (0xb436 bytes) truncating to largest ever: 0x3f09e 000503 1783350765.036994 trunc from 0x03bf4b to 0x03f09d (0x3153 bytes) 000505 1783350765.056059 write 0x02121c thru 0x02ebdc (0xd9c1 bytes) 000506 1783350765.059986 read 0x023a7f thru 0x024b15 (0x1097 bytes) 000507 1783350765.067355 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: 0xf84e8887 (stale) Mirror 5: 0x6d1f08a1 (stale) CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 1: 0xcbe5517b Mirror 6: 0xcbe5517b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f09e: skipped 000508 1783350765.788567 read 0x00b4fe thru 0x00c257 (0xd5a bytes) 000509 1783350765.799558 write 0x01e5a6 thru 0x02ae20 (0xc87b bytes) 000511 1783350765.893240 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: 0xf84e8887 (stale) Mirror 5: 0x6d1f08a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: 0xf84e8887 (stale) Mirror 5: 0x6d1f08a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f09e: skipped 000513 1783350766.244636 mapread 0x003174 thru 0x006f52 (0x3ddf bytes) 000514 1783350766.267109 write 0x01e6cb thru 0x02bfc1 (0xd8f7 bytes) 000516 1783350766.278248 mapwrite 0x0131a5 thru 0x01d0f5 (0x9f51 bytes) 000517 1783350766.339298 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 4: 0xf84e8887 (stale) Mirror 5: 0x6d1f08a1 (stale) CRC-32 checksum value for chunk [0, 0x3f09e): Mirror 1: 0x9c8c3915 Mirror 6: 0x9c8c3915 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f09e: skipped 000519 1783350767.077263 mapwrite 0x02b005 thru 0x02d044 (0x2040 bytes) 000520 1783350767.282650 write 0x03828e thru 0x03f895 (0x7608 bytes) 000522 1783350767.296968 read 0x00aae4 thru 0x011714 (0x6c31 bytes) 000523 1783350767.307166 read 0x0387a5 thru 0x03d799 (0x4ff5 bytes) 000524 1783350767.307879 write 0x01afc8 thru 0x01bb66 (0xb9f bytes) 000525 1783350767.315751 read 0x01fb69 thru 0x0226d8 (0x2b70 bytes) 000526 1783350767.316590 trunc from 0x03f896 to 0x036292 (0x9603 bytes) 000527 1783350767.367024 write 0x03acbb thru 0x03c7b5 (0x1afb bytes) 000528 1783350767.368224 mapwrite 0x021667 thru 0x023eed (0x2887 bytes) 000529 1783350767.439189 trunc from 0x03c7b6 to 0x00d0f0 (0x2f6c5 bytes) 000531 1783350767.478152 mapwrite 0x038203 thru 0x03ffff (0x7dfd bytes) 000532 1783350767.627846 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3f3b79da (stale) Mirror 5: 0x4c1a719b (stale) Mirror 6: 0xf375bf97 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa04bd132 Mirror 7: 0xa04bd132 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000537 1783350768.403676 read 0x017555 thru 0x02286a (0xb316 bytes) 000538 1783350768.426961 MIRROR_RESYNC 5 mirrors 000540 1783350769.273714 mapread 0x03063c thru 0x033e04 (0x37c9 bytes) 000542 1783350769.296454 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000543 1783350769.609054 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000546 1783350769.927152 write 0x0015ba thru 0x00ddf3 (0xc83a bytes) 000547 1783350770.001635 read 0x019836 thru 0x025c65 (0xc430 bytes) 000549 1783350770.013546 mapread 0x030d52 thru 0x032c8f (0x1f3e bytes) 000550 1783350770.030702 trunc from 0x040000 to 0x031a20 (0xe5df bytes) 000551 1783350770.070509 mapwrite 0x00d3c8 thru 0x015561 (0x819a bytes) 000552 1783350770.139650 mapwrite 0x03bc14 thru 0x03ffff (0x43ec bytes) 000553 1783350770.218456 read 0x02bd55 thru 0x033b5d (0x7e09 bytes) 000555 1783350770.226887 mapwrite 0x026f85 thru 0x02e140 (0x71bc bytes) 000556 1783350770.272664 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa04bd132 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa04bd132 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000557 1783350770.534969 trunc from 0x040000 to 0x032601 (0xd9fe bytes) 000559 1783350770.554982 write 0x011323 thru 0x01ead5 (0xd7b3 bytes) 000560 1783350770.579306 trunc from 0x032602 to 0x016378 (0x1c289 bytes) 000562 1783350770.606210 read 0x006c4a thru 0x007b0a (0xec1 bytes) 000563 1783350770.614960 trunc from 0x016379 to 0x035cb7 (0x1f93f bytes) 000566 1783350770.654482 write 0x01893f thru 0x01d4d8 (0x4b9a bytes) 000570 1783350770.679019 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000571 1783350770.863460 write 0x0243da thru 0x02dcd2 (0x98f9 bytes) 000572 1783350770.877655 trunc from 0x035cb8 to 0x028fbf (0xccf8 bytes) 000573 1783350770.894668 read 0x02298c thru 0x028fbf (0x6634 bytes) 000574 1783350770.901722 trunc from 0x028fc0 to 0x01c359 (0xcc66 bytes) 000577 1783350770.927459 read 0x004a48 thru 0x00e921 (0x9eda bytes) 000578 1783350770.938705 write 0x0156ec thru 0x021ec1 (0xc7d6 bytes) 000581 1783350770.943109 mapwrite 0x00f73f thru 0x017f4a (0x880c bytes) 000582 1783350770.998869 mapwrite 0x007774 thru 0x00dfb4 (0x6841 bytes) 000583 1783350771.085819 trunc from 0x021ec2 to 0x00a2b7 (0x17c0a bytes) 000585 1783350771.104947 write 0x028b5c thru 0x032ea7 (0xa34c bytes) 000589 1783350771.110680 write 0x006f5a thru 0x007648 (0x6ef bytes) 000590 1783350771.113558 trunc from 0x032ea8 to 0x01c07a (0x16e2d bytes) 000593 1783350771.137279 read 0x002b14 thru 0x011d3f (0xf22c bytes) 000594 1783350771.148745 trunc from 0x01c07b to 0x004c19 (0x17461 bytes) 000596 1783350771.171125 trunc from 0x004c1a to 0x00d675 (0x8a5c bytes) 000597 1783350771.193179 mapwrite 0x0170c0 thru 0x02185f (0xa7a0 bytes) 000600 1783350771.328934 mapwrite 0x004ca3 thru 0x00a8ee (0x5c4c bytes) 000601 1783350771.454737 trunc from 0x021860 to 0x017c19 (0x9c46 bytes) 000603 1783350771.470978 mapread 0x003288 thru 0x00b985 (0x86fe bytes) 000607 1783350771.481403 write 0x036696 thru 0x03ffff (0x996a bytes) 000611 1783350771.486466 trunc from 0x040000 to 0x03d12a (0x2ed5 bytes) 000612 1783350771.505067 mapwrite 0x02a6f9 thru 0x03a1ee (0xfaf6 bytes) 000613 1783350771.559618 mapwrite 0x029767 thru 0x03688f (0xd129 bytes) 000614 1783350771.616061 mapread 0x011165 thru 0x02097c (0xf818 bytes) 000616 1783350771.715612 trunc from 0x03d12b to 0x02517b (0x17faf bytes) 000617 1783350771.739452 write 0x0176c6 thru 0x01c4a1 (0x4ddc bytes) 000618 1783350771.741402 write 0x01770b thru 0x01938d (0x1c83 bytes) 000619 1783350771.742477 read 0x012fc9 thru 0x013012 (0x4a bytes) 000621 1783350771.744625 mapread 0x022126 thru 0x02517b (0x3056 bytes) 000623 1783350771.746939 mapwrite 0x0174cf thru 0x023a5a (0xc58c bytes) 000625 1783350771.833346 MIRROR_EXTEND to 2 mirrors 000626 1783350772.514914 trunc from 0x02517c to 0x03a2fe (0x15183 bytes) 000629 1783350772.622662 mapwrite 0x02b75f thru 0x02f6fb (0x3f9d bytes) 000630 1783350772.731505 mapread 0x0340f4 thru 0x03a2fe (0x620b bytes) 000631 1783350772.751401 read 0x02f4ac thru 0x03a2fe (0xae53 bytes) 000632 1783350772.752138 mapwrite 0x01f4b5 thru 0x024384 (0x4ed0 bytes) 000633 1783350772.803147 read 0x0312fe thru 0x034218 (0x2f1b bytes) 000635 1783350772.805377 trunc from 0x03a2ff to 0x037274 (0x308a bytes) 000636 1783350772.831733 write 0x026f0b thru 0x027330 (0x426 bytes) 000637 1783350772.838547 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37275): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37275): Mirror 2: 0x8ebc76c6 (stale) CRC-32 checksum value for chunk [0, 0x37275): Mirror 1: 0x801ca00c Mirror 3: 0x801ca00c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37275: skipped 000638 1783350773.599273 trunc from 0x037275 to 0x027fb1 (0xf2c3 bytes) 000639 1783350773.675429 mapwrite 0x01eb08 thru 0x022141 (0x363a bytes) 000642 1783350773.740746 MIRROR_RESYNC 3 mirrors 000643 1783350774.475599 read 0x01d34e thru 0x01dff8 (0xcab bytes) 000644 1783350774.482196 mapread 0x026e26 thru 0x027fb1 (0x118c bytes) 000645 1783350774.497106 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000647 1783350774.791043 mapread 0x00584f thru 0x00a066 (0x4818 bytes) 000648 1783350774.813187 read 0x00056e thru 0x00939d (0x8e30 bytes) 000649 1783350774.822350 MIRROR_RESYNC 2 mirrors 000650 1783350775.083774 mapread 0x024c4a thru 0x027fb1 (0x3368 bytes) 000652 1783350775.104923 mapread 0x019c42 thru 0x01abbd (0xf7c bytes) 000657 1783350775.115415 mapwrite 0x0235a3 thru 0x029959 (0x63b7 bytes) 000658 1783350775.246847 trunc from 0x02995a to 0x0267d9 (0x3180 bytes) 000659 1783350775.278098 MIRROR_RESYNC 2 mirrors 000660 1783350775.814109 mapwrite 0x0281e6 thru 0x028cb5 (0xad0 bytes) 000661 1783350775.951308 MIRROR_RESYNC 2 mirrors 000662 1783350776.518578 write 0x008a13 thru 0x00aa89 (0x2077 bytes) 000663 1783350776.589953 trunc from 0x028cb6 to 0x021673 (0x7642 bytes) 000666 1783350776.622122 read 0x01830b thru 0x01b0d3 (0x2dc9 bytes) 000667 1783350776.632549 MIRROR_RESYNC 2 mirrors 000668 1783350777.020088 read 0x020aa6 thru 0x021673 (0xbce bytes) 000671 1783350777.030094 mapwrite 0x00f157 thru 0x01d236 (0xe0e0 bytes) 000672 1783350777.215443 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000673 1783350777.344374 trunc from 0x021674 to 0x03cbc4 (0x1b551 bytes) 000677 1783350777.384502 mapread 0x016a60 thru 0x0241bc (0xd75d bytes) 000678 1783350777.432404 MIRROR_EXTEND to 2 mirrors 000679 1783350777.913099 trunc from 0x03cbc5 to 0x0207b7 (0x1c40d bytes) 000680 1783350778.032846 mapread 0x01c5ef thru 0x0207b7 (0x41c9 bytes) 000681 1783350778.055604 mapwrite 0x0035df thru 0x011148 (0xdb6a bytes) 000682 1783350778.139063 read 0x01649f thru 0x01eb4d (0x86af bytes) 000684 1783350778.140649 mapwrite 0x03154e thru 0x037039 (0x5aec bytes) 000685 1783350778.221470 trunc from 0x03703a to 0x033ced (0x334c bytes) 000686 1783350778.242174 mapread 0x02bc6f thru 0x033ced (0x807f bytes) 000687 1783350778.243615 read 0x014a78 thru 0x01c597 (0x7b20 bytes) 000688 1783350778.245342 mapwrite 0x00b799 thru 0x00bc65 (0x4cd bytes) 000689 1783350778.291299 read 0x0256eb thru 0x033ced (0xe603 bytes) 000691 1783350778.302660 write 0x016f15 thru 0x022c12 (0xbcfe bytes) 000692 1783350778.315433 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000693 1783350778.463633 mapread 0x01cf74 thru 0x01f4b7 (0x2544 bytes) 000694 1783350778.490427 read 0x02cb10 thru 0x033ced (0x71de bytes) 000696 1783350778.503082 mapread 0x012391 thru 0x014333 (0x1fa3 bytes) 000700 1783350778.516490 write 0x033f8b thru 0x0369cd (0x2a43 bytes) 000701 1783350778.518598 trunc from 0x0369ce to 0x01f731 (0x1729c bytes) 000702 1783350778.537470 trunc from 0x01f732 to 0x004921 (0x1ae10 bytes) 000703 1783350778.554444 mapread 0x004578 thru 0x004921 (0x3aa bytes) 000704 1783350778.562718 mapread 0x0030cc thru 0x004921 (0x1856 bytes) 000705 1783350778.563918 mapread 0x003bda thru 0x004921 (0xd48 bytes) 000706 1783350778.564915 mapread 0x002c60 thru 0x004921 (0x1cc2 bytes) 000707 1783350778.569992 trunc from 0x004922 to 0x0135a3 (0xec82 bytes) 000712 1783350778.601875 write 0x02bd37 thru 0x037b67 (0xbe31 bytes) 000713 1783350778.609161 MIRROR_EXTEND to 2 mirrors 000715 1783350779.098501 read 0x015cb9 thru 0x01b04e (0x5396 bytes) 000716 1783350779.111021 trunc from 0x037b68 to 0x03e91c (0x6db5 bytes) 000717 1783350779.197058 trunc from 0x03e91d to 0x006f11 (0x37a0b bytes) 000720 1783350779.227516 write 0x0043a9 thru 0x0096db (0x5333 bytes) 000721 1783350779.233226 trunc from 0x0096dc to 0x021ac7 (0x183ec bytes) 000722 1783350779.250448 trunc from 0x021ac8 to 0x02330b (0x1844 bytes) 000723 1783350779.271540 mapread 0x021944 thru 0x02330b (0x19c8 bytes) 000724 1783350779.300511 mapwrite 0x025577 thru 0x0261e6 (0xc70 bytes) 000725 1783350779.436565 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: 0x1de0a832 (stale) CRC-32 checksum value for chunk [0, 0x261e7): Mirror 1: 0xfea8b9f5 Mirror 3: 0xfea8b9f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261e7: skipped 000726 1783350780.174903 mapwrite 0x016f47 thru 0x019b14 (0x2bce bytes) 000727 1783350780.366032 mapwrite 0x017c4e thru 0x022544 (0xa8f7 bytes) 000728 1783350780.423524 mapwrite 0x013f3f thru 0x020559 (0xc61b bytes) 000729 1783350780.495190 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: 0x1de0a832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: 0x1de0a832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261e7: skipped 000734 1783350780.815984 mapwrite 0x008583 thru 0x0087d6 (0x254 bytes) 000735 1783350780.886633 read 0x014a23 thru 0x01a5ef (0x5bcd bytes) 000736 1783350780.909761 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261e7): Mirror 2: 0x1de0a832 (stale) CRC-32 checksum value for chunk [0, 0x261e7): Mirror 1: 0x37e0d1d6 Mirror 3: 0x37e0d1d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261e7: skipped 000737 1783350781.697321 write 0x03781f thru 0x03f578 (0x7d5a bytes) 000738 1783350781.772540 mapwrite 0x03068d thru 0x0369a2 (0x6316 bytes) 000741 1783350781.844454 trunc from 0x03f579 to 0x005457 (0x3a121 bytes) 000743 1783350781.869213 mapread 0x0048f7 thru 0x005457 (0xb61 bytes) 000745 1783350781.887522 trunc from 0x005458 to 0x020b70 (0x1b719 bytes) 000748 1783350781.907286 mapwrite 0x02096a thru 0x02d4a6 (0xcb3d bytes) 000749 1783350782.047718 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d4a7): Mirror 2: 0x1a0d824c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d4a7): Mirror 2: 0x1a0d824c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d4a7: skipped 000751 1783350782.291689 write 0x019d7a thru 0x01f654 (0x58db bytes) 000752 1783350782.300998 mapread 0x01f92f thru 0x0220d0 (0x27a2 bytes) 000753 1783350782.331723 mapread 0x00f242 thru 0x0194ba (0xa279 bytes) 000754 1783350782.363248 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d4a7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d4a7): Mirror 2: 0x1a0d824c (stale) CRC-32 checksum value for chunk [0, 0x2d4a7): Mirror 1: 0x4072bac2 Mirror 3: 0x4072bac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d4a7: skipped 000756 1783350783.101320 mapwrite 0x02e6ac thru 0x03cd62 (0xe6b7 bytes) 000757 1783350783.293694 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 2: 0xecefa1ae (stale) Mirror 3: 0xd306d207 (stale) CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 1: 0x1edd8cd4 Mirror 4: 0x1edd8cd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd63: skipped 000759 1783350783.949337 mapread 0x010d0f thru 0x01d3cf (0xc6c1 bytes) 000761 1783350783.995266 trunc from 0x03cd63 to 0x0012be (0x3baa4 bytes) 000764 1783350784.100488 trunc from 0x0012bf to 0x028169 (0x26eab bytes) 000765 1783350784.124326 write 0x00f67c thru 0x01f2f0 (0xfc75 bytes) 000766 1783350784.140836 write 0x0061c4 thru 0x00aec3 (0x4d00 bytes) 000767 1783350784.159781 mapread 0x00e6a8 thru 0x011225 (0x2b7e bytes) 000768 1783350784.167950 mapread 0x00a5e1 thru 0x010e9f (0x68bf bytes) 000770 1783350784.169909 read 0x0232e7 thru 0x028169 (0x4e83 bytes) 000771 1783350784.180405 mapread 0x0184e0 thru 0x01cd9b (0x48bc bytes) 000772 1783350784.182578 write 0x035473 thru 0x038d82 (0x3910 bytes) 000774 1783350784.184987 trunc from 0x038d83 to 0x0228a2 (0x164e0 bytes) 000775 1783350784.205864 read 0x01a8b0 thru 0x01cfd4 (0x2725 bytes) 000777 1783350784.207359 mapwrite 0x021167 thru 0x0293c1 (0x825b bytes) 000781 1783350784.294927 mapread 0x00516a thru 0x008f95 (0x3e2c bytes) 000782 1783350784.296185 trunc from 0x0293c2 to 0x00d5b7 (0x1be0a bytes) 000784 1783350784.318208 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd5b8): Mirror 2: 0x71642377 (stale) Mirror 4: 0xa2ee7efa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd5b8): Mirror 2: 0x71642377 (stale) Mirror 4: 0xa2ee7efa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd5b8: skipped 000785 1783350784.656080 read 0x003c6e thru 0x00d5b7 (0x994a bytes) 000786 1783350784.663750 read 0x00621b thru 0x00d5b7 (0x739d bytes) 000787 1783350784.674155 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd5b8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd5b8): Mirror 2: 0x71642377 (stale) Mirror 4: 0xa2ee7efa (stale) CRC-32 checksum value for chunk [0, 0xd5b8): Mirror 1: 0x5d8f745d Mirror 5: 0x5d8f745d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd5b8: skipped 000788 1783350785.340234 write 0x0217eb thru 0x02af0f (0x9725 bytes) 000790 1783350785.410594 write 0x034b69 thru 0x03ffff (0xb497 bytes) 000791 1783350785.414836 write 0x01daaa thru 0x02afda (0xd531 bytes) 000793 1783350785.423663 write 0x02f9e6 thru 0x032ed9 (0x34f4 bytes) 000794 1783350785.431873 read 0x032761 thru 0x032ce7 (0x587 bytes) 000795 1783350785.432576 mapwrite 0x034531 thru 0x03e7a5 (0xa275 bytes) 000797 1783350785.483687 read 0x02b303 thru 0x039a0a (0xe708 bytes) 000799 1783350785.511976 MIRROR_RESYNC 4 mirrors 000800 1783350786.201150 mapwrite 0x006159 thru 0x00b178 (0x5020 bytes) 000801 1783350786.416898 MIRROR_RESYNC 4 mirrors 000806 1783350787.214511 trunc from 0x040000 to 0x009de2 (0x3621d bytes) 000809 1783350787.284158 write 0x0250a5 thru 0x025ace (0xa2a bytes) 000810 1783350787.289366 MIRROR_RESYNC 4 mirrors 000811 1783350788.069702 write 0x03cc74 thru 0x03ffff (0x338c bytes) 000813 1783350788.129565 write 0x036890 thru 0x03c60d (0x5d7e bytes) 000814 1783350788.138458 trunc from 0x040000 to 0x01b22f (0x24dd0 bytes) 000815 1783350788.168196 mapwrite 0x02316a thru 0x02cf4e (0x9de5 bytes) 000816 1783350788.273009 mapwrite 0x03bacb thru 0x03ffff (0x4535 bytes) 000817 1783350788.356987 mapread 0x01de07 thru 0x02bd13 (0xdf0d bytes) 000818 1783350788.370150 mapread 0x03239b thru 0x03ffff (0xdc65 bytes) 000819 1783350788.389578 mapread 0x01d8ed thru 0x0279a9 (0xa0bd bytes) 000821 1783350788.391588 mapwrite 0x024cc6 thru 0x02eccc (0xa007 bytes) 000823 1783350788.436673 write 0x0001f7 thru 0x006b98 (0x69a2 bytes) 000824 1783350788.444844 mapread 0x00b40f thru 0x00b58d (0x17f bytes) 000827 1783350788.457307 mapread 0x017c33 thru 0x017f7a (0x348 bytes) 000828 1783350788.458869 read 0x02f9fe thru 0x038631 (0x8c34 bytes) 000830 1783350788.467443 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x21d5fc67 (stale) Mirror 4: 0x21d5fc67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x21d5fc67 (stale) Mirror 4: 0x21d5fc67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1783350788.803115 mapread 0x00d4df thru 0x00ea71 (0x1593 bytes) 000835 1783350788.823173 trunc from 0x040000 to 0x01b39d (0x24c62 bytes) 000837 1783350788.840706 trunc from 0x01b39e to 0x024cd9 (0x993c bytes) 000838 1783350788.876141 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24cda): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24cda): Mirror 2: 0xbf70c2bd (stale) Mirror 4: 0xbf70c2bd (stale) CRC-32 checksum value for chunk [0, 0x24cda): Mirror 1: 0xf24df479 Mirror 5: 0xf24df479 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24cda: skipped 000839 1783350789.816428 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24cda): Mirror 4: 0xbf70c2bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24cda): Mirror 4: 0xbf70c2bd (stale) CRC-32 checksum value for chunk [0, 0x24cda): Mirror 1: 0xf24df479 Mirror 5: 0xf24df479 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24cda: skipped 000840 1783350790.472728 mapwrite 0x033241 thru 0x0371ea (0x3faa bytes) 000842 1783350790.700132 trunc from 0x0371eb to 0x02dc23 (0x95c7 bytes) 000844 1783350790.737079 trunc from 0x02dc24 to 0x0188cb (0x15358 bytes) 000845 1783350790.758961 mapread 0x005f6a thru 0x00b64c (0x56e3 bytes) 000846 1783350790.775594 write 0x02aaf5 thru 0x02e4da (0x39e6 bytes) 000847 1783350790.778240 mapwrite 0x00ffa5 thru 0x015577 (0x55d3 bytes) 000848 1783350790.853498 trunc from 0x02e4db to 0x037ad6 (0x95fc bytes) 000849 1783350790.877309 mapread 0x006257 thru 0x00b7c3 (0x556d bytes) 000850 1783350790.878458 trunc from 0x037ad7 to 0x03b463 (0x398d bytes) 000854 1783350790.898486 read 0x02d73d thru 0x03b463 (0xdd27 bytes) 000855 1783350790.916714 trunc from 0x03b464 to 0x0312f4 (0xa16f bytes) 000856 1783350790.947720 write 0x02d045 thru 0x03017d (0x3139 bytes) 000857 1783350790.948747 mapwrite 0x020208 thru 0x020a49 (0x842 bytes) 000858 1783350791.036196 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x312f5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312f5): Mirror 4: 0xb117f589 (stale) Mirror 5: 0x790b2d83 (stale) CRC-32 checksum value for chunk [0, 0x312f5): Mirror 1: 0xc7008844 Mirror 6: 0xc7008844 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312f5: skipped 000859 1783350791.868882 mapread 0x00e2de thru 0x010652 (0x2375 bytes) 000860 1783350791.921476 trunc from 0x0312f5 to 0x02c911 (0x49e3 bytes) 000861 1783350792.053547 MIRROR_RESYNC 4 mirrors 000862 1783350792.944621 write 0x03af48 thru 0x03ffff (0x50b8 bytes) 000864 1783350792.994609 read 0x031af8 thru 0x033b47 (0x2050 bytes) 000865 1783350793.000598 trunc from 0x040000 to 0x00e519 (0x31ae6 bytes) 000866 1783350793.029475 trunc from 0x00e51a to 0x037457 (0x28f3e bytes) 000867 1783350793.067103 read 0x023aea thru 0x02dcef (0xa206 bytes) 000868 1783350793.084666 write 0x015915 thru 0x01b2db (0x59c7 bytes) 000869 1783350793.086015 mapread 0x019d60 thru 0x01f2c1 (0x5562 bytes) 000870 1783350793.088051 write 0x019754 thru 0x0222c9 (0x8b76 bytes) 000872 1783350793.102895 MIRROR_RESYNC 4 mirrors 000873 1783350793.983027 write 0x01f720 thru 0x02ebdd (0xf4be bytes) 000874 1783350794.064012 mapwrite 0x004ac6 thru 0x012c7b (0xe1b6 bytes) 000876 1783350794.180340 read 0x033906 thru 0x037457 (0x3b52 bytes) 000877 1783350794.193359 MIRROR_RESYNC 4 mirrors 000879 1783350795.108098 MIRROR_EXTEND to 5 mirrors 000881 1783350795.683083 mapread 0x01837d thru 0x01aedf (0x2b63 bytes) 000882 1783350795.701674 write 0x03551b thru 0x03a5ec (0x50d2 bytes) 000883 1783350795.792046 read 0x00cc66 thru 0x01c2d5 (0xf670 bytes) 000884 1783350795.801329 mapwrite 0x00d207 thru 0x011ebf (0x4cb9 bytes) 000887 1783350795.862462 MIRROR_RESYNC 5 mirrors 000889 1783350796.722167 write 0x02806e thru 0x02c9bd (0x4950 bytes) 000890 1783350796.810460 mapwrite 0x02da2a thru 0x03bdc3 (0xe39a bytes) 000894 1783350796.938471 read 0x005986 thru 0x01048d (0xab08 bytes) 000895 1783350796.945775 mapread 0x00a92a thru 0x00bf8e (0x1665 bytes) 000897 1783350796.952163 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 4: 0xe1cb01ba (stale) Mirror 5: 0xe1cb01ba (stale) Mirror 6: 0xe1cb01ba (stale) Mirror 7: 0xe1cb01ba (stale) CRC-32 checksum value for chunk [0, 0x3bdc4): Mirror 1: 0x80befdce Mirror 8: 0x80befdce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdc4: skipped 000899 1783350797.776913 mapwrite 0x039022 thru 0x03ffff (0x6fde bytes) 000902 1783350797.948767 write 0x0187be thru 0x02348e (0xacd1 bytes) 000904 1783350797.976729 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x301edb18 (stale) Mirror 6: 0x301edb18 (stale) Mirror 7: 0x301edb18 (stale) Mirror 8: 0x4600e245 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x301edb18 (stale) Mirror 6: 0x301edb18 (stale) Mirror 7: 0x301edb18 (stale) Mirror 8: 0x4600e245 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1783350798.341232 write 0x00063b thru 0x001cce (0x1694 bytes) 000908 1783350798.357449 MIRROR_RESYNC 5 mirrors 000910 1783350799.160303 read 0x0229c9 thru 0x025316 (0x294e bytes) 000911 1783350799.168395 mapread 0x015cbb thru 0x017d4c (0x2092 bytes) 000912 1783350799.195946 mapwrite 0x02d24a thru 0x033623 (0x63da bytes) 000913 1783350799.340620 write 0x01abc6 thru 0x01ca7a (0x1eb5 bytes) 000914 1783350799.354358 trunc from 0x040000 to 0x035bd6 (0xa429 bytes) 000915 1783350799.378567 mapread 0x032ec3 thru 0x034922 (0x1a60 bytes) 000916 1783350799.381619 write 0x01073a thru 0x019922 (0x91e9 bytes) 000917 1783350799.406411 mapwrite 0x036034 thru 0x03fa84 (0x9a51 bytes) 000918 1783350799.505378 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa85): Mirror 6: 0xec22016a (stale) Mirror 7: 0xec22016a (stale) Mirror 8: 0xec22016a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa85): Mirror 6: 0xec22016a (stale) Mirror 7: 0xec22016a (stale) Mirror 8: 0xec22016a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa85: skipped 000920 1783350799.924977 mapread 0x023f02 thru 0x025a17 (0x1b16 bytes) 000921 1783350799.945531 mapwrite 0x032169 thru 0x03ce34 (0xaccc bytes) 000922 1783350800.018621 write 0x015681 thru 0x019795 (0x4115 bytes) 000923 1783350800.032642 trunc from 0x03fa85 to 0x0291f9 (0x1688b bytes) 000925 1783350800.053689 trunc from 0x0291fa to 0x0044a4 (0x24d55 bytes) 000926 1783350800.074688 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x44a5): Mirror 6: 0xcca4573 (stale) Mirror 7: 0xcca4573 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x44a5): Mirror 6: 0xcca4573 (stale) Mirror 7: 0xcca4573 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x44a5: skipped 000927 1783350800.332395 mapread 0x001b55 thru 0x004132 (0x25de bytes) 000928 1783350800.348851 mapwrite 0x02ffbb thru 0x030c38 (0xc7e bytes) 000930 1783350800.423235 mapwrite 0x02ea9d thru 0x033258 (0x47bc bytes) 000931 1783350800.483380 mapread 0x0302db thru 0x033258 (0x2f7e bytes) 000932 1783350800.484573 trunc from 0x033259 to 0x0275a8 (0xbcb0 bytes) 000933 1783350800.499949 trunc from 0x0275a9 to 0x00b8ab (0x1bcfd bytes) 000934 1783350800.518947 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb8ac): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb8ac): Mirror 6: 0xe76c3678 (stale) Mirror 7: 0xe76c3678 (stale) CRC-32 checksum value for chunk [0, 0xb8ac): Mirror 1: 0xc8dc0a66 Mirror 8: 0xc8dc0a66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb8ac: skipped 000937 1783350801.258666 write 0x0258a8 thru 0x02751b (0x1c74 bytes) 000939 1783350801.355620 mapwrite 0x02ca3b thru 0x02cd78 (0x33e bytes) 000941 1783350801.427858 write 0x0286cc thru 0x029ae9 (0x141e bytes) 000943 1783350801.445163 mapwrite 0x02a273 thru 0x030050 (0x5dde bytes) 000944 1783350801.526047 mapread 0x01b236 thru 0x021217 (0x5fe2 bytes) 000945 1783350801.544392 mapwrite 0x035772 thru 0x03ffff (0xa88e bytes) 000949 1783350801.626476 mapwrite 0x003582 thru 0x01054b (0xcfca bytes) 000950 1783350801.708490 trunc from 0x040000 to 0x0224d1 (0x1db2e bytes) 000952 1783350801.736054 mapwrite 0x03021c thru 0x034147 (0x3f2c bytes) 000953 1783350801.819278 read 0x0034fa thru 0x004d26 (0x182d bytes) 000954 1783350801.820431 mapread 0x00cddd thru 0x011899 (0x4abd bytes) 000956 1783350801.829709 mapread 0x0149ae thru 0x022639 (0xdc8c bytes) 000958 1783350801.833424 write 0x02de96 thru 0x033671 (0x57dc bytes) 000964 1783350801.839683 write 0x033511 thru 0x03d53d (0xa02d bytes) 000965 1783350801.841196 read 0x00b497 thru 0x00e0ed (0x2c57 bytes) 000967 1783350801.842649 read 0x031bc9 thru 0x0390bb (0x74f3 bytes) 000968 1783350801.843321 mapwrite 0x024311 thru 0x0255eb (0x12db bytes) 000970 1783350801.908200 read 0x02be7c thru 0x03b8c3 (0xfa48 bytes) 000972 1783350801.910032 read 0x0067a9 thru 0x013a71 (0xd2c9 bytes) 000973 1783350801.910908 trunc from 0x03d53e to 0x0203c7 (0x1d176 bytes) 000974 1783350801.932124 read 0x005c7c thru 0x00ecb4 (0x9039 bytes) 000975 1783350801.933021 trunc from 0x0203c8 to 0x00a02b (0x1639c bytes) 000977 1783350801.954585 trunc from 0x00a02c to 0x02cc60 (0x22c35 bytes) 000981 1783350801.982923 read 0x00a182 thru 0x0190af (0xef2e bytes) 000982 1783350801.992768 read 0x00a17a thru 0x00d1e9 (0x3070 bytes) 000986 1783350801.995939 read 0x0263a0 thru 0x02cc60 (0x68c1 bytes) 000987 1783350801.997443 trunc from 0x02cc61 to 0x006b5c (0x26104 bytes) 000989 1783350802.026643 read 0x001806 thru 0x006b5c (0x5357 bytes) 000990 1783350802.028030 mapread 0x0024ce thru 0x003ea4 (0x19d7 bytes) 000991 1783350802.029161 write 0x0039c1 thru 0x004538 (0xb78 bytes) 000992 1783350802.036337 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b5d): Mirror 6: 0xfe86b187 (stale) Mirror 7: 0xfe86b187 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b5d): Mirror 6: 0xfe86b187 (stale) Mirror 7: 0xfe86b187 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b5d: skipped 000993 1783350802.321530 write 0x00a28d thru 0x00ba9c (0x1810 bytes) 000994 1783350802.323254 read 0x0088a8 thru 0x00ba9c (0x31f5 bytes) 000995 1783350802.330068 write 0x03c5ed thru 0x03ffff (0x3a13 bytes) 000998 1783350802.339151 trunc from 0x040000 to 0x0177e2 (0x2881d bytes) 000999 1783350802.361454 trunc from 0x0177e3 to 0x01a3b5 (0x2bd3 bytes) 001000 1783350802.392057 mapwrite 0x006085 thru 0x0075c4 (0x1540 bytes) All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 11:13:26 (1783350806) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 11:13:28 (1783350808) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201212 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151504 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 11:13:35 (1783350815) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4044576 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 484608 .. Extending file size to 4997728 .. Extending file size to 9043936 .. Extending file size to 7912320 .. Extending file size to 7972032 .. Extending file size to 8479904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9238144 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57009 57011 57013 57014 57015 done 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) 192.168.204.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:14:51 (1783350891) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2667936 .. Extending file size to 6905024 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8451008 .. Extending file size to 6964512 .. failed Extending file size to 9053856 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8013312 .. failed Extending file size to 8258016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8916288 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9065248 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9007552 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9120704 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58575 58576 58577 58579 58581 done 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) 192.168.204.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:16:05 (1783350965) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.055603 s, 37.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219611 s, 9.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4f72915 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:16:19 (1783350979) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0810451 s, 25.9 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 11:16:24 (1783350984) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:16:29 (1783350989) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160429 s, 13.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:16:35 (1783350995) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188464 s, 11.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:16:41 (1783351001) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118825 s, 17.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x331:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:17:10 (1783351030) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104848 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x332:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x332:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:18:05 (1783351085) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21173 s, 9.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:18:59 (1783351139) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. Creating new pool test_204f oleg437-server: Pool lustre.test_204f created Adding targets to pool oleg437-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg437-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152928 s, 13.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x335:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x335:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg437-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg437-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg437-server: Pool lustre.test_204f destroyed PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:20:06 (1783351206) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:20:12 (1783351212) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:20:18 (1783351218) oleg437-server: Pool lustre.test_206 created Adding targets to pool oleg437-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg437-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg437-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg437-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg437-server: Pool lustre.test_206 destroyed PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:20:37 (1783351237) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728091 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460255 s, 22.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x674a1f84 Mirror 3: 0x674a1f84 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:20:45 (1783351245) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:20:47 (1783351247) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:20:50 (1783351250) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0641293 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x860000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2020000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2020000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2020000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c40000 Read this (4 bytes): BBBB 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) PASS 209a (25s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:21:15 (1783351275) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0109769 s, 746 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:21:19 (1783351279) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032278 s, 32.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24f lmm_fid: [0x200000404:0x24f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24f lmm_fid: [0x200000404:0x24f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834511 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-06 11:21:20.000000000 -0400 Modify: 2026-07-06 11:21:20.000000000 -0400 Change: 2026-07-06 11:21:21.000000000 -0400 Birth: 2026-07-06 11:21:19.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:21:25 (1783351285) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:21:27 (1783351287) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.359709 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0444694 s, 92.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 6 11:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x251:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 6 11:21 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:21:33 (1783351293) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:21:37 (1783351297) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1574 sec =========== 11:21:43 (1783351303) === sanity-flr: start cleanup 11:21:44 (1783351304) === === sanity-flr: finish cleanup 11:21:48 (1783351308) ===