-----============= acceptance-small: sanity-flr ============----- Thu Jul 16 14:03:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg448-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:04:06 (1784225046) === oleg448-client.virtnet: executing check_config_client /mnt/lustre oleg448-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg448-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff894e11b86800.idle_timeout=debug osc.lustre-OST0001-osc-ffff894e11b86800.idle_timeout=debug disable quota as required oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:04:27 (1784225067) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7586] == sanity-flr test 0a: lfs mirror create with -N option == 14:04:29 (1784225069) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:04:40 (1784225080) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:04:42 (1784225082) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:04:44 (1784225084) PASS 0d (16s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:05:01 (1784225101) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:05:02 (1784225102) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:05:04 (1784225104) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff894e11b86800.stats=clear osc.lustre-OST0001-osc-ffff894e11b86800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:05:09 (1784225109) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:05:16 (1784225116) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:05:21 (1784225121) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:05:24 (1784225124) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:05:29 (1784225129) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 14:05:35 (1784225135) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:05:40 (1784225140) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:05:42 (1784225142) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:05:47 (1784225147) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:05:54 (1784225154) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:05:59 (1784225159) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245303 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099111 s, 10.6 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:06:05 (1784225165) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.265534 s, 59.2 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:06:11 (1784225171) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:06:18 (1784225178) Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:06:54 (1784225214) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0457031 s, 45.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (21s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:07:15 (1784225235) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (40s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:07:55 (1784225275) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.77117 s, 18.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 16 14:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.687s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.575s user 0m0.001s sys 0m0.184s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.711s user 0m0.002s sys 0m0.175s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894e11b86800.ost_server_uuid in FULL state after 0 sec PASS 33b (50s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:08:45 (1784225325) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:08:47 (1784225327) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:08:49 (1784225329) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 14:08:51 (1784225331) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 14:08:55 (1784225335) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.387247 s, 32.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:09:03 (1784225343) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199215 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159083 s, 65.9 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:09:06 (1784225346) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:09:11 (1784225351) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00285572 s, 359 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000974203 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000951361 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000928612 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 14:09:15 (1784225355) create test files with size 15760851 .. 05ebb73ab85a3c6654f1a70f9bb9b9e4 - e21ac547f4a12b99e2e36267ef3081d0 - e4bee6f5e577165b4b5367dc9441cef2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 14:09:25 (1784225365) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:09:31 (1784225371) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:09:34 (1784225374) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548383 s, 19.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260686 s, 40.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 14:09:38 (1784225378) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0769595 s, 27.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.176758 s, 23.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143625 s, 14.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 14:09:46 (1784225386) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 14:09:47 (1784225387) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:09:49 (1784225389) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189019 s, 55.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 14:09:53 (1784225393) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:09:54 (1784225394) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:09:59 (1784225399) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.135521 s, 77.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:10:14 (1784225414) llite.lustre-ffff894e301a9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112416 s, 93.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff894e03146000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:10:21 (1784225421) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (19s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:10:40 (1784225440) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (19s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:10:59 (1784225459) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:11:01 (1784225461) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0840985 s, 37.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0731305 s, 43.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x272fc75 (nosync,stale) Mirror 2: 0x272fc75 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 826ff00877e5b4d4268b71b918fd6f16 - mirror 1 checksum: 4956d6974852238ecdba06734c399386 - mirror 2 checksum: 826ff00877e5b4d4268b71b918fd6f16 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T14:11:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T14:11:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 826ff00877e5b4d4268b71b918fd6f16 - mirror 1 checksum: 4956d6974852238ecdba06734c399386 - mirror 2 checksum: 4956d6974852238ecdba06734c399386 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T14:11:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T14:11:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:11:06 (1784225466) Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 14:11:10 (1784225470) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server Failover mds1 to oleg448-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 14:11:26 (1784225486) targets are mounted 14:11:26 (1784225486) facet_failover done oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202511 s, 51.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:11:36 (1784225496) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:11:37 (1784225497) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:11:39 (1784225499) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00422876 s, 3.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:11:42 (1784225502) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:11:44 (1784225504) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:11:45 (1784225505) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:12:05 (1784225525) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:12:39 (1784225559) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (29s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:13:08 (1784225588) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:13:35 (1784225615) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 14:13:44 (1784225624) mirror create pid 53836 mirror split pid 53837 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7769: 53836 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53837 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:14:51 (1784225691) Seed set to 5693 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x349b9 000001 1784225693.675392 trunc from 00000000 to 0x0349b8 (0x349b9 bytes) 000003 1784225693.699792 read 0x011898 thru 0x0141ce (0x2937 bytes) 000004 1784225693.706563 read 0x0113d4 thru 0x0188be (0x74eb bytes) truncating to largest ever: 0x369e7 000006 1784225693.726303 trunc from 0x0349b9 to 0x0369e6 (0x202e bytes) 000007 1784225693.751311 mapwrite 0x020324 thru 0x024202 (0x3edf bytes) 000010 1784225693.807783 mapread 0x006ae8 thru 0x016ae7 (0x10000 bytes) 000012 1784225693.813092 write 0x03230b thru 0x036fdd (0x4cd3 bytes) 000013 1784225693.830943 write 0x01b07d thru 0x020880 (0x5804 bytes) 000015 1784225693.844193 write 0x030d9c thru 0x034800 (0x3a65 bytes) 000016 1784225693.849657 write 0x019008 thru 0x01b6c1 (0x26ba bytes) 000017 1784225693.859106 MIRROR_EXTEND to 2 mirrors 000018 1784225694.815387 mapread 0x01d98f thru 0x027c4c (0xa2be bytes) 000021 1784225694.856088 trunc from 0x036fde to 0x01b330 (0x1bcad bytes) 000023 1784225694.995795 mapread 0x00759d thru 0x015142 (0xdba6 bytes) 000027 1784225695.033903 mapwrite 0x00ab28 thru 0x0113e5 (0x68be bytes) 000028 1784225695.117826 read 0x01872e thru 0x01b330 (0x2c03 bytes) 000029 1784225695.125970 mapread 0x006c60 thru 0x00bec9 (0x526a bytes) 000030 1784225695.134140 MIRROR_RESYNC 2 mirrors 000031 1784225695.742645 read 0x007f79 thru 0x00a90d (0x2995 bytes) truncating to largest ever: 0x3cd5e 000032 1784225695.750248 trunc from 0x01b331 to 0x03cd5d (0x21a2d bytes) 000033 1784225695.878519 write 0x002ea1 thru 0x00e693 (0xb7f3 bytes) 000034 1784225695.894714 mapwrite 0x035b22 thru 0x03ffff (0xa4de bytes) 000035 1784225696.053967 read 0x009633 thru 0x010996 (0x7364 bytes) 000036 1784225696.076404 mapwrite 0x019cd1 thru 0x01b70b (0x1a3b bytes) 000037 1784225696.139255 write 0x000e88 thru 0x003125 (0x229e bytes) 000038 1784225696.140493 write 0x02bbb9 thru 0x031698 (0x5ae0 bytes) 000040 1784225696.145293 write 0x01fd28 thru 0x0201ab (0x484 bytes) 000041 1784225696.151600 read 0x033dfc thru 0x039b47 (0x5d4c bytes) 000043 1784225696.159506 mapwrite 0x002920 thru 0x00bd32 (0x9413 bytes) 000045 1784225696.234548 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd7b20707 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa27c3e65 Mirror 3: 0xa27c3e65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000046 1784225697.143060 mapwrite 0x0301d2 thru 0x036e46 (0x6c75 bytes) 000047 1784225697.283844 read 0x02b0eb thru 0x02f670 (0x4586 bytes) 000049 1784225697.285981 mapwrite 0x00cd99 thru 0x01b33f (0xe5a7 bytes) 000050 1784225697.360812 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa27c3e65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa27c3e65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000051 1784225697.788492 trunc from 0x040000 to 0x000c53 (0x3f3ac bytes) 000052 1784225697.841091 read 0x0001c2 thru 0x0002a2 (0xe1 bytes) 000054 1784225697.859511 MIRROR_RESYNC 2 mirrors 000056 1784225698.724683 trunc from 0x000c54 to 0x01fc76 (0x1f023 bytes) 000059 1784225698.821872 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000060 1784225699.011818 trunc from 0x01fc77 to 0x009aaa (0x161cc bytes) 000062 1784225699.059841 trunc from 0x009aab to 0x02f20d (0x25763 bytes) 000063 1784225699.074584 read 0x0244a2 thru 0x02be2a (0x7989 bytes) 000064 1784225699.092953 MIRROR_EXTEND to 2 mirrors 000066 1784225699.900785 read 0x020ee1 thru 0x02f20d (0xe32d bytes) 000067 1784225699.916156 mapread 0x025fbe thru 0x02f20d (0x9250 bytes) 000068 1784225699.918152 trunc from 0x02f20e to 0x004126 (0x2b0e7 bytes) 000071 1784225700.079766 trunc from 0x004127 to 0x023598 (0x1f472 bytes) 000073 1784225700.108930 trunc from 0x023599 to 0x02aa30 (0x7498 bytes) 000074 1784225700.134923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000075 1784225700.280212 read 0x008465 thru 0x0141f5 (0xbd91 bytes) 000076 1784225700.293828 write 0x03f4a5 thru 0x03ffff (0xb5b bytes) 000077 1784225700.295157 read 0x03b562 thru 0x03f7b8 (0x4257 bytes) 000078 1784225700.338179 mapwrite 0x01836f thru 0x01e16d (0x5dff bytes) 000079 1784225700.426512 trunc from 0x040000 to 0x0388b5 (0x774a bytes) 000080 1784225700.453912 mapwrite 0x00b439 thru 0x00d476 (0x203e bytes) 000081 1784225700.510042 write 0x0293f2 thru 0x0349d5 (0xb5e4 bytes) 000082 1784225700.522114 trunc from 0x0388b6 to 0x03b22f (0x297a bytes) 000083 1784225700.564578 mapwrite 0x021440 thru 0x02bd7d (0xa93e bytes) 000085 1784225700.669343 read 0x001742 thru 0x00983e (0x80fd bytes) 000088 1784225700.691869 MIRROR_EXTEND to 2 mirrors 000092 1784225701.624877 mapread 0x0031bb thru 0x00ea0b (0xb851 bytes) 000093 1784225701.673513 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000094 1784225701.872016 trunc from 0x03b230 to 0x0026db (0x38b54 bytes) 000096 1784225701.922599 trunc from 0x0026dc to 0x00d01a (0xa93f bytes) 000097 1784225701.941914 write 0x032a87 thru 0x03645c (0x39d6 bytes) 000098 1784225701.944263 mapread 0x01e2b5 thru 0x0277f9 (0x9545 bytes) 000099 1784225701.988845 trunc from 0x03645d to 0x02074f (0x15d0d bytes) 000100 1784225702.013661 mapread 0x001f91 thru 0x00fa46 (0xdab6 bytes) 000102 1784225702.035001 mapwrite 0x0330a1 thru 0x036419 (0x3379 bytes) 000103 1784225702.141457 trunc from 0x03641a to 0x00dc2f (0x287ea bytes) 000107 1784225702.181241 mapread 0x002f61 thru 0x004cad (0x1d4d bytes) 000108 1784225702.187099 mapwrite 0x03f851 thru 0x03ffff (0x7af bytes) 000109 1784225702.285268 mapread 0x00eb8f thru 0x010c88 (0x20fa bytes) 000110 1784225702.294139 mapread 0x023bbe thru 0x029bbb (0x5ffe bytes) 000111 1784225702.320792 MIRROR_EXTEND to 2 mirrors 000113 1784225703.147569 mapread 0x00202a thru 0x011b32 (0xfb09 bytes) 000114 1784225703.183112 trunc from 0x040000 to 0x01914d (0x26eb2 bytes) 000115 1784225703.291949 read 0x015c56 thru 0x01914d (0x34f8 bytes) 000116 1784225703.305604 trunc from 0x01914e to 0x0353da (0x1c28d bytes) 000117 1784225703.343151 write 0x027b45 thru 0x02ea9f (0x6f5b bytes) 000118 1784225703.366676 read 0x01fea0 thru 0x02f5c5 (0xf726 bytes) 000119 1784225703.380100 write 0x0398bf thru 0x03ffff (0x6741 bytes) 000120 1784225703.382921 mapwrite 0x02afeb thru 0x035260 (0xa276 bytes) 000127 1784225703.481371 trunc from 0x040000 to 0x01dba8 (0x22457 bytes) 000129 1784225703.520676 mapwrite 0x01fb31 thru 0x020893 (0xd63 bytes) 000131 1784225703.637959 mapread 0x0202eb thru 0x020893 (0x5a9 bytes) 000132 1784225703.647259 MIRROR_RESYNC 2 mirrors 000133 1784225704.383721 trunc from 0x020894 to 0x02d184 (0xc8f1 bytes) 000134 1784225704.470059 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000135 1784225704.651777 read 0x01230d thru 0x016cd7 (0x49cb bytes) 000139 1784225704.669674 write 0x034ce7 thru 0x03ffff (0xb319 bytes) truncating to largest ever: 0x3d1ff 000140 1784225704.672492 trunc from 0x040000 to 0x03d1fe (0x2e01 bytes) 000141 1784225704.701988 MIRROR_EXTEND to 2 mirrors 000144 1784225705.549398 MIRROR_RESYNC 2 mirrors 000146 1784225705.939053 mapwrite 0x016880 thru 0x019fa7 (0x3728 bytes) 000147 1784225706.128928 trunc from 0x03d1ff to 0x01fa35 (0x1d7c9 bytes) truncating to largest ever: 0x3dca4 000148 1784225706.178670 trunc from 0x01fa36 to 0x03dca3 (0x1e26e bytes) 000149 1784225706.201724 mapread 0x010d15 thru 0x01c908 (0xbbf4 bytes) 000151 1784225706.203793 mapread 0x004fdc thru 0x00c68c (0x76b1 bytes) 000153 1784225706.222516 MIRROR_RESYNC 2 mirrors 000154 1784225706.794849 trunc from 0x03dca4 to 0x020d0e (0x1cf95 bytes) 000155 1784225706.939682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000158 1784225707.128973 mapwrite 0x001b43 thru 0x010c46 (0xf104 bytes) 000159 1784225707.222938 mapread 0x015dfc thru 0x020d0e (0xaf13 bytes) 000161 1784225707.243817 mapread 0x01c051 thru 0x020d0e (0x4cbe bytes) 000162 1784225707.245889 write 0x02fe87 thru 0x038726 (0x88a0 bytes) 000163 1784225707.257699 mapwrite 0x031308 thru 0x034caf (0x39a8 bytes) 000165 1784225707.323990 mapread 0x02d92f thru 0x0386fb (0xadcd bytes) 000166 1784225707.337816 read 0x031b45 thru 0x033a6d (0x1f29 bytes) 000168 1784225707.339849 mapwrite 0x003118 thru 0x00f55d (0xc446 bytes) 000169 1784225707.392773 trunc from 0x038727 to 0x03da2a (0x5304 bytes) 000171 1784225707.434161 mapread 0x00d729 thru 0x018bd3 (0xb4ab bytes) 000172 1784225707.437980 trunc from 0x03da2b to 0x03bf30 (0x1afa bytes) 000174 1784225707.458113 mapread 0x015854 thru 0x018c5c (0x3409 bytes) 000180 1784225707.464962 trunc from 0x03bf31 to 0x0367cb (0x5765 bytes) 000181 1784225707.493833 write 0x02ac6d thru 0x02dab9 (0x2e4d bytes) 000182 1784225707.498070 mapwrite 0x0108e9 thru 0x01b0ec (0xa804 bytes) 000184 1784225707.556383 mapread 0x01c407 thru 0x02b79c (0xf396 bytes) 000188 1784225707.574968 write 0x008071 thru 0x00f6bc (0x764c bytes) 000192 1784225707.591082 read 0x017eb5 thru 0x0244c2 (0xc60e bytes) 000193 1784225707.592385 trunc from 0x0367cc to 0x037613 (0xe48 bytes) 000197 1784225707.626664 mapwrite 0x02827b thru 0x02b3a8 (0x312e bytes) 000198 1784225707.709096 write 0x030463 thru 0x033230 (0x2dce bytes) 000199 1784225707.710597 mapwrite 0x028ede thru 0x0383a1 (0xf4c4 bytes) 000200 1784225707.825995 MIRROR_EXTEND to 2 mirrors 000202 1784225708.476146 read 0x02db18 thru 0x0383a1 (0xa88a bytes) 000205 1784225708.499493 mapwrite 0x03737e thru 0x039e69 (0x2aec bytes) 000206 1784225708.717747 trunc from 0x039e6a to 0x0297e7 (0x10682 bytes) 000207 1784225708.738530 mapwrite 0x008baa thru 0x01201d (0x9474 bytes) 000208 1784225708.823062 trunc from 0x0297e8 to 0x023166 (0x6681 bytes) 000209 1784225708.836890 mapread 0x01e81a thru 0x023166 (0x494d bytes) 000215 1784225708.860898 mapread 0x017b5e thru 0x01a490 (0x2933 bytes) 000216 1784225708.862583 write 0x016f7e thru 0x01edaf (0x7e32 bytes) 000217 1784225708.868652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000219 1784225709.087668 read 0x008b15 thru 0x013b30 (0xb01c bytes) 000220 1784225709.103152 read 0x0053d1 thru 0x007f38 (0x2b68 bytes) 000221 1784225709.113695 trunc from 0x023167 to 0x03c690 (0x1952a bytes) 000222 1784225709.130673 trunc from 0x03c691 to 0x038653 (0x403d bytes) 000224 1784225709.161088 write 0x011e7b thru 0x020f85 (0xf10b bytes) 000225 1784225709.164042 write 0x021ad2 thru 0x02e14b (0xc67a bytes) 000228 1784225709.173523 trunc from 0x038654 to 0x030880 (0x7dd3 bytes) 000229 1784225709.203909 write 0x03d4fe thru 0x03fa1d (0x2520 bytes) 000234 1784225709.211559 mapwrite 0x00d635 thru 0x00ebdb (0x15a7 bytes) 000239 1784225709.320798 mapread 0x02787c thru 0x031c0b (0xa390 bytes) 000241 1784225709.334628 mapwrite 0x030961 thru 0x037cbd (0x735d bytes) 000244 1784225709.394398 write 0x020915 thru 0x02c102 (0xb7ee bytes) 000246 1784225709.401711 mapread 0x03c486 thru 0x03fa1d (0x3598 bytes) 000250 1784225709.405721 read 0x02edc0 thru 0x032cd4 (0x3f15 bytes) 000251 1784225709.407421 trunc from 0x03fa1e to 0x02d918 (0x12105 bytes) 000252 1784225709.434592 MIRROR_EXTEND to 2 mirrors 000253 1784225710.166983 mapread 0x0188a2 thru 0x021b49 (0x92a8 bytes) 000254 1784225710.188615 read 0x0208ef thru 0x020e6e (0x580 bytes) 000256 1784225710.189850 write 0x016c40 thru 0x022b5c (0xbf1d bytes) 000258 1784225710.295793 MIRROR_RESYNC 2 mirrors 000261 1784225711.076438 mapread 0x02c0e9 thru 0x02d918 (0x1830 bytes) 000263 1784225711.101171 read 0x010c59 thru 0x0131d4 (0x257c bytes) 000267 1784225711.123097 MIRROR_RESYNC 2 mirrors 000271 1784225711.625974 write 0x00e057 thru 0x01701f (0x8fc9 bytes) 000272 1784225711.757014 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000274 1784225711.935003 mapwrite 0x003c2a thru 0x00c5c3 (0x899a bytes) 000276 1784225712.023717 mapwrite 0x03098a thru 0x0363aa (0x5a21 bytes) 000277 1784225712.119029 trunc from 0x0363ab to 0x039656 (0x32ac bytes) 000279 1784225712.149486 mapwrite 0x0316b5 thru 0x03eb23 (0xd46f bytes) 000281 1784225712.259053 mapread 0x020512 thru 0x02db04 (0xd5f3 bytes) 000284 1784225712.275366 write 0x012350 thru 0x0144aa (0x215b bytes) 000288 1784225712.292668 mapwrite 0x029b44 thru 0x03629b (0xc758 bytes) 000291 1784225712.342713 read 0x01dc22 thru 0x02435b (0x673a bytes) 000297 1784225712.349033 mapread 0x013a29 thru 0x0196c4 (0x5c9c bytes) 000298 1784225712.373054 write 0x003e70 thru 0x0105ce (0xc75f bytes) 000299 1784225712.376667 trunc from 0x03eb24 to 0x03c4d9 (0x264a bytes) 000301 1784225712.406435 MIRROR_EXTEND to 2 mirrors 000302 1784225713.330557 mapwrite 0x01e20e thru 0x02c024 (0xde17 bytes) 000303 1784225713.566595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000305 1784225713.755854 mapwrite 0x0122ba thru 0x01f1bb (0xcf02 bytes) 000306 1784225713.849390 trunc from 0x03c4da to 0x009724 (0x32db5 bytes) 000307 1784225713.871344 read 0x0001d2 thru 0x009724 (0x9553 bytes) 000308 1784225713.890800 MIRROR_EXTEND to 2 mirrors 000311 1784225714.638941 mapwrite 0x0111df thru 0x011781 (0x5a3 bytes) 000312 1784225714.895774 mapwrite 0x002cf8 thru 0x00d66e (0xa977 bytes) 000313 1784225715.005319 MIRROR_RESYNC 2 mirrors 000314 1784225715.750323 MIRROR_RESYNC 2 mirrors 000315 1784225716.202998 trunc from 0x011782 to 0x003287 (0xe4fa bytes) 000316 1784225716.296895 mapread 0x0017e2 thru 0x003287 (0x1aa6 bytes) 000317 1784225716.315944 read 0x002b78 thru 0x003287 (0x710 bytes) 000318 1784225716.321850 MIRROR_RESYNC 2 mirrors 000319 1784225717.008434 MIRROR_RESYNC 2 mirrors 000321 1784225717.471895 mapread 0x001cd0 thru 0x003287 (0x15b8 bytes) 000323 1784225717.495540 read 0x001b9d thru 0x003287 (0x16eb bytes) 000324 1784225717.496397 mapwrite 0x008b90 thru 0x016f89 (0xe3fa bytes) 000325 1784225717.788151 mapread 0x004045 thru 0x0108ee (0xc8aa bytes) 000326 1784225717.789979 read 0x003812 thru 0x00f21f (0xba0e bytes) 000328 1784225717.797856 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000331 1784225718.020088 MIRROR_EXTEND to 2 mirrors 000332 1784225718.764867 trunc from 0x016f8a to 0x00e692 (0x88f7 bytes) 000333 1784225718.869174 read 0x0085e3 thru 0x00dcf2 (0x5710 bytes) 000335 1784225718.881757 mapwrite 0x00c7be thru 0x0141eb (0x7a2e bytes) 000336 1784225719.038099 mapread 0x010e9e thru 0x0141eb (0x334e bytes) 000338 1784225719.042848 read 0x00c185 thru 0x00c6da (0x556 bytes) 000339 1784225719.043768 read 0x00bb90 thru 0x0141eb (0x865c bytes) 000340 1784225719.052133 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x141ec): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141ec): Mirror 2: 0x977d536c (stale) CRC-32 checksum value for chunk [0, 0x141ec): Mirror 1: 0xa379cba6 Mirror 3: 0xa379cba6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141ec: skipped 000341 1784225720.032002 trunc from 0x0141ec to 0x007272 (0xcf79 bytes) 000342 1784225720.143038 mapwrite 0x019f36 thru 0x028d58 (0xee23 bytes) 000344 1784225720.361273 mapread 0x00bd4f thru 0x01435d (0x860f bytes) 000346 1784225720.377384 read 0x00fed8 thru 0x0191ad (0x92d6 bytes) 000347 1784225720.378710 read 0x012aa6 thru 0x01ebc8 (0xc123 bytes) 000349 1784225720.380147 mapwrite 0x003580 thru 0x0060fe (0x2b7f bytes) 000351 1784225720.481377 mapread 0x00c146 thru 0x018a68 (0xc923 bytes) 000352 1784225720.486485 trunc from 0x028d59 to 0x029453 (0x6fb bytes) 000353 1784225720.526739 read 0x005b5e thru 0x011b3c (0xbfdf bytes) 000357 1784225720.532103 read 0x01b391 thru 0x029453 (0xe0c3 bytes) 000358 1784225720.540118 mapread 0x020373 thru 0x0225a1 (0x222f bytes) 000359 1784225720.541808 mapread 0x02293e thru 0x029453 (0x6b16 bytes) 000362 1784225720.544562 mapwrite 0x02bd2e thru 0x02c509 (0x7dc bytes) 000363 1784225720.631490 write 0x03d73b thru 0x03f0cb (0x1991 bytes) 000364 1784225720.633310 mapwrite 0x03886c thru 0x03c449 (0x3bde bytes) 000366 1784225720.689755 trunc from 0x03f0cc to 0x0155cf (0x29afc bytes) 000368 1784225720.716793 mapwrite 0x004f12 thru 0x00d65d (0x874c bytes) 000369 1784225720.787782 mapwrite 0x035989 thru 0x03d7fe (0x7e76 bytes) 000372 1784225720.933974 read 0x03120b thru 0x037cc5 (0x6abb bytes) 000373 1784225720.947642 write 0x021c32 thru 0x0220b2 (0x481 bytes) 000375 1784225720.955882 MIRROR_RESYNC 3 mirrors 000378 1784225721.789998 MIRROR_RESYNC 3 mirrors 000379 1784225722.255068 mapread 0x02ecfe thru 0x03d660 (0xe963 bytes) 000380 1784225722.286758 mapwrite 0x036742 thru 0x036ce2 (0x5a1 bytes) 000383 1784225722.425814 mapread 0x03577d thru 0x0381dc (0x2a60 bytes) 000385 1784225722.437120 MIRROR_RESYNC 3 mirrors 000387 1784225723.073612 write 0x024223 thru 0x031bec (0xd9ca bytes) 000388 1784225723.163447 write 0x015f5f thru 0x023738 (0xd7da bytes) 000391 1784225723.184740 trunc from 0x03d7ff to 0x025c6e (0x17b90 bytes) 000392 1784225723.252758 mapwrite 0x0144be thru 0x01fc78 (0xb7bb bytes) 000394 1784225723.317824 mapwrite 0x01aa9a thru 0x02a0f8 (0xf65f bytes) 000395 1784225723.512189 mapwrite 0x02f105 thru 0x03dbb4 (0xeab0 bytes) 000396 1784225723.679686 trunc from 0x03dbb5 to 0x0041f5 (0x399bf bytes) 000397 1784225723.710347 write 0x0113cd thru 0x020b8e (0xf7c2 bytes) 000399 1784225723.719997 read 0x011f85 thru 0x01f316 (0xd392 bytes) 000400 1784225723.720852 read 0x008131 thru 0x0117fd (0x96cd bytes) 000403 1784225723.747643 MIRROR_RESYNC 3 mirrors 000404 1784225724.632056 mapread 0x010e62 thru 0x01d511 (0xc6b0 bytes) 000405 1784225724.691435 read 0x004706 thru 0x01105e (0xc959 bytes) 000407 1784225724.710649 read 0x00ac1e thru 0x0158c6 (0xaca9 bytes) 000408 1784225724.711990 write 0x0279e0 thru 0x02c3c3 (0x49e4 bytes) 000412 1784225724.796896 write 0x03b8f1 thru 0x03ffff (0x470f bytes) 000413 1784225724.799323 mapread 0x00b318 thru 0x010067 (0x4d50 bytes) 000414 1784225724.827106 mapread 0x03aab9 thru 0x03f78f (0x4cd7 bytes) 000415 1784225724.835239 trunc from 0x040000 to 0x037bfb (0x8404 bytes) 000416 1784225724.892646 mapread 0x035e51 thru 0x037bfb (0x1dab bytes) 000417 1784225724.894535 read 0x002b79 thru 0x00cbca (0xa052 bytes) 000421 1784225724.899284 write 0x006962 thru 0x0078b2 (0xf51 bytes) 000422 1784225724.906409 MIRROR_RESYNC 3 mirrors 000425 1784225725.805533 mapread 0x021d9d thru 0x02fbb1 (0xde15 bytes) 000427 1784225725.857868 MIRROR_RESYNC 3 mirrors 000428 1784225726.303511 read 0x02ca80 thru 0x02dff9 (0x157a bytes) 000429 1784225726.304987 read 0x001044 thru 0x007ea9 (0x6e66 bytes) 000430 1784225726.318132 mapwrite 0x03cee4 thru 0x03ffff (0x311c bytes) 000431 1784225726.481709 mapwrite 0x0234c5 thru 0x023586 (0xc2 bytes) 000434 1784225726.540689 mapread 0x013dfa thru 0x01bdd9 (0x7fe0 bytes) 000435 1784225726.635802 read 0x03b44f thru 0x03ffff (0x4bb1 bytes) 000436 1784225726.651784 MIRROR_RESYNC 3 mirrors 000440 1784225727.423106 mapread 0x005c03 thru 0x00867d (0x2a7b bytes) 000441 1784225727.501185 MIRROR_RESYNC 3 mirrors 000442 1784225727.999772 mapread 0x027477 thru 0x031993 (0xa51d bytes) 000444 1784225728.066746 write 0x0374d2 thru 0x03c789 (0x52b8 bytes) 000446 1784225728.226826 mapread 0x01d857 thru 0x01e668 (0xe12 bytes) 000447 1784225728.249346 read 0x0058c1 thru 0x00904d (0x378d bytes) 000449 1784225728.308115 mapwrite 0x01e8b6 thru 0x022acd (0x4218 bytes) 000450 1784225728.385395 write 0x01a914 thru 0x02a407 (0xfaf4 bytes) 000454 1784225728.404459 trunc from 0x040000 to 0x011835 (0x2e7ca bytes) 000455 1784225728.455486 trunc from 0x011836 to 0x02a738 (0x18f03 bytes) 000456 1784225728.472699 mapread 0x017d2d thru 0x01c019 (0x42ed bytes) 000457 1784225728.483532 mapwrite 0x016328 thru 0x023eb7 (0xdb90 bytes) 000458 1784225728.542913 trunc from 0x02a739 to 0x02e49b (0x3d63 bytes) 000460 1784225728.560308 write 0x030807 thru 0x03efb8 (0xe7b2 bytes) 000462 1784225728.565486 mapwrite 0x00afac thru 0x01a6a5 (0xf6fa bytes) 000463 1784225728.672358 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efb9): Mirror 3: 0x13acd0ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efb9): Mirror 3: 0x13acd0ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efb9: skipped 000466 1784225728.986979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3efb9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efb9): Mirror 3: 0x13acd0ba (stale) CRC-32 checksum value for chunk [0, 0x3efb9): Mirror 1: 0x753fed2b Mirror 4: 0x753fed2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efb9: skipped 000469 1784225730.230138 mapwrite 0x03ce1d thru 0x03ffff (0x31e3 bytes) 000470 1784225730.521622 write 0x0200f0 thru 0x02a4d2 (0xa3e3 bytes) 000472 1784225730.535728 read 0x0290fe thru 0x03234c (0x924f bytes) 000474 1784225730.547991 write 0x03da22 thru 0x03ffff (0x25de bytes) 000475 1784225730.549300 trunc from 0x040000 to 0x0216e4 (0x1e91b bytes) 000477 1784225730.587895 write 0x01c08d thru 0x0233b2 (0x7326 bytes) 000478 1784225730.605249 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x233b3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233b3): Mirror 3: 0xf2bf7810 (stale) Mirror 4: 0xbebfe24e (stale) CRC-32 checksum value for chunk [0, 0x233b3): Mirror 1: 0x328dd794 Mirror 5: 0x328dd794 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233b3: skipped 000479 1784225731.674944 mapread 0x00e157 thru 0x01c732 (0xe5dc bytes) 000481 1784225731.743565 trunc from 0x0233b3 to 0x0347dc (0x1142a bytes) 000482 1784225731.905980 read 0x019b8d thru 0x027cdc (0xe150 bytes) 000485 1784225731.927869 mapwrite 0x017093 thru 0x018ed4 (0x1e42 bytes) 000487 1784225732.033850 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x347dd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347dd): Mirror 3: 0xaf615739 (stale) Mirror 4: 0x93a4e113 (stale) Mirror 5: 0x6799dd4f (stale) CRC-32 checksum value for chunk [0, 0x347dd): Mirror 1: 0xadb02ed6 Mirror 6: 0xadb02ed6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347dd: skipped 000488 1784225733.288551 trunc from 0x0347dd to 0x026929 (0xdeb3 bytes) 000490 1784225733.415152 mapwrite 0x036fd6 thru 0x0379a3 (0x9ce bytes) 000494 1784225733.583095 trunc from 0x0379a4 to 0x017f5d (0x1fa46 bytes) 000495 1784225733.606075 write 0x012f84 thru 0x0143da (0x1457 bytes) 000496 1784225733.621556 mapwrite 0x01813a thru 0x01c2fa (0x41c1 bytes) 000499 1784225733.753033 mapwrite 0x03d1e8 thru 0x03ffff (0x2e18 bytes) 000500 1784225733.856908 trunc from 0x040000 to 0x013618 (0x2c9e7 bytes) 000501 1784225733.888128 read 0x003fa4 thru 0x010a49 (0xcaa6 bytes) 000502 1784225733.902848 write 0x03a0c9 thru 0x03a7ef (0x727 bytes) 000503 1784225733.904840 mapwrite 0x037d6c thru 0x03d064 (0x52f9 bytes) 000504 1784225734.037068 read 0x004a59 thru 0x012953 (0xdefb bytes) 000505 1784225734.038113 read 0x00c257 thru 0x016fb1 (0xad5b bytes) 000506 1784225734.053430 mapwrite 0x01cf55 thru 0x01dc26 (0xcd2 bytes) 000507 1784225734.099013 mapwrite 0x0181bf thru 0x01f131 (0x6f73 bytes) 000509 1784225734.198171 mapread 0x014c0e thru 0x0249b0 (0xfda3 bytes) 000512 1784225734.204047 mapwrite 0x00244d thru 0x00b4fe (0x90b2 bytes) 000514 1784225734.259151 mapwrite 0x025d47 thru 0x028223 (0x24dd bytes) 000515 1784225734.304069 mapread 0x01ca93 thru 0x02c6ee (0xfc5c bytes) 000516 1784225734.312279 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d065): Mirror 3: 0x18a1d149 (stale) Mirror 5: 0x6ffc5810 (stale) Mirror 6: 0xc49b402c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d065): Mirror 3: 0x18a1d149 (stale) Mirror 5: 0x6ffc5810 (stale) Mirror 6: 0xc49b402c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d065: skipped 000517 1784225734.699620 mapwrite 0x007fe1 thru 0x008c5a (0xc7a bytes) 000519 1784225734.775904 mapwrite 0x025e59 thru 0x0358fc (0xfaa4 bytes) 000520 1784225734.870623 trunc from 0x03d065 to 0x02791c (0x15748 bytes) 000521 1784225734.899900 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2791d): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2791d): Mirror 3: 0x48184e48 (stale) Mirror 5: 0x18f2691 (stale) Mirror 6: 0xd55dfcc3 (stale) CRC-32 checksum value for chunk [0, 0x2791d): Mirror 1: 0x542d76c3 Mirror 7: 0x542d76c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2791d: skipped 000522 1784225735.853590 MIRROR_RESYNC 5 mirrors 000524 1784225736.923418 read 0x01bd24 thru 0x01ef20 (0x31fd bytes) 000525 1784225736.934514 read 0x00d205 thru 0x016511 (0x930d bytes) 000526 1784225736.960313 mapwrite 0x0341e4 thru 0x03ffff (0xbe1c bytes) 000528 1784225737.228307 trunc from 0x040000 to 0x01b562 (0x24a9d bytes) 000529 1784225737.262415 mapwrite 0x0258a5 thru 0x0342f9 (0xea55 bytes) 000530 1784225737.553458 mapwrite 0x029646 thru 0x02d534 (0x3eef bytes) 000531 1784225737.644603 read 0x0229e3 thru 0x0290a4 (0x66c2 bytes) 000533 1784225737.662847 read 0x019ea9 thru 0x025597 (0xb6ef bytes) 000534 1784225737.664760 write 0x00bba9 thru 0x017278 (0xb6d0 bytes) 000537 1784225737.677185 MIRROR_RESYNC 5 mirrors 000538 1784225738.818701 read 0x008f40 thru 0x01452d (0xb5ee bytes) 000539 1784225738.831882 write 0x024129 thru 0x026ebc (0x2d94 bytes) 000541 1784225738.908908 write 0x01615b thru 0x019d60 (0x3c06 bytes) 000542 1784225738.926535 write 0x00b9aa thru 0x01aefb (0xf552 bytes) 000543 1784225738.955521 read 0x019d4b thru 0x01b5cf (0x1885 bytes) 000544 1784225738.981888 MIRROR_RESYNC 5 mirrors 000545 1784225740.241435 MIRROR_RESYNC 5 mirrors 000546 1784225740.718039 mapwrite 0x0157a5 thru 0x01ca4e (0x72aa bytes) 000549 1784225740.927430 read 0x0214f3 thru 0x030582 (0xf090 bytes) 000551 1784225740.954888 trunc from 0x0342fa to 0x01798d (0x1c96c bytes) 000552 1784225741.008688 write 0x0138a0 thru 0x015c5f (0x23c0 bytes) 000553 1784225741.021353 MIRROR_RESYNC 5 mirrors 000559 1784225742.174753 write 0x03e34a thru 0x03ffff (0x1cb6 bytes) 000560 1784225742.253453 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe414752f (stale) Mirror 5: 0xe414752f (stale) Mirror 6: 0xe414752f (stale) Mirror 7: 0xe414752f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaa92a8b8 Mirror 8: 0xaa92a8b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000563 1784225743.344936 trunc from 0x040000 to 0x00afb4 (0x3504b bytes) 000564 1784225743.417974 mapwrite 0x016bac thru 0x01a18e (0x35e3 bytes) 000566 1784225743.513330 mapread 0x001455 thru 0x006e7f (0x5a2b bytes) 000569 1784225743.531723 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1a18f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a18f): Mirror 3: 0xc53ae3df (stale) Mirror 5: 0xc53ae3df (stale) Mirror 6: 0xc53ae3df (stale) Mirror 7: 0xc53ae3df (stale) Mirror 8: 0xc53ae3df (stale) CRC-32 checksum value for chunk [0, 0x1a18f): Mirror 1: 0x1d0ffe1c Mirror 9: 0x1d0ffe1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a18f: skipped 000572 1784225744.407372 MIRROR_RESYNC 7 mirrors 000575 1784225745.600773 write 0x029296 thru 0x037afb (0xe866 bytes) 000576 1784225745.722957 MIRROR_RESYNC 7 mirrors 000578 1784225747.162577 write 0x00f5ca thru 0x015d05 (0x673c bytes) 000579 1784225747.308580 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37afc): Mirror 3: 0x1f63fe56 (stale) Mirror 5: 0x1f63fe56 (stale) Mirror 7: 0x1f63fe56 (stale) Mirror 8: 0x1f63fe56 (stale) Mirror 9: 0x1f63fe56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37afc): Mirror 3: 0x1f63fe56 (stale) Mirror 5: 0x1f63fe56 (stale) Mirror 7: 0x1f63fe56 (stale) Mirror 8: 0x1f63fe56 (stale) Mirror 9: 0x1f63fe56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37afc: skipped 000580 1784225747.933995 write 0x01ba5a thru 0x01f8e0 (0x3e87 bytes) 000582 1784225747.947070 trunc from 0x037afc to 0x018983 (0x1f178 bytes) 000583 1784225747.972239 mapwrite 0x014158 thru 0x0183d1 (0x427a bytes) 000585 1784225748.056161 mapwrite 0x0367af thru 0x0376dc (0xf2e bytes) 000586 1784225748.146790 mapread 0x005f6e thru 0x00d8f1 (0x7984 bytes) 000587 1784225748.185241 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376dd): Mirror 3: 0xca5f5252 (stale) Mirror 5: 0xca5f5252 (stale) Mirror 7: 0xca5f5252 (stale) Mirror 9: 0xca5f5252 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376dd): Mirror 3: 0xca5f5252 (stale) Mirror 5: 0xca5f5252 (stale) Mirror 7: 0xca5f5252 (stale) Mirror 9: 0xca5f5252 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376dd: skipped 000588 1784225748.793245 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376dd): Mirror 5: 0xca5f5252 (stale) Mirror 7: 0xca5f5252 (stale) Mirror 9: 0xca5f5252 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376dd): Mirror 5: 0xca5f5252 (stale) Mirror 7: 0xca5f5252 (stale) Mirror 9: 0xca5f5252 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376dd: skipped 000589 1784225749.340657 write 0x007b79 thru 0x00dba7 (0x602f bytes) 000597 1784225749.381632 read 0x02685a thru 0x035e93 (0xf63a bytes) 000598 1784225749.407920 write 0x033d7f thru 0x036ccb (0x2f4d bytes) 000599 1784225749.415024 mapread 0x030ab6 thru 0x035f7e (0x54c9 bytes) 000600 1784225749.416407 trunc from 0x0376dd to 0x01c3f8 (0x1b2e4 bytes) 000602 1784225749.455033 write 0x01efcb thru 0x02a3c7 (0xb3fd bytes) 000603 1784225749.458528 read 0x01080d thru 0x018d6d (0x8561 bytes) 000605 1784225749.480149 trunc from 0x02a3c8 to 0x0232ca (0x70fd bytes) 000610 1784225749.519057 read 0x01ef31 thru 0x0232ca (0x439a bytes) 000611 1784225749.531422 MIRROR_RESYNC 4 mirrors 000612 1784225750.803926 trunc from 0x0232cb to 0x037301 (0x14037 bytes) 000614 1784225750.918800 MIRROR_RESYNC 4 mirrors 000615 1784225751.857983 MIRROR_RESYNC 4 mirrors 000616 1784225752.409867 mapread 0x02748e thru 0x036525 (0xf098 bytes) 000617 1784225752.451423 mapread 0x01c452 thru 0x0233c4 (0x6f73 bytes) 000618 1784225752.469476 mapwrite 0x0371b1 thru 0x03ffff (0x8e4f bytes) 000620 1784225752.701116 MIRROR_RESYNC 4 mirrors 000621 1784225753.594119 MIRROR_RESYNC 4 mirrors 000624 1784225754.175901 write 0x0108f8 thru 0x010e8b (0x594 bytes) 000625 1784225754.264884 mapwrite 0x0144ee thru 0x018aea (0x45fd bytes) 000627 1784225754.382624 mapread 0x008142 thru 0x00f4bf (0x737e bytes) 000628 1784225754.398277 MIRROR_RESYNC 4 mirrors 000630 1784225755.326000 mapwrite 0x00ab29 thru 0x012ed9 (0x83b1 bytes) 000631 1784225755.489109 mapread 0x011017 thru 0x012c3d (0x1c27 bytes) 000632 1784225755.490550 mapwrite 0x00c682 thru 0x01506b (0x89ea bytes) 000633 1784225755.545077 trunc from 0x040000 to 0x0191ae (0x26e51 bytes) 000634 1784225755.611509 mapread 0x005f18 thru 0x008c45 (0x2d2e bytes) 000638 1784225755.624145 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x191af): Mirror 5: 0x6da60d98 (stale) Mirror 9: 0x6da60d98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191af): Mirror 5: 0x6da60d98 (stale) Mirror 9: 0x6da60d98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191af: skipped 000641 1784225756.083142 trunc from 0x0191af to 0x00abd9 (0xe5d5 bytes) 000643 1784225756.104639 mapwrite 0x03b5c8 thru 0x03ffff (0x4a38 bytes) 000648 1784225756.218693 write 0x01d22c thru 0x020506 (0x32db bytes) 000649 1784225756.232978 read 0x01b3da thru 0x027d1b (0xc942 bytes) 000650 1784225756.244942 read 0x004903 thru 0x009310 (0x4a0e bytes) 000654 1784225756.266841 mapwrite 0x007bb7 thru 0x00dbcf (0x6019 bytes) 000655 1784225756.374420 read 0x03296f thru 0x03898f (0x6021 bytes) 000656 1784225756.376838 mapwrite 0x00e55a thru 0x015edb (0x7982 bytes) 000657 1784225756.472412 write 0x02a5c8 thru 0x02f14e (0x4b87 bytes) 000658 1784225756.479085 mapread 0x03bdd4 thru 0x03ffff (0x422c bytes) 000662 1784225756.487696 mapread 0x021b90 thru 0x028d77 (0x71e8 bytes) 000668 1784225756.499827 write 0x008d01 thru 0x00bb3b (0x2e3b bytes) 000670 1784225756.502954 mapwrite 0x01f14b thru 0x02d46a (0xe320 bytes) 000673 1784225756.592564 read 0x0255b8 thru 0x031a60 (0xc4a9 bytes) 000674 1784225756.593458 mapread 0x022de9 thru 0x025d7f (0x2f97 bytes) 000675 1784225756.594609 write 0x033e38 thru 0x03f9a5 (0xbb6e bytes) 000676 1784225756.598404 write 0x01f11c thru 0x024d36 (0x5c1b bytes) 000678 1784225756.614424 MIRROR_RESYNC 3 mirrors 000681 1784225757.521969 mapread 0x015d7f thru 0x01e0e8 (0x836a bytes) 000683 1784225757.547805 trunc from 0x040000 to 0x00c836 (0x337c9 bytes) 000684 1784225757.641429 read 0x002aa4 thru 0x006606 (0x3b63 bytes) 000687 1784225757.656882 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc837): Mirror 5: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc837): Mirror 5: 0xb2dcd05a (stale) Mirror 9: 0xb2dcd05a (stale) CRC-32 checksum value for chunk [0, 0xc837): Mirror 1: 0xb2dcd05a Mirror 10: 0xb2dcd05a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc837: skipped 000689 1784225758.430870 read 0x0051ef thru 0x007d4a (0x2b5c bytes) 000691 1784225758.441057 write 0x0373dd thru 0x03ffff (0x8c23 bytes) 000692 1784225758.589013 trunc from 0x040000 to 0x03af51 (0x50ae bytes) 000697 1784225758.674250 write 0x03ebaf thru 0x03ffff (0x1451 bytes) 000700 1784225758.682655 read 0x0116c9 thru 0x0180ca (0x6a02 bytes) 000701 1784225758.708099 trunc from 0x040000 to 0x02abec (0x15413 bytes) 000702 1784225758.763352 write 0x0123b1 thru 0x0203b8 (0xe008 bytes) 000703 1784225758.766819 read 0x0104eb thru 0x011be5 (0x16fb bytes) 000704 1784225758.768856 mapwrite 0x00e9ad thru 0x01e93c (0xff90 bytes) 000705 1784225758.867480 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2abed): Mirror 5: 0xf15f4479 (stale) Mirror 9: 0xf15f4479 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abed): Mirror 5: 0xf15f4479 (stale) Mirror 9: 0xf15f4479 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abed: skipped 000706 1784225759.265459 mapwrite 0x01d796 thru 0x0283c0 (0xac2b bytes) 000707 1784225759.342700 read 0x0180f9 thru 0x01fcc7 (0x7bcf bytes) 000708 1784225759.343659 trunc from 0x02abed to 0x01b421 (0xf7cb bytes) 000709 1784225759.366081 mapread 0x01604d thru 0x016eb9 (0xe6d bytes) 000710 1784225759.376720 write 0x03fcf6 thru 0x03ffff (0x30a bytes) 000711 1784225759.378838 mapwrite 0x02bd87 thru 0x02c3f9 (0x673 bytes) 000715 1784225759.439686 read 0x00991e thru 0x00e1ec (0x48cf bytes) 000716 1784225759.453646 read 0x03b88e thru 0x03ffff (0x4772 bytes) 000717 1784225759.454583 write 0x00c294 thru 0x00d8fd (0x166a bytes) 000718 1784225759.461439 MIRROR_RESYNC 3 mirrors 000719 1784225760.577193 mapwrite 0x01fb73 thru 0x028f98 (0x9426 bytes) 000724 1784225760.829967 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4937bc55 (stale) Mirror 9: 0x4937bc55 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd200ca97 Mirror 10: 0xd200ca97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000725 1784225761.801104 read 0x0372fd thru 0x03ffff (0x8d03 bytes) 000727 1784225761.824930 trunc from 0x040000 to 0x03ad33 (0x52cc bytes) 000728 1784225761.935645 write 0x0098db thru 0x017123 (0xd849 bytes) 000729 1784225761.946882 mapread 0x02b743 thru 0x0397bb (0xe079 bytes) 000730 1784225761.980454 write 0x019d6f thru 0x025649 (0xb8db bytes) 000732 1784225761.996464 write 0x010788 thru 0x013908 (0x3181 bytes) 000733 1784225761.998760 write 0x016ec7 thru 0x019625 (0x275f bytes) 000735 1784225762.005733 MIRROR_RESYNC 4 mirrors 000737 1784225762.847476 MIRROR_EXTEND to 5 mirrors 000743 1784225763.849701 trunc from 0x03ad34 to 0x00f3f1 (0x2b942 bytes) 000747 1784225763.995264 MIRROR_RESYNC 5 mirrors 000748 1784225765.145132 read 0x004948 thru 0x006825 (0x1ede bytes) 000751 1784225765.152337 write 0x03f94d thru 0x03ffff (0x6b3 bytes) 000753 1784225765.208587 trunc from 0x040000 to 0x00f0bd (0x30f42 bytes) 000754 1784225765.235018 write 0x033ef2 thru 0x039ef7 (0x6006 bytes) 000755 1784225765.239343 write 0x01bd5c thru 0x020a7b (0x4d20 bytes) 000756 1784225765.250079 mapwrite 0x03d78f thru 0x03ffff (0x2871 bytes) 000757 1784225765.352954 mapwrite 0x0160fc thru 0x01b057 (0x4f5c bytes) 000758 1784225765.453616 read 0x035439 thru 0x03781c (0x23e4 bytes) 000761 1784225765.455508 trunc from 0x040000 to 0x006620 (0x399df bytes) 000763 1784225765.479136 write 0x0040b7 thru 0x0106bb (0xc605 bytes) 000764 1784225765.490552 read 0x00a44f thru 0x0106bb (0x626d bytes) 000765 1784225765.491440 mapwrite 0x03afe3 thru 0x03ffff (0x501d bytes) 000767 1784225765.563656 write 0x01a1e8 thru 0x023e59 (0x9c72 bytes) 000769 1784225765.567154 mapread 0x01e1d9 thru 0x02d506 (0xf32e bytes) 000771 1784225765.570612 mapwrite 0x01eb12 thru 0x02a1d4 (0xb6c3 bytes) 000772 1784225765.626548 MIRROR_RESYNC 5 mirrors 000773 1784225766.834449 write 0x030bf3 thru 0x032ee0 (0x22ee bytes) 000774 1784225766.933097 mapwrite 0x02d5f8 thru 0x02e4c4 (0xecd bytes) 000776 1784225767.054231 mapwrite 0x006c67 thru 0x016185 (0xf51f bytes) 000781 1784225767.182301 trunc from 0x040000 to 0x014d44 (0x2b2bb bytes) 000783 1784225767.209050 trunc from 0x014d45 to 0x00a400 (0xa944 bytes) 000785 1784225767.227951 write 0x029938 thru 0x02e31d (0x49e6 bytes) 000786 1784225767.230150 write 0x031db6 thru 0x039b98 (0x7de3 bytes) 000787 1784225767.233229 read 0x00ea86 thru 0x018996 (0x9f11 bytes) 000788 1784225767.240963 mapwrite 0x017341 thru 0x01cbde (0x589e bytes) 000791 1784225767.293803 mapwrite 0x01bbeb thru 0x0215a2 (0x59b8 bytes) 000793 1784225767.339472 trunc from 0x039b99 to 0x01f226 (0x1a972 bytes) 000794 1784225767.359345 write 0x01388f thru 0x0171fd (0x396f bytes) 000796 1784225767.362273 trunc from 0x01f227 to 0x018b2d (0x66f9 bytes) 000801 1784225767.384291 mapread 0x0106a0 thru 0x018b2d (0x848e bytes) 000802 1784225767.385815 mapread 0x017982 thru 0x018b2d (0x11ac bytes) 000803 1784225767.387824 mapwrite 0x01c106 thru 0x027d1f (0xbc1a bytes) 000804 1784225767.473936 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27d20): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27d20): Mirror 5: 0xba10d0b (stale) Mirror 9: 0xba10d0b (stale) Mirror 10: 0xba10d0b (stale) Mirror 11: 0xba10d0b (stale) CRC-32 checksum value for chunk [0, 0x27d20): Mirror 1: 0x184fb536 Mirror 12: 0x184fb536 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27d20: skipped 000806 1784225768.464290 MIRROR_RESYNC 6 mirrors 000808 1784225769.615559 write 0x03363b thru 0x038afa (0x54c0 bytes) 000809 1784225769.703961 mapwrite 0x03b1d4 thru 0x03ffff (0x4e2c bytes) 000810 1784225769.866164 write 0x02802e thru 0x036487 (0xe45a bytes) 000811 1784225769.897467 MIRROR_RESYNC 6 mirrors 000812 1784225771.271952 MIRROR_RESYNC 6 mirrors 000813 1784225771.992938 write 0x01c68c thru 0x025d61 (0x96d6 bytes) 000814 1784225772.084804 mapread 0x006076 thru 0x0077db (0x1766 bytes) truncating to largest ever: 0x3ed57 000815 1784225772.100507 trunc from 0x040000 to 0x03ed56 (0x12a9 bytes) 000817 1784225772.141431 trunc from 0x03ed57 to 0x038a95 (0x62c1 bytes) 000818 1784225772.168719 write 0x031b1b thru 0x03f72f (0xdc15 bytes) 000820 1784225772.182848 mapread 0x013b78 thru 0x022174 (0xe5fd bytes) 000822 1784225772.195510 write 0x01c6af thru 0x02b32e (0xec80 bytes) 000825 1784225772.204910 read 0x00ffe7 thru 0x010977 (0x991 bytes) 000828 1784225772.207318 write 0x011038 thru 0x017c47 (0x6c10 bytes) 000829 1784225772.209930 trunc from 0x03f730 to 0x02ebe0 (0x10b4f bytes) 000832 1784225772.240906 mapwrite 0x021a6d thru 0x028a1c (0x6fb0 bytes) 000833 1784225772.300626 mapwrite 0x00b6bd thru 0x00f8c2 (0x4206 bytes) 000834 1784225772.358600 trunc from 0x02ebe1 to 0x01c158 (0x12a88 bytes) 000835 1784225772.381737 read 0x0036e7 thru 0x00ebcf (0xb4e9 bytes) 000837 1784225772.395713 mapwrite 0x000108 thru 0x009ad9 (0x99d2 bytes) 000838 1784225772.491284 mapwrite 0x033122 thru 0x035650 (0x252f bytes) 000840 1784225772.604399 read 0x033d03 thru 0x034609 (0x907 bytes) 000842 1784225772.606461 write 0x0042be thru 0x00f529 (0xb26c bytes) 000844 1784225772.609283 mapread 0x01e800 thru 0x020727 (0x1f28 bytes) 000846 1784225772.615706 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35651): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35651): Mirror 5: 0x74fcb912 (stale) Mirror 9: 0x74fcb912 (stale) Mirror 10: 0x74fcb912 (stale) Mirror 11: 0x74fcb912 (stale) Mirror 12: 0x74fcb912 (stale) CRC-32 checksum value for chunk [0, 0x35651): Mirror 1: 0x7a2b0b58 Mirror 13: 0x7a2b0b58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35651: skipped 000849 1784225773.955189 write 0x01003f thru 0x016633 (0x65f5 bytes) 000850 1784225774.078412 read 0x01bbfc thru 0x02544a (0x984f bytes) 000851 1784225774.090972 write 0x0065c7 thru 0x00866e (0x20a8 bytes) 000852 1784225774.114318 write 0x02d1ea thru 0x0366a5 (0x94bc bytes) 000853 1784225774.145901 mapwrite 0x0368c4 thru 0x03ffff (0x973c bytes) 000859 1784225774.335157 MIRROR_RESYNC 7 mirrors 000860 1784225775.893255 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000862 1784225776.494762 write 0x03a772 thru 0x03e6ef (0x3f7e bytes) 000863 1784225776.641764 write 0x00e8ba thru 0x0162ad (0x79f4 bytes) 000864 1784225776.670777 trunc from 0x040000 to 0x012408 (0x2dbf7 bytes) 000867 1784225776.725247 mapread 0x0012b2 thru 0x004e15 (0x3b64 bytes) 000868 1784225776.782526 mapwrite 0x0343d8 thru 0x03ffff (0xbc28 bytes) 000869 1784225776.936748 MIRROR_RESYNC 6 mirrors 000870 1784225778.226645 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000871 1784225778.707503 read 0x03194e thru 0x03aa53 (0x9106 bytes) 000875 1784225778.740180 trunc from 0x040000 to 0x010d46 (0x2f2b9 bytes) 000877 1784225778.902750 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x10d47): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d47): Mirror 5: 0xd3ac4387 (stale) Mirror 10: 0xd3ac4387 (stale) Mirror 11: 0xd3ac4387 (stale) Mirror 12: 0xd3ac4387 (stale) CRC-32 checksum value for chunk [0, 0x10d47): Mirror 1: 0xd3ac4387 Mirror 13: 0xd3ac4387 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d47: skipped 000880 1784225780.125967 write 0x016e1d thru 0x01c916 (0x5afa bytes) 000883 1784225780.232044 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c917): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c917): Mirror 5: 0x1e8eaa1 (stale) Mirror 10: 0x1e8eaa1 (stale) Mirror 11: 0x1e8eaa1 (stale) Mirror 12: 0x1e8eaa1 (stale) Mirror 13: 0xcd37265c (stale) CRC-32 checksum value for chunk [0, 0x1c917): Mirror 1: 0xaf92612f Mirror 14: 0xaf92612f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c917: skipped 000884 1784225781.371591 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1c917): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c917): Mirror 5: 0x1e8eaa1 (stale) Mirror 10: 0x1e8eaa1 (stale) Mirror 11: 0x1e8eaa1 (stale) Mirror 12: 0x1e8eaa1 (stale) Mirror 13: 0xcd37265c (stale) CRC-32 checksum value for chunk [0, 0x1c917): Mirror 1: 0xaf92612f Mirror 14: 0xaf92612f Mirror 15: 0xaf92612f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c917: skipped 000885 1784225782.574077 write 0x032c99 thru 0x035d46 (0x30ae bytes) 000888 1784225782.691621 trunc from 0x035d47 to 0x01e26e (0x17ad8 bytes) 000889 1784225782.724580 read 0x01c7ad thru 0x01e26e (0x1ac2 bytes) 000891 1784225782.747213 trunc from 0x01e26f to 0x0137a4 (0xaaca bytes) 000894 1784225782.777378 write 0x020131 thru 0x0287e4 (0x86b4 bytes) 000896 1784225782.784998 mapread 0x02006b thru 0x024cdf (0x4c75 bytes) 000897 1784225782.786614 mapread 0x01fe7e thru 0x0287e4 (0x8967 bytes) 000900 1784225782.795732 read 0x018062 thru 0x01ea39 (0x69d8 bytes) 000903 1784225782.826136 write 0x038c67 thru 0x03ffff (0x7399 bytes) 000904 1784225782.828023 trunc from 0x040000 to 0x019a4d (0x265b2 bytes) 000905 1784225782.868818 MIRROR_RESYNC 8 mirrors 000906 1784225784.610914 trunc from 0x019a4e to 0x02f113 (0x156c6 bytes) 000907 1784225784.722961 mapread 0x0005d6 thru 0x00c548 (0xbf73 bytes) 000909 1784225784.759458 read 0x01491d thru 0x021f86 (0xd66a bytes) 000911 1784225784.778188 write 0x0317e7 thru 0x033006 (0x1820 bytes) 000912 1784225784.780171 write 0x014f76 thru 0x02262d (0xd6b8 bytes) 000913 1784225784.799765 mapread 0x029154 thru 0x033006 (0x9eb3 bytes) 000914 1784225784.823073 write 0x03ec5a thru 0x03f9f3 (0xd9a bytes) 000915 1784225784.844546 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f9f4): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9f4): Mirror 5: 0xbfb5f71f (stale) Mirror 10: 0xbfb5f71f (stale) Mirror 11: 0xbfb5f71f (stale) Mirror 12: 0xbfb5f71f (stale) Mirror 13: 0xbfb5f71f (stale) Mirror 14: 0xbfb5f71f (stale) Mirror 15: 0xbfb5f71f (stale) CRC-32 checksum value for chunk [0, 0x3f9f4): Mirror 1: 0xd0550dad Mirror 16: 0xd0550dad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9f4: skipped 000916 1784225786.314142 MIRROR_RESYNC 9 mirrors 000918 1784225788.122917 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000919 1784225788.732151 mapwrite 0x03738e thru 0x03ffff (0x8c72 bytes) 000921 1784225788.977886 trunc from 0x040000 to 0x006af2 (0x3950d bytes) 000922 1784225789.009578 mapwrite 0x03efdb thru 0x03ffff (0x1025 bytes) 000923 1784225789.188622 read 0x031acf thru 0x03d9ef (0xbf21 bytes) 000924 1784225789.214559 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbf12225c (stale) Mirror 10: 0xbf12225c (stale) Mirror 11: 0xbf12225c (stale) Mirror 13: 0xbf12225c (stale) Mirror 14: 0xbf12225c (stale) Mirror 16: 0xbf12225c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbf12225c (stale) Mirror 10: 0xbf12225c (stale) Mirror 11: 0xbf12225c (stale) Mirror 13: 0xbf12225c (stale) Mirror 14: 0xbf12225c (stale) Mirror 16: 0xbf12225c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1784225789.841064 write 0x0196f7 thru 0x020052 (0x695c bytes) 000927 1784225789.863324 mapwrite 0x026d5d thru 0x036823 (0xfac7 bytes) 000928 1784225789.999731 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbf12225c (stale) Mirror 10: 0xbf12225c (stale) Mirror 11: 0xbf12225c (stale) Mirror 13: 0xbf12225c (stale) Mirror 14: 0xbf12225c (stale) Mirror 16: 0xbf12225c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb025842 Mirror 17: 0xeb025842 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1784225791.430415 read 0x023289 thru 0x02783b (0x45b3 bytes) 000935 1784225791.445655 read 0x00b6a5 thru 0x0155de (0x9f3a bytes) 000937 1784225791.465237 read 0x00575c thru 0x015229 (0xface bytes) 000939 1784225791.466159 write 0x018aa7 thru 0x0209fc (0x7f56 bytes) 000940 1784225791.608083 mapwrite 0x020eb0 thru 0x030917 (0xfa68 bytes) 000942 1784225791.675915 mapread 0x033ae4 thru 0x03f9e3 (0xbf00 bytes) 000944 1784225791.682464 trunc from 0x040000 to 0x028ea1 (0x1715e bytes) 000946 1784225791.710965 mapread 0x01b7ec thru 0x01d0a6 (0x18bb bytes) 000947 1784225791.713733 trunc from 0x028ea2 to 0x0139e0 (0x154c1 bytes) 000948 1784225791.742379 trunc from 0x0139e1 to 0x031376 (0x1d996 bytes) 000949 1784225791.791546 mapwrite 0x02a8c4 thru 0x037eec (0xd629 bytes) 000954 1784225792.106534 MIRROR_RESYNC 8 mirrors 000955 1784225793.833746 read 0x025094 thru 0x02a885 (0x57f2 bytes) 000956 1784225793.860266 mapwrite 0x03cfdc thru 0x03ffff (0x3024 bytes) 000957 1784225794.102710 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11076df (stale) Mirror 10: 0x11076df (stale) Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11076df (stale) Mirror 10: 0x11076df (stale) Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000958 1784225794.874643 mapread 0x00e17e thru 0x010b60 (0x29e3 bytes) 000959 1784225794.897984 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11076df (stale) Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11076df (stale) Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000961 1784225795.424025 mapread 0x02301a thru 0x02a0ef (0x70d6 bytes) 000962 1784225795.494212 write 0x025854 thru 0x02cd8e (0x753b bytes) 000964 1784225795.497892 read 0x01ca23 thru 0x01f2cb (0x28a9 bytes) 000966 1784225795.528641 read 0x02511f thru 0x02cabf (0x79a1 bytes) 000967 1784225795.535624 write 0x00057b thru 0x004e04 (0x488a bytes) 000968 1784225795.586513 read 0x02e1ab thru 0x032c90 (0x4ae6 bytes) 000969 1784225795.606866 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x11076df (stale) Mirror 13: 0x11076df (stale) Mirror 16: 0x11076df (stale) Mirror 17: 0x11076df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000970 1784225796.273990 mapwrite 0x017612 thru 0x0207c0 (0x91af bytes) 000971 1784225796.453698 read 0x01221a thru 0x01f267 (0xd04e bytes) 000972 1784225796.457665 mapwrite 0x005a77 thru 0x00d7f3 (0x7d7d bytes) 000973 1784225796.589663 write 0x00574a thru 0x00e8cc (0x9183 bytes) 000976 1784225796.601795 mapwrite 0x01015b thru 0x0128c5 (0x276b bytes) 000977 1784225796.674309 trunc from 0x040000 to 0x02f48f (0x10b70 bytes) 000978 1784225796.704344 MIRROR_RESYNC 5 mirrors 000979 1784225798.056602 write 0x032ff9 thru 0x036147 (0x314f bytes) 000980 1784225798.123948 mapwrite 0x006717 thru 0x0070f2 (0x9dc bytes) 000981 1784225798.209820 write 0x021234 thru 0x022db1 (0x1b7e bytes) 000982 1784225798.240913 MIRROR_RESYNC 5 mirrors 000983 1784225799.364395 trunc from 0x036148 to 0x010af7 (0x25650 bytes) 000987 1784225799.564095 write 0x026bda thru 0x035d03 (0xf12a bytes) 000988 1784225799.568277 trunc from 0x035d04 to 0x002d63 (0x32fa0 bytes) 000989 1784225799.612124 trunc from 0x002d64 to 0x01cf89 (0x1a226 bytes) 000991 1784225799.646840 trunc from 0x01cf8a to 0x037466 (0x1a4dd bytes) 000992 1784225799.680270 write 0x02bd66 thru 0x034251 (0x84ec bytes) 000993 1784225799.711880 trunc from 0x037467 to 0x011810 (0x25c56 bytes) 000995 1784225799.742317 mapwrite 0x03eb8d thru 0x03ffff (0x1473 bytes) 000996 1784225799.861472 read 0x03af6c thru 0x03dac4 (0x2b59 bytes) 000997 1784225799.875890 trunc from 0x040000 to 0x030946 (0xf6b9 bytes) 000999 1784225799.898986 write 0x02ef06 thru 0x0333d9 (0x44d4 bytes) All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 14:16:45 (1784225805) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 14:16:48 (1784225808) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309778 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238842 s, 4.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 14:16:58 (1784225818) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1291168 .. Extending file size to 1375552 .. Extending file size to 6965984 .. Extending file size to 2150496 .. Extending file size to 2626816 .. Extending file size to 8208768 .. Extending file size to 8349056 .. Extending file size to 8753088 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8878976 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8619488 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56977 56979 56980 56981 56983 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:18:22 (1784225902) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7122176 .. Extending file size to 3322624 .. done Extending file size to 9146144 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7380128 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7482688 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9026176 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58275 58277 58278 58280 58281 failed 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:19:43 (1784225983) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133583 s, 15.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334985 s, 6.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x38123b42 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:20:00 (1784226000) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100639 s, 20.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 14:20:06 (1784226006) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:20:14 (1784226014) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182547 s, 11.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:20:21 (1784226021) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16874 s, 12.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:20:28 (1784226028) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188105 s, 11.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:20:59 (1784226059) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952412 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:21:53 (1784226113) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.382598 s, 5.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:22:50 (1784226170) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. Creating new pool test_204f oleg448-server: Pool lustre.test_204f created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135672 s, 15.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg448-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:23:58 (1784226238) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:24:04 (1784226244) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:24:10 (1784226250) oleg448-server: Pool lustre.test_206 created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg448-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:24:32 (1784226272) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669769 s, 15.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329 s, 31.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xafbb58c6 Mirror 3: 0xafbb58c6 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:24:40 (1784226280) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:24:43 (1784226283) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:24:46 (1784226286) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.139141 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cf0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1590000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1590000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1590000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ed0000 Read this (4 bytes): BBBB 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:25:14 (1784226314) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0214508 s, 382 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:25:19 (1784226319) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483199 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834445 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-16 14:25:20.000000000 -0400 Modify: 2026-07-16 14:25:20.000000000 -0400 Change: 2026-07-16 14:25:21.000000000 -0400 Birth: 2026-07-16 14:25:21.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:25:25 (1784226325) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:25:28 (1784226328) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.483027 s, 84.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0481093 s, 85.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 16 14:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 16 14:25 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:25:34 (1784226334) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:25:40 (1784226340) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1310 sec =========== 14:25:46 (1784226346) === sanity-flr: start cleanup 14:25:47 (1784226347) === === sanity-flr: finish cleanup 14:25:52 (1784226352) ===