-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 10:34:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg414-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:35:20 (1789050920) === oleg414-client.virtnet: executing check_config_client /mnt/lustre oleg414-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg414-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9692d1901800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9692d1901800.idle_timeout=debug disable quota as required oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:36:00 (1789050960) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanity-flr test 0a: lfs mirror create with -N option == 10:36:03 (1789050963) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:36:21 (1789050981) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:36:25 (1789050985) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:36:29 (1789050989) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:36:54 (1789051014) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:36:58 (1789051018) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:37:02 (1789051022) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9692d1901800.stats=clear osc.lustre-OST0001-osc-ffff9692d1901800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:37:10 (1789051030) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:37:23 (1789051043) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:37:34 (1789051054) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:37:38 (1789051058) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:37:47 (1789051067) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 10:37:57 (1789051077) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:38:04 (1789051084) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:38:08 (1789051088) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:38:14 (1789051094) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:38:24 (1789051104) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:38:31 (1789051111) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378573 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150226 s, 7.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:38:41 (1789051121) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.142453 s, 36.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:38:50 (1789051130) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:39:02 (1789051142) Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (88s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:40:30 (1789051230) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.906046 s, 12.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:41:22 (1789051282) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (100s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:43:02 (1789051382) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.58585 s, 11.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 10:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.263s user 0m0.001s sys 0m0.195s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.370s user 0m0.001s sys 0m0.195s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.465s user 0m0.001s sys 0m0.203s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9692d1901800.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:44:44 (1789051484) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:44:48 (1789051488) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:44:51 (1789051491) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:44:54 (1789051494) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 10:45:03 (1789051503) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.584866 s, 21.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:45:18 (1789051518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199078 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469862 s, 22.3 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:45:26 (1789051526) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:45:36 (1789051536) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00269796 s, 380 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121332 s, 844 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000958991 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00189562 s, 540 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 10:45:47 (1789051547) create test files with size 15750589 .. 4e7893bcf1ea39ce3ee00066758b224c - 2d3432b0714756f4729476c6a426a638 - a2de95056102d5e60935eeddee9947f6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:46:09 (1789051569) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:46:27 (1789051587) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:46:34 (1789051594) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122246 s, 8.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386803 s, 27.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:46:42 (1789051602) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17389 s, 12.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.408976 s, 10.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255649 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 10:47:01 (1789051621) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 10:47:06 (1789051626) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:47:11 (1789051631) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178242 s, 58.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 10:47:19 (1789051639) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:47:22 (1789051642) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:47:32 (1789051652) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.320353 s, 32.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:47:44 (1789051664) llite.lustre-ffff9692d14f4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.256781 s, 40.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9692c92a1000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:47:58 (1789051678) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:48:35 (1789051715) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:49:11 (1789051751) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 10:49:14 (1789051754) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.277224 s, 11.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.175391 s, 17.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc5ad136b (nosync,stale) Mirror 2: 0xc5ad136b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 48ab82ad47043c8c4faae34bd2bf215a - mirror 1 checksum: 4d46724af660f9a4fd9d27af295e69da - mirror 2 checksum: 48ab82ad47043c8c4faae34bd2bf215a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T10:49:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T10:49:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 48ab82ad47043c8c4faae34bd2bf215a - mirror 1 checksum: 4d46724af660f9a4fd9d27af295e69da - mirror 2 checksum: 4d46724af660f9a4fd9d27af295e69da - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T10:49:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T10:49:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:49:28 (1789051768) Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 10:49:36 (1789051776) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 10:50:00 (1789051800) targets are mounted 10:50:00 (1789051800) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327506 s, 32.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:50:18 (1789051818) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:50:23 (1789051823) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:50:27 (1789051827) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0118762 s, 1.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:50:36 (1789051836) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:50:40 (1789051840) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.012895 s, 318 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.086722 s, 47.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:50:49 (1789051849) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:51:13 (1789051873) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:51:41 (1789051901) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:52:06 (1789051926) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:52:31 (1789051951) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 10:52:38 (1789051958) mirror create pid 54350 mirror split pid 54351 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54350 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54351 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 10:53:45 (1789052025) Seed set to 2028 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x29d6e 000002 1789052028.490055 trunc from 00000000 to 0x029d6d (0x29d6e bytes) 000003 1789052028.532089 MIRROR_EXTEND to 2 mirrors 000005 1789052029.656097 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000007 1789052029.776907 mapwrite 0x035fb4 thru 0x03eb59 (0x8ba6 bytes) 000009 1789052029.899397 read 0x002cb4 thru 0x003bf8 (0xf45 bytes) 000010 1789052029.909552 mapread 0x02296f thru 0x0230f0 (0x782 bytes) 000014 1789052029.923704 mapwrite 0x022f55 thru 0x02c46d (0x9519 bytes) 000019 1789052030.084947 mapwrite 0x025aa7 thru 0x02bda4 (0x62fe bytes) 000020 1789052030.320814 MIRROR_EXTEND to 2 mirrors 000021 1789052031.211810 punch from 0x036058 to 0x03eb59 (0x8b02 bytes) 000023 1789052031.343498 MIRROR_RESYNC 2 mirrors 000024 1789052032.300736 mapwrite 0x0398ce thru 0x03a431 (0xb64 bytes) 000025 1789052032.473833 mapread 0x02f01a thru 0x02ffc6 (0xfad bytes) 000028 1789052032.487509 write 0x031999 thru 0x037e6e (0x64d6 bytes) 000032 1789052032.491837 mapread 0x02b626 thru 0x037486 (0xbe61 bytes) 000033 1789052032.493992 punch from 0x021044 to 0x02dbd2 (0xcb8f bytes) 000034 1789052032.510468 read 0x019010 thru 0x028c27 (0xfc18 bytes) 000035 1789052032.528584 read 0x038e4a thru 0x039bba (0xd71 bytes) 000036 1789052032.529372 read 0x020de2 thru 0x0297fb (0x8a1a bytes) 000037 1789052032.565631 punch from 0x00a23d to 0x00e913 (0x46d7 bytes) truncating to largest ever: 0x2b405 000038 1789052032.575176 trunc from 0x03eb5a to 0x02b404 (0x13755 bytes) 000040 1789052032.657694 punch from 0x01f454 to 0x0289b8 (0x9565 bytes) 000041 1789052032.666008 trunc from 0x02b405 to 0x01498e (0x16a76 bytes) 000042 1789052032.714479 read 0x013e25 thru 0x01498e (0xb6a bytes) 000044 1789052032.716899 mapread 0x005fc0 thru 0x011995 (0xb9d6 bytes) 000046 1789052032.731183 mapwrite 0x00a826 thru 0x015c79 (0xb454 bytes) 000047 1789052032.864806 write 0x007b0b thru 0x00b971 (0x3e67 bytes) 000048 1789052032.869746 write 0x01575a thru 0x0157e8 (0x8f bytes) 000049 1789052032.870876 mapwrite 0x025a16 thru 0x028b21 (0x310c bytes) 000051 1789052032.969868 write 0x030091 thru 0x03b055 (0xafc5 bytes) 000053 1789052032.972973 punch from 0x017eb7 to 0x01d7c0 (0x590a bytes) 000054 1789052032.988497 read 0x0274e0 thru 0x036dba (0xf8db bytes) 000055 1789052032.998163 read 0x032912 thru 0x03b055 (0x8744 bytes) 000061 1789052033.001729 mapread 0x005da0 thru 0x007788 (0x19e9 bytes) 000062 1789052033.002845 trunc from 0x03b056 to 0x0180fc (0x22f59 bytes) 000063 1789052033.021876 read 0x0062f3 thru 0x007478 (0x1186 bytes) 000064 1789052033.024120 punch from 0x00c615 to 0x00cd67 (0x753 bytes) 000065 1789052033.028807 mapwrite 0x00393b thru 0x007323 (0x39e9 bytes) 000066 1789052033.085957 punch from 0x013b80 to 0x01561f (0x1aa0 bytes) 000067 1789052033.090052 write 0x033eb4 thru 0x03d6c3 (0x9810 bytes) 000071 1789052033.094550 punch from 0x02d046 to 0x02f43f (0x23fa bytes) 000072 1789052033.107606 punch from 0x03a89a to 0x03d6c3 (0x2e2a bytes) 000073 1789052033.112695 write 0x03a15f thru 0x03ffff (0x5ea1 bytes) 000074 1789052033.137225 punch from 0x019ac1 to 0x01f3df (0x591f bytes) 000075 1789052033.149797 punch from 0x02df84 to 0x03aa9b (0xcb18 bytes) 000076 1789052033.158622 mapread 0x0256c0 thru 0x0277cf (0x2110 bytes) 000078 1789052033.162938 trunc from 0x040000 to 0x00811c (0x37ee3 bytes) truncating to largest ever: 0x2eeca 000080 1789052033.185292 trunc from 0x00811d to 0x02eec9 (0x26dad bytes) 000081 1789052033.208543 punch from 0x02ae6c to 0x02eec9 (0x405e bytes) 000082 1789052033.212240 mapread 0x01c698 thru 0x01fcad (0x3616 bytes) 000083 1789052033.230200 mapwrite 0x032805 thru 0x03523c (0x2a38 bytes) 000084 1789052033.299970 mapwrite 0x012f1b thru 0x01a1fd (0x72e3 bytes) 000085 1789052033.478190 read 0x0124f1 thru 0x01ceb0 (0xa9c0 bytes) 000086 1789052033.525996 mapread 0x014a56 thru 0x018d5d (0x4308 bytes) 000087 1789052033.530512 read 0x00a5a5 thru 0x00ce4a (0x28a6 bytes) 000088 1789052033.535489 write 0x00d8a4 thru 0x01181d (0x3f7a bytes) 000089 1789052033.545235 punch from 0x03349f to 0x0337bf (0x321 bytes) 000091 1789052033.579747 trunc from 0x03523d to 0x011819 (0x23a23 bytes) 000093 1789052033.623355 write 0x00ce34 thru 0x01927e (0xc44b bytes) truncating to largest ever: 0x35ebf 000094 1789052033.626090 trunc from 0x01927f to 0x035ebe (0x1cc40 bytes) 000096 1789052033.650487 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000098 1789052033.830936 punch from 0x02e3d5 to 0x035ebe (0x7aea bytes) 000099 1789052033.836201 trunc from 0x035ebf to 0x034fb9 (0xf05 bytes) 000100 1789052033.850680 mapread 0x00943d thru 0x011172 (0x7d36 bytes) 000101 1789052033.870343 trunc from 0x034fba to 0x0312de (0x3cdb bytes) 000102 1789052033.885259 punch from 0x003ac6 to 0x00a71d (0x6c58 bytes) 000103 1789052033.889847 trunc from 0x0312df to 0x0204ad (0x10e31 bytes) 000105 1789052033.907550 write 0x0059a6 thru 0x011ffb (0xc656 bytes) truncating to largest ever: 0x39380 000107 1789052033.914676 trunc from 0x0204ae to 0x03937f (0x18ed2 bytes) 000109 1789052033.929761 write 0x035c3d thru 0x03be20 (0x61e4 bytes) 000111 1789052033.936713 mapread 0x019cf7 thru 0x01b5e1 (0x18eb bytes) 000112 1789052033.945992 read 0x0331ce thru 0x03831d (0x5150 bytes) 000115 1789052033.951896 trunc from 0x03be21 to 0x0002cb (0x3bb55 bytes) 000117 1789052033.972839 trunc from 0x0002cc to 0x012063 (0x11d98 bytes) 000118 1789052033.994415 read 0x010963 thru 0x012063 (0x1701 bytes) 000121 1789052034.005832 write 0x01096f thru 0x0208f1 (0xff83 bytes) 000122 1789052034.008226 trunc from 0x0208f2 to 0x02c88a (0xbf99 bytes) 000124 1789052034.023888 punch from 0x008c5a to 0x00b334 (0x26db bytes) 000126 1789052034.036844 punch from 0x0134a0 to 0x01f462 (0xbfc3 bytes) 000127 1789052034.042476 mapread 0x01ea7d thru 0x0257b3 (0x6d37 bytes) 000128 1789052034.052884 punch from 0x029e1c to 0x02c88a (0x2a6f bytes) 000131 1789052034.059046 punch from 0x00f20a to 0x015226 (0x601d bytes) 000133 1789052034.064793 trunc from 0x02c88b to 0x014b20 (0x17d6a bytes) 000134 1789052034.083685 write 0x0350c4 thru 0x03b8f3 (0x6830 bytes) 000135 1789052034.085071 punch from 0x00eba5 to 0x0161cb (0x7627 bytes) 000136 1789052034.098959 punch from 0x013506 to 0x021c6d (0xe768 bytes) 000137 1789052034.105491 trunc from 0x03b8f4 to 0x03847c (0x3477 bytes) 000139 1789052034.126692 write 0x01321f thru 0x01d83d (0xa61f bytes) 000143 1789052034.149959 mapwrite 0x031ae8 thru 0x0359ed (0x3f06 bytes) 000144 1789052034.189754 punch from 0x0095bf to 0x015253 (0xbc95 bytes) 000145 1789052034.199755 mapwrite 0x00ff17 thru 0x0139d7 (0x3ac1 bytes) 000146 1789052034.282994 trunc from 0x03847d to 0x015c90 (0x227ec bytes) 000147 1789052034.299789 mapread 0x00d770 thru 0x014036 (0x68c7 bytes) 000148 1789052034.316554 mapread 0x015a46 thru 0x015c90 (0x24b bytes) 000150 1789052034.322936 mapread 0x0117bc thru 0x015c90 (0x44d5 bytes) 000152 1789052034.324374 trunc from 0x015c91 to 0x035737 (0x1faa7 bytes) 000153 1789052034.341846 mapread 0x01cddc thru 0x02c9ef (0xfc14 bytes) 000155 1789052034.454435 read 0x013718 thru 0x017e8c (0x4775 bytes) 000156 1789052034.463755 write 0x006285 thru 0x00a475 (0x41f1 bytes) 000157 1789052034.464931 read 0x02ea9d thru 0x033c17 (0x517b bytes) 000158 1789052034.466040 write 0x005778 thru 0x012090 (0xc919 bytes) 000159 1789052034.467747 trunc from 0x035738 to 0x00e4a3 (0x27294 bytes) 000160 1789052034.480522 mapwrite 0x020c6c thru 0x028621 (0x79b6 bytes) 000161 1789052034.585510 read 0x00a083 thru 0x00fbb3 (0x5b31 bytes) 000162 1789052034.590188 mapwrite 0x02fca7 thru 0x03d503 (0xd85d bytes) 000165 1789052034.799478 punch from 0x034ca1 to 0x03d503 (0x8863 bytes) 000167 1789052034.819560 read 0x01fbfc thru 0x025843 (0x5c48 bytes) 000168 1789052034.822726 mapwrite 0x01b45e thru 0x02815c (0xccff bytes) 000169 1789052034.927692 write 0x034bc5 thru 0x03c268 (0x76a4 bytes) 000170 1789052034.941681 write 0x01342c thru 0x018c2a (0x57ff bytes) 000171 1789052034.976441 trunc from 0x03d504 to 0x008ec8 (0x3463b bytes) 000172 1789052035.003341 write 0x00be03 thru 0x00c803 (0xa01 bytes) 000174 1789052035.004837 mapwrite 0x009b57 thru 0x0120b1 (0x855b bytes) 000178 1789052035.232990 read 0x00fd64 thru 0x0120b1 (0x234e bytes) 000179 1789052035.233850 punch from 0x00633e to 0x00db0b (0x77ce bytes) 000180 1789052035.240772 write 0x03d6a2 thru 0x03ffff (0x295e bytes) 000181 1789052035.249229 trunc from 0x040000 to 0x00d764 (0x3289b bytes) 000183 1789052035.282575 punch from 0x0036f5 to 0x00d764 (0xa070 bytes) 000185 1789052035.294236 write 0x00efcd thru 0x01055e (0x1592 bytes) 000186 1789052035.297324 read 0x002149 thru 0x00fcc7 (0xdb7f bytes) 000189 1789052035.323386 mapread 0x00257e thru 0x003b3b (0x15be bytes) 000190 1789052035.324249 write 0x036326 thru 0x03ffff (0x9cda bytes) 000191 1789052035.330468 trunc from 0x040000 to 0x03419b (0xbe64 bytes) 000193 1789052035.390442 write 0x03245d thru 0x033a55 (0x15f9 bytes) 000194 1789052035.408517 punch from 0x008ce5 to 0x015193 (0xc4af bytes) 000196 1789052035.427097 mapread 0x007aff thru 0x014bc0 (0xd0c2 bytes) 000197 1789052035.521510 trunc from 0x03419c to 0x02227e (0x11f1d bytes) truncating to largest ever: 0x3a64b 000199 1789052035.548899 trunc from 0x02227f to 0x03a64a (0x183cc bytes) 000201 1789052035.580584 write 0x008831 thru 0x00ebe8 (0x63b8 bytes) 000204 1789052035.588750 MIRROR_EXTEND to 2 mirrors 000205 1789052036.912915 write 0x020cf7 thru 0x030131 (0xf43b bytes) 000209 1789052036.998716 write 0x01fd0e thru 0x020d14 (0x1007 bytes) 000210 1789052037.007463 read 0x033fe3 thru 0x03a64a (0x6668 bytes) 000211 1789052037.021116 trunc from 0x03a64b to 0x01fde4 (0x1a866 bytes) 000213 1789052037.079789 mapread 0x01757e thru 0x01fde4 (0x8867 bytes) 000214 1789052037.100210 trunc from 0x01fde5 to 0x015925 (0xa4bf bytes) 000215 1789052037.124393 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000216 1789052037.261323 mapread 0x008e75 thru 0x00cd98 (0x3f24 bytes) 000220 1789052037.287587 trunc from 0x015926 to 0x00fd8b (0x5b9a bytes) 000223 1789052037.308188 punch from 0x000baf to 0x00c6dd (0xbb2f bytes) 000224 1789052037.325421 read 0x00bced thru 0x00cb66 (0xe7a bytes) 000225 1789052037.340315 mapwrite 0x0185c9 thru 0x01eb1c (0x6554 bytes) 000226 1789052037.474407 trunc from 0x01eb1d to 0x01de3e (0xcde bytes) 000227 1789052037.501325 write 0x00df4e thru 0x01483d (0x68f0 bytes) 000228 1789052037.502585 write 0x0246e5 thru 0x026ed6 (0x27f2 bytes) 000230 1789052037.513474 write 0x005acb thru 0x00cc3d (0x7173 bytes) truncating to largest ever: 0x3d4f5 000232 1789052037.527738 trunc from 0x026ed7 to 0x03d4f4 (0x1661e bytes) 000233 1789052037.554781 read 0x0043ae thru 0x007560 (0x31b3 bytes) 000234 1789052037.564822 mapwrite 0x01e415 thru 0x028a2b (0xa617 bytes) 000236 1789052037.670726 trunc from 0x03d4f5 to 0x03d33e (0x1b6 bytes) 000237 1789052037.703247 MIRROR_EXTEND to 2 mirrors 000239 1789052039.043019 mapread 0x01091c thru 0x0118df (0xfc4 bytes) 000240 1789052039.072878 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000241 1789052039.220106 mapwrite 0x018733 thru 0x01da3a (0x5308 bytes) 000242 1789052039.309551 write 0x002e39 thru 0x012b65 (0xfd2d bytes) 000243 1789052039.321620 mapwrite 0x010bb2 thru 0x01af73 (0xa3c2 bytes) 000244 1789052039.381185 MIRROR_EXTEND to 2 mirrors 000246 1789052040.227512 punch from 0x017ebf to 0x027917 (0xfa59 bytes) 000247 1789052040.288562 trunc from 0x03d33f to 0x01ee3b (0x1e503 bytes) 000248 1789052040.309092 trunc from 0x01ee3c to 0x037150 (0x18315 bytes) 000250 1789052040.329360 punch from 0x01574b to 0x02268e (0xcf44 bytes) 000251 1789052040.334180 punch from 0x0112c0 to 0x01b25f (0x9fa0 bytes) 000252 1789052040.338872 mapread 0x000006 thru 0x00d6c1 (0xd6bc bytes) 000253 1789052040.359633 trunc from 0x037151 to 0x035d09 (0x1447 bytes) 000254 1789052040.384074 MIRROR_RESYNC 2 mirrors 000255 1789052041.060816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000256 1789052041.186526 write 0x033e9e thru 0x03ffff (0xc162 bytes) 000257 1789052041.203724 mapread 0x03b249 thru 0x03ffff (0x4db7 bytes) 000258 1789052041.204488 mapread 0x02da89 thru 0x03b7db (0xdd53 bytes) 000259 1789052041.278883 MIRROR_EXTEND to 2 mirrors 000260 1789052042.155010 write 0x002f8c thru 0x004224 (0x1299 bytes) 000262 1789052042.214840 write 0x026c86 thru 0x02a55a (0x38d5 bytes) 000263 1789052042.223714 read 0x013428 thru 0x0142ce (0xea7 bytes) 000264 1789052042.229396 trunc from 0x040000 to 0x01afe9 (0x25016 bytes) 000265 1789052042.251392 MIRROR_RESYNC 2 mirrors 000266 1789052042.858312 mapwrite 0x02bb37 thru 0x0375cf (0xba99 bytes) 000267 1789052043.028244 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000269 1789052043.219732 mapwrite 0x02d983 thru 0x031bbe (0x423c bytes) 000270 1789052043.313694 punch from 0x01e68b to 0x027600 (0x8f76 bytes) 000271 1789052043.318850 write 0x0192d7 thru 0x02070e (0x7438 bytes) 000272 1789052043.328904 read 0x029349 thru 0x0360ce (0xcd86 bytes) 000274 1789052043.331401 read 0x0063f8 thru 0x00ed96 (0x899f bytes) 000275 1789052043.353461 read 0x00964b thru 0x014dba (0xb770 bytes) 000276 1789052043.354552 read 0x032da3 thru 0x0375cf (0x482d bytes) 000277 1789052043.355519 write 0x031c36 thru 0x039707 (0x7ad2 bytes) 000278 1789052043.358355 mapwrite 0x02a1d8 thru 0x033b09 (0x9932 bytes) 000282 1789052043.416460 trunc from 0x039708 to 0x03014f (0x95b8 bytes) 000283 1789052043.443854 mapread 0x0064c2 thru 0x0150e9 (0xec28 bytes) 000284 1789052043.446316 trunc from 0x030150 to 0x0090ef (0x27060 bytes) 000285 1789052043.462550 punch from 0x0017d1 to 0x0090ef (0x791f bytes) 000286 1789052043.467487 mapwrite 0x035946 thru 0x03ffff (0xa6ba bytes) 000287 1789052043.552830 mapwrite 0x020c7c thru 0x02512c (0x44b1 bytes) 000289 1789052043.594579 punch from 0x024b02 to 0x02c9e7 (0x7ee6 bytes) 000290 1789052043.599130 write 0x0377c0 thru 0x039101 (0x1942 bytes) 000291 1789052043.601042 mapwrite 0x0072fb thru 0x007337 (0x3d bytes) 000293 1789052043.642005 mapread 0x03f87a thru 0x03ffff (0x786 bytes) 000294 1789052043.642890 write 0x0287c7 thru 0x02b291 (0x2acb bytes) 000295 1789052043.654786 write 0x03f1f9 thru 0x03ffff (0xe07 bytes) 000297 1789052043.656486 write 0x0066f4 thru 0x00d35e (0x6c6b bytes) 000298 1789052043.662086 MIRROR_EXTEND to 2 mirrors 000300 1789052044.794265 trunc from 0x040000 to 0x021e44 (0x1e1bb bytes) 000301 1789052044.999783 mapread 0x010c10 thru 0x01a802 (0x9bf3 bytes) 000302 1789052045.047232 punch from 0x011c72 to 0x01f5e0 (0xd96f bytes) 000303 1789052045.055143 trunc from 0x021e45 to 0x00af4a (0x16efa bytes) 000304 1789052045.151333 mapwrite 0x03e173 thru 0x03ffff (0x1e8d bytes) 000305 1789052045.284767 write 0x004648 thru 0x0130b7 (0xea70 bytes) 000306 1789052045.304110 write 0x01c36d thru 0x01c9f6 (0x68a bytes) 000307 1789052045.316459 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7f7aa4e3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6652f59f Mirror 3: 0x6652f59f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1789052046.591679 write 0x035837 thru 0x03ffff (0xa7c9 bytes) 000310 1789052046.684733 mapread 0x02ac90 thru 0x031869 (0x6bda bytes) 000312 1789052046.702402 mapread 0x01abbf thru 0x023d7c (0x91be bytes) 000313 1789052046.716013 write 0x031384 thru 0x039c57 (0x88d4 bytes) 000315 1789052046.718604 write 0x01e8d7 thru 0x024f2d (0x6657 bytes) 000316 1789052046.719751 trunc from 0x040000 to 0x0385cd (0x7a32 bytes) 000317 1789052046.743052 write 0x039550 thru 0x03cd99 (0x384a bytes) 000319 1789052046.746011 punch from 0x0025c9 to 0x00ccf0 (0xa728 bytes) 000320 1789052046.766616 mapwrite 0x00ada1 thru 0x0143a8 (0x9608 bytes) 000321 1789052046.825815 read 0x00d814 thru 0x01bca0 (0xe48d bytes) 000322 1789052046.831565 MIRROR_RESYNC 3 mirrors 000324 1789052047.749520 read 0x039ec4 thru 0x03cd99 (0x2ed6 bytes) 000327 1789052047.755460 punch from 0x025f4e to 0x033b0c (0xdbbf bytes) 000329 1789052047.847445 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 2: 0xf3ea440b (stale) Mirror 3: 0xf3ea440b (stale) CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 1: 0x672b2860 Mirror 4: 0x672b2860 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd9a: skipped 000330 1789052048.988756 read 0x00014a thru 0x00cd92 (0xcc49 bytes) 000331 1789052049.025609 mapread 0x03603b thru 0x038a07 (0x29cd bytes) 000332 1789052049.092743 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 2: 0xf3ea440b (stale) Mirror 3: 0xf3ea440b (stale) CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 1: 0x672b2860 Mirror 4: 0x672b2860 Mirror 5: 0x672b2860 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd9a: skipped 000334 1789052050.469621 mapwrite 0x00855f thru 0x009674 (0x1116 bytes) 000337 1789052050.644127 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 3: 0xf3ea440b (stale) Mirror 4: 0x672b2860 (stale) Mirror 5: 0x672b2860 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd9a): Mirror 3: 0xf3ea440b (stale) Mirror 4: 0x672b2860 (stale) Mirror 5: 0x672b2860 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd9a: skipped 000340 1789052051.168649 read 0x02b07e thru 0x0308f3 (0x5876 bytes) truncating to largest ever: 0x3f4af 000342 1789052051.178482 trunc from 0x03cd9a to 0x03f4ae (0x2715 bytes) 000343 1789052051.206374 read 0x025554 thru 0x029198 (0x3c45 bytes) 000344 1789052051.231451 mapread 0x032697 thru 0x038c3c (0x65a6 bytes) 000345 1789052051.233260 read 0x00f5d8 thru 0x01847f (0x8ea8 bytes) 000346 1789052051.234061 mapread 0x00fcea thru 0x0140c1 (0x43d8 bytes) 000349 1789052051.238355 trunc from 0x03f4af to 0x008213 (0x3729b bytes) 000351 1789052051.265831 write 0x0056a4 thru 0x006615 (0xf72 bytes) 000353 1789052051.273105 mapread 0x0065cb thru 0x008183 (0x1bb9 bytes) 000356 1789052051.279418 trunc from 0x008214 to 0x023f0b (0x1bcf8 bytes) 000359 1789052051.326651 trunc from 0x023f0c to 0x03b8b6 (0x179ab bytes) 000360 1789052051.361707 mapread 0x01fe0f thru 0x02b3fd (0xb5ef bytes) 000361 1789052051.378678 write 0x005737 thru 0x008c0f (0x34d9 bytes) 000363 1789052051.380751 read 0x016640 thru 0x01e8e5 (0x82a6 bytes) 000364 1789052051.388679 mapwrite 0x02757d thru 0x032e4a (0xb8ce bytes) 000365 1789052051.440513 write 0x01f34b thru 0x028d0b (0x99c1 bytes) 000366 1789052051.443380 read 0x00daf8 thru 0x01d703 (0xfc0c bytes) 000368 1789052051.445028 mapread 0x01b49f thru 0x01fb0f (0x4671 bytes) 000369 1789052051.451724 punch from 0x02d030 to 0x03ac06 (0xdbd7 bytes) 000370 1789052051.464227 trunc from 0x03b8b7 to 0x032e15 (0x8aa1 bytes) 000371 1789052051.487887 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32e16): Mirror 3: 0x50cdf711 (stale) Mirror 5: 0x73683f8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e16): Mirror 3: 0x50cdf711 (stale) Mirror 5: 0x73683f8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e16: skipped 000372 1789052051.919078 trunc from 0x032e16 to 0x031396 (0x1a7f bytes) 000373 1789052051.934778 mapread 0x00b5b6 thru 0x015b63 (0xa5ae bytes) 000375 1789052051.963568 read 0x01ab2d thru 0x02629b (0xb76f bytes) 000376 1789052051.978588 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31397): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31397): Mirror 3: 0x3efe63a5 (stale) Mirror 5: 0x34e7bd75 (stale) CRC-32 checksum value for chunk [0, 0x31397): Mirror 1: 0xc2ac9c8a Mirror 6: 0xc2ac9c8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31397: skipped 000378 1789052052.889731 mapwrite 0x02ad31 thru 0x03813a (0xd40a bytes) 000379 1789052053.285668 read 0x015b98 thru 0x0259a1 (0xfe0a bytes) 000381 1789052053.296981 trunc from 0x03813b to 0x01c6c8 (0x1ba72 bytes) 000382 1789052053.344167 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c6c9): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c6c9): Mirror 3: 0xabb12c65 (stale) Mirror 5: 0xabb12c65 (stale) Mirror 6: 0x54bccaed (stale) CRC-32 checksum value for chunk [0, 0x1c6c9): Mirror 1: 0x54bccaed Mirror 7: 0x54bccaed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c6c9: skipped 000384 1789052054.437537 mapread 0x003d97 thru 0x00f53f (0xb7a9 bytes) 000385 1789052054.481237 punch from 0x0163ca to 0x01c6c8 (0x62ff bytes) 000386 1789052054.583418 write 0x037aba thru 0x03ffff (0x8546 bytes) 000388 1789052054.588391 punch from 0x011d6c to 0x01320d (0x14a2 bytes) 000389 1789052054.603853 punch from 0x03901b to 0x03f125 (0x610b bytes) 000391 1789052054.608920 trunc from 0x040000 to 0x0025c1 (0x3da3e bytes) 000393 1789052054.633223 write 0x02696b thru 0x030e83 (0xa519 bytes) 000395 1789052054.639784 mapwrite 0x0103a1 thru 0x01b6e1 (0xb341 bytes) 000396 1789052054.713752 write 0x0110f2 thru 0x0200aa (0xefb9 bytes) 000398 1789052054.725859 punch from 0x025c47 to 0x02fcf8 (0xa0b2 bytes) 000399 1789052054.767094 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e84): Mirror 5: 0xb0950c0a (stale) Mirror 6: 0x96651b2b (stale) Mirror 7: 0x9f846063 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e84): Mirror 5: 0xb0950c0a (stale) Mirror 6: 0x96651b2b (stale) Mirror 7: 0x9f846063 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e84: skipped 000400 1789052055.122549 trunc from 0x030e84 to 0x004de8 (0x2c09b bytes) 000401 1789052055.144923 mapread 0x003c17 thru 0x004de8 (0x11d2 bytes) 000402 1789052055.158949 write 0x0331e7 thru 0x03ffff (0xce19 bytes) 000404 1789052055.162442 mapread 0x00fbe3 thru 0x017bcc (0x7fea bytes) 000405 1789052055.170943 trunc from 0x040000 to 0x00e04a (0x31fb5 bytes) 000407 1789052055.187852 punch from 0x007f74 to 0x00e04a (0x60d7 bytes) 000409 1789052055.194577 write 0x026da6 thru 0x02eaeb (0x7d46 bytes) 000410 1789052055.196028 mapwrite 0x03c689 thru 0x03ffff (0x3977 bytes) 000411 1789052055.285554 mapread 0x020e24 thru 0x029a50 (0x8c2d bytes) 000412 1789052055.302248 mapwrite 0x00a519 thru 0x014a14 (0xa4fc bytes) 000413 1789052055.430066 mapwrite 0x0272cb thru 0x035de1 (0xeb17 bytes) 000414 1789052055.503976 mapwrite 0x01006c thru 0x01d795 (0xd72a bytes) 000415 1789052055.548806 mapwrite 0x024a5b thru 0x02aa51 (0x5ff7 bytes) 000416 1789052055.685843 write 0x03b9d0 thru 0x03ffff (0x4630 bytes) 000417 1789052055.691120 mapread 0x01a1f9 thru 0x0227ba (0x85c2 bytes) 000418 1789052055.692312 read 0x0213e2 thru 0x028c54 (0x7873 bytes) 000421 1789052055.711119 MIRROR_RESYNC 4 mirrors 000422 1789052056.727345 write 0x01ce4b thru 0x026e18 (0x9fce bytes) 000423 1789052056.815392 MIRROR_RESYNC 4 mirrors 000425 1789052057.799497 punch from 0x001526 to 0x006a2f (0x550a bytes) 000426 1789052057.913843 punch from 0x006d16 to 0x00efe4 (0x82cf bytes) 000427 1789052057.921781 mapwrite 0x023f63 thru 0x031bce (0xdc6c bytes) 000430 1789052058.056151 read 0x02ab33 thru 0x02e854 (0x3d22 bytes) 000431 1789052058.058294 punch from 0x01e6f0 to 0x02649d (0x7dae bytes) 000432 1789052058.064112 punch from 0x019125 to 0x027a45 (0xe921 bytes) 000434 1789052058.070083 mapread 0x029581 thru 0x02e773 (0x51f3 bytes) 000436 1789052058.072164 mapwrite 0x01b6fa thru 0x01d92e (0x2235 bytes) 000439 1789052058.116323 read 0x0135cb thru 0x01698f (0x33c5 bytes) 000440 1789052058.130859 mapread 0x0304c6 thru 0x030fca (0xb05 bytes) 000442 1789052058.142151 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8492fe6f (stale) Mirror 7: 0x8492fe6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8492fe6f (stale) Mirror 7: 0x8492fe6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1789052058.538188 trunc from 0x040000 to 0x0069cc (0x39633 bytes) 000446 1789052058.571232 write 0x015fa5 thru 0x01e9cc (0x8a28 bytes) 000447 1789052058.575155 trunc from 0x01e9cd to 0x01684c (0x8180 bytes) 000449 1789052058.616782 trunc from 0x01684d to 0x0078ad (0xef9f bytes) 000450 1789052058.644863 punch from 0x003b02 to 0x0078ad (0x3dac bytes) 000451 1789052058.654637 read 0x000b7c thru 0x0078ad (0x6d32 bytes) 000453 1789052058.670184 mapread 0x003d5c thru 0x0078ad (0x3b52 bytes) 000454 1789052058.673750 trunc from 0x0078ae to 0x0151e0 (0xd933 bytes) 000455 1789052058.703694 mapread 0x001ddf thru 0x003c2c (0x1e4e bytes) 000456 1789052058.705095 trunc from 0x0151e1 to 0x007416 (0xddca bytes) 000458 1789052058.727841 MIRROR_RESYNC 3 mirrors 000459 1789052059.294800 write 0x0028d2 thru 0x00dc96 (0xb3c5 bytes) 000463 1789052059.379701 trunc from 0x00dc97 to 0x01a027 (0xc391 bytes) 000465 1789052059.396562 write 0x00ae6e thru 0x01191a (0x6aad bytes) 000466 1789052059.401764 mapwrite 0x01cc3c thru 0x028be5 (0xbfaa bytes) 000468 1789052059.515836 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28be6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28be6): Mirror 6: 0xb49a372d (stale) Mirror 7: 0xb49a372d (stale) CRC-32 checksum value for chunk [0, 0x28be6): Mirror 1: 0xe3e4dc4f Mirror 8: 0xe3e4dc4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28be6: skipped 000469 1789052060.266897 trunc from 0x028be6 to 0x0117bf (0x17426 bytes) 000470 1789052060.338018 read 0x003370 thru 0x00ae54 (0x7ae5 bytes) 000471 1789052060.346183 trunc from 0x0117c0 to 0x0174c4 (0x5d05 bytes) 000472 1789052060.363092 read 0x00026b thru 0x0077ff (0x7595 bytes) 000473 1789052060.371642 trunc from 0x0174c5 to 0x0263d8 (0xef14 bytes) 000474 1789052060.392797 write 0x03e8e8 thru 0x03ffff (0x1718 bytes) 000476 1789052060.394447 write 0x020dac thru 0x025d4b (0x4fa0 bytes) 000477 1789052060.416893 mapwrite 0x021c41 thru 0x02899e (0x6d5e bytes) 000478 1789052060.498332 mapread 0x00c0f4 thru 0x0197dd (0xd6ea bytes) 000479 1789052060.504191 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1789052060.822247 mapwrite 0x01f39d thru 0x0297fd (0xa461 bytes) 000481 1789052060.895518 punch from 0x00cee9 to 0x010aa3 (0x3bbb bytes) 000482 1789052060.907064 mapread 0x027f2d thru 0x033fbf (0xc093 bytes) 000483 1789052060.935013 read 0x028bdb thru 0x031ecb (0x92f1 bytes) 000484 1789052060.936787 mapread 0x00e54d thru 0x011c9d (0x3751 bytes) 000486 1789052060.948715 mapwrite 0x0124ef thru 0x01804d (0x5b5f bytes) 000488 1789052061.019810 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86dbe409 Mirror 8: 0x86dbe409 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1789052062.145825 trunc from 0x040000 to 0x039734 (0x68cb bytes) 000494 1789052062.288512 trunc from 0x039735 to 0x012be8 (0x26b4c bytes) 000495 1789052062.340519 mapread 0x006c92 thru 0x007a12 (0xd81 bytes) 000496 1789052062.352485 read 0x0010b1 thru 0x007b6f (0x6abf bytes) 000497 1789052062.363041 mapread 0x005153 thru 0x0068b1 (0x175f bytes) 000502 1789052062.366121 write 0x039a7b thru 0x03ffff (0x6585 bytes) 000503 1789052062.368203 write 0x010cda thru 0x01b718 (0xaa3f bytes) 000504 1789052062.381062 trunc from 0x040000 to 0x005df9 (0x3a206 bytes) 000506 1789052062.396817 read 0x003177 thru 0x005df9 (0x2c83 bytes) 000507 1789052062.397622 mapwrite 0x004408 thru 0x00dcca (0x98c3 bytes) 000508 1789052062.545112 punch from 0x003b00 to 0x00685b (0x2d5c bytes) 000509 1789052062.566087 mapread 0x006fa9 thru 0x00dcca (0x6d22 bytes) 000510 1789052062.574639 write 0x03de08 thru 0x03ffff (0x21f8 bytes) 000511 1789052062.575949 mapwrite 0x0082bd thru 0x00fa92 (0x77d6 bytes) 000512 1789052062.719223 punch from 0x033183 to 0x03f740 (0xc5be bytes) 000513 1789052062.731239 mapwrite 0x036ccc thru 0x0376dc (0xa11 bytes) 000514 1789052062.847482 read 0x0078b0 thru 0x013a04 (0xc155 bytes) 000515 1789052062.848093 mapwrite 0x02ee88 thru 0x03c55e (0xd6d7 bytes) 000516 1789052062.937129 punch from 0x03ffb7 to 0x03ffff (0x49 bytes) 000517 1789052062.951883 write 0x0122bf thru 0x01e5b9 (0xc2fb bytes) 000518 1789052062.953368 punch from 0x02b6f0 to 0x0362e2 (0xabf3 bytes) 000522 1789052062.976622 write 0x034cb1 thru 0x038702 (0x3a52 bytes) 000523 1789052062.999101 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0x86dbe409 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6e7fda4 Mirror 9: 0xe6e7fda4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000524 1789052064.218017 mapread 0x036ae2 thru 0x03c1dd (0x56fc bytes) 000526 1789052064.300523 mapread 0x021dd4 thru 0x02d071 (0xb29e bytes) 000527 1789052064.352159 punch from 0x03e5fd to 0x03ffff (0x1a03 bytes) 000528 1789052064.508224 trunc from 0x040000 to 0x024a98 (0x1b567 bytes) 000529 1789052064.543763 mapwrite 0x010441 thru 0x01560e (0x51ce bytes) 000530 1789052064.630200 trunc from 0x024a99 to 0x038c14 (0x1417c bytes) 000531 1789052064.660883 punch from 0x01902b to 0x01f360 (0x6336 bytes) 000533 1789052064.668819 punch from 0x016cc6 to 0x01af7d (0x42b8 bytes) 000534 1789052064.672038 mapwrite 0x03f9a3 thru 0x03ffff (0x65d bytes) 000536 1789052064.765434 mapwrite 0x017992 thru 0x017a0a (0x79 bytes) 000537 1789052064.841107 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0x86dbe409 (stale) Mirror 9: 0xe6e7fda4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfaa73a8c Mirror 10: 0xfaa73a8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000538 1789052065.800708 mapread 0x036ecc thru 0x03ffff (0x9134 bytes) 000539 1789052065.840950 write 0x027797 thru 0x037556 (0xfdc0 bytes) 000540 1789052065.960633 mapread 0x0037e6 thru 0x004f91 (0x17ac bytes) 000542 1789052065.977027 mapread 0x0287bf thru 0x02c558 (0x3d9a bytes) 000543 1789052065.982316 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0x86dbe409 (stale) Mirror 9: 0xe6e7fda4 (stale) Mirror 10: 0xfaa73a8c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfe168069 Mirror 11: 0xfe168069 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1789052066.910240 write 0x02fbe7 thru 0x0368b7 (0x6cd1 bytes) 000547 1789052067.025138 mapread 0x007a50 thru 0x00d202 (0x57b3 bytes) 000548 1789052067.047391 punch from 0x03706e to 0x03d99b (0x692e bytes) 000549 1789052067.069838 mapread 0x019135 thru 0x02663b (0xd507 bytes) 000550 1789052067.103164 mapread 0x0183ed thru 0x026b22 (0xe736 bytes) 000551 1789052067.105728 mapwrite 0x01b868 thru 0x02ac3a (0xf3d3 bytes) 000552 1789052067.182052 write 0x0138c5 thru 0x01fee2 (0xc61e bytes) 000553 1789052067.189004 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0x86dbe409 (stale) Mirror 9: 0xe6e7fda4 (stale) Mirror 10: 0xfaa73a8c (stale) Mirror 11: 0xfe168069 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x12bb5773 Mirror 12: 0x12bb5773 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000554 1789052068.348971 trunc from 0x040000 to 0x01c38e (0x23c71 bytes) 000555 1789052068.417875 mapwrite 0x02ed56 thru 0x0315c3 (0x286e bytes) 000556 1789052068.495297 punch from 0x02c91a to 0x02dde5 (0x14cc bytes) 000557 1789052068.500178 mapread 0x01e88c thru 0x026bfd (0x8372 bytes) 000558 1789052068.509862 punch from 0x0031ec to 0x00e95e (0xb773 bytes) 000559 1789052068.513979 write 0x0140cc thru 0x0196d9 (0x560e bytes) 000562 1789052068.521857 write 0x037626 thru 0x03ffff (0x89da bytes) 000563 1789052068.523910 read 0x00caae thru 0x013a4c (0x6f9f bytes) 000565 1789052068.531991 punch from 0x00c303 to 0x014d0a (0x8a08 bytes) 000566 1789052068.543335 mapwrite 0x00e3ef thru 0x010f58 (0x2b6a bytes) truncating to largest ever: 0x3f7d9 000567 1789052068.595669 trunc from 0x040000 to 0x03f7d8 (0x827 bytes) 000568 1789052068.613956 mapread 0x020cd1 thru 0x0248b1 (0x3be1 bytes) 000570 1789052068.616359 punch from 0x027ef9 to 0x028751 (0x859 bytes) 000571 1789052068.621002 mapwrite 0x037bb9 thru 0x03ffff (0x8447 bytes) 000572 1789052068.682886 punch from 0x03ad00 to 0x03dff1 (0x32f2 bytes) 000575 1789052068.689894 trunc from 0x040000 to 0x00f0cc (0x30f33 bytes) 000576 1789052068.722558 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xf0cd): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0cd): Mirror 6: 0x21b30a86 (stale) Mirror 7: 0x21b30a86 (stale) Mirror 8: 0x4aaf90d1 (stale) Mirror 9: 0xcade6bdb (stale) Mirror 10: 0xcade6bdb (stale) Mirror 11: 0xcade6bdb (stale) Mirror 12: 0xcade6bdb (stale) CRC-32 checksum value for chunk [0, 0xf0cd): Mirror 1: 0xa9e2ab Mirror 13: 0xa9e2ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0cd: skipped 000577 1789052070.358116 mapread 0x00a3c0 thru 0x00f0cc (0x4d0d bytes) 000578 1789052070.410543 mapread 0x006ef3 thru 0x00d5d9 (0x66e7 bytes) 000579 1789052070.458538 trunc from 0x00f0cd to 0x039e3a (0x2ad6e bytes) 000580 1789052070.586519 write 0x00962a thru 0x01496d (0xb344 bytes) 000583 1789052070.600596 write 0x02cf76 thru 0x02d963 (0x9ee bytes) 000584 1789052070.642181 punch from 0x015f4e to 0x01c14b (0x61fe bytes) 000585 1789052070.710307 read 0x032d58 thru 0x034887 (0x1b30 bytes) 000586 1789052070.739038 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e3b): Mirror 6: 0xc144d7c6 (stale) Mirror 7: 0xc144d7c6 (stale) Mirror 8: 0xca549dad (stale) Mirror 10: 0x52fa613f (stale) Mirror 11: 0x6ff50ef0 (stale) Mirror 12: 0x353ae936 (stale) Mirror 13: 0x3fe8cf96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e3b): Mirror 6: 0xc144d7c6 (stale) Mirror 7: 0xc144d7c6 (stale) Mirror 8: 0xca549dad (stale) Mirror 10: 0x52fa613f (stale) Mirror 11: 0x6ff50ef0 (stale) Mirror 12: 0x353ae936 (stale) Mirror 13: 0x3fe8cf96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e3b: skipped 000587 1789052071.313770 read 0x007c6c thru 0x00adc4 (0x3159 bytes) 000588 1789052071.321581 write 0x024950 thru 0x029702 (0x4db3 bytes) 000589 1789052071.346632 mapwrite 0x01bde6 thru 0x0207f1 (0x4a0c bytes) 000590 1789052071.440017 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39e3b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e3b): Mirror 6: 0xc144d7c6 (stale) Mirror 7: 0xc144d7c6 (stale) Mirror 8: 0xca549dad (stale) Mirror 10: 0x52fa613f (stale) Mirror 11: 0x6ff50ef0 (stale) Mirror 12: 0x353ae936 (stale) Mirror 13: 0x3fe8cf96 (stale) CRC-32 checksum value for chunk [0, 0x39e3b): Mirror 1: 0x31f96ef7 Mirror 14: 0x31f96ef7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e3b: skipped 000591 1789052072.476595 mapread 0x003298 thru 0x004f04 (0x1c6d bytes) 000592 1789052072.500108 write 0x007529 thru 0x007ca4 (0x77c bytes) 000593 1789052072.594674 read 0x005e7f thru 0x014cb0 (0xee32 bytes) 000594 1789052072.615573 trunc from 0x039e3b to 0x029e47 (0xfff3 bytes) 000595 1789052072.640945 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e48): Mirror 6: 0x6307cf9c (stale) Mirror 7: 0x6307cf9c (stale) Mirror 8: 0xf464fc27 (stale) Mirror 10: 0x99d6ef80 (stale) Mirror 12: 0x630a4e81 (stale) Mirror 13: 0x29b8d31b (stale) Mirror 14: 0x43a169d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e48): Mirror 6: 0x6307cf9c (stale) Mirror 7: 0x6307cf9c (stale) Mirror 8: 0xf464fc27 (stale) Mirror 10: 0x99d6ef80 (stale) Mirror 12: 0x630a4e81 (stale) Mirror 13: 0x29b8d31b (stale) Mirror 14: 0x43a169d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e48: skipped 000596 1789052073.134141 trunc from 0x029e48 to 0x0245b6 (0x5891 bytes) 000597 1789052073.159836 read 0x01516d thru 0x02168c (0xc520 bytes) 000598 1789052073.188197 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x245b7): Mirror 7: 0xc64b9a79 (stale) Mirror 8: 0xc6425823 (stale) Mirror 10: 0xe6e20f19 (stale) Mirror 12: 0xb41d0a88 (stale) Mirror 13: 0x8a275ada (stale) Mirror 14: 0x808e9f5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x245b7): Mirror 7: 0xc64b9a79 (stale) Mirror 8: 0xc6425823 (stale) Mirror 10: 0xe6e20f19 (stale) Mirror 12: 0xb41d0a88 (stale) Mirror 13: 0x8a275ada (stale) Mirror 14: 0x808e9f5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x245b7: skipped 000599 1789052073.764148 mapread 0x018d5a thru 0x0245b6 (0xb85d bytes) 000602 1789052073.801368 read 0x012561 thru 0x013d87 (0x1827 bytes) 000604 1789052073.808173 trunc from 0x0245b7 to 0x004dd1 (0x1f7e5 bytes) 000605 1789052073.837537 MIRROR_RESYNC 7 mirrors 000606 1789052075.290358 trunc from 0x004dd2 to 0x017060 (0x1228f bytes) 000607 1789052075.405733 write 0x00de64 thru 0x016d80 (0x8f1d bytes) 000608 1789052075.422433 MIRROR_RESYNC 7 mirrors 000609 1789052076.915972 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000610 1789052077.498503 MIRROR_EXTEND to 7 mirrors 000611 1789052078.597153 write 0x00d22a thru 0x0130b4 (0x5e8b bytes) 000612 1789052078.724774 mapwrite 0x0295c5 thru 0x03616c (0xcba8 bytes) 000614 1789052078.838070 mapwrite 0x03a09f thru 0x03ffff (0x5f61 bytes) 000616 1789052078.924679 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3e201129 (stale) Mirror 8: 0x3e201129 (stale) Mirror 12: 0x3e201129 (stale) Mirror 13: 0x3e201129 (stale) Mirror 14: 0x3e201129 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3e201129 (stale) Mirror 8: 0x3e201129 (stale) Mirror 12: 0x3e201129 (stale) Mirror 13: 0x3e201129 (stale) Mirror 14: 0x3e201129 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000617 1789052079.469691 trunc from 0x040000 to 0x037e5a (0x81a5 bytes) 000620 1789052079.491186 mapwrite 0x038aa1 thru 0x039a3a (0xf9a bytes) 000621 1789052079.575071 write 0x035a8f thru 0x0377f6 (0x1d68 bytes) 000622 1789052079.584709 read 0x0289b0 thru 0x0334d7 (0xab28 bytes) 000623 1789052079.607794 mapwrite 0x00c30a thru 0x00df36 (0x1c2d bytes) 000624 1789052079.667480 write 0x001b1d thru 0x0088de (0x6dc2 bytes) 000625 1789052079.679184 write 0x032858 thru 0x0347bc (0x1f65 bytes) 000628 1789052079.699465 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39a3b): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a3b): Mirror 7: 0xe303bc3a (stale) Mirror 8: 0xe303bc3a (stale) Mirror 12: 0xe303bc3a (stale) Mirror 13: 0xe303bc3a (stale) Mirror 14: 0xe303bc3a (stale) CRC-32 checksum value for chunk [0, 0x39a3b): Mirror 1: 0x53e5a9f1 Mirror 15: 0x53e5a9f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a3b: skipped 000629 1789052081.048974 read 0x028bff thru 0x02d04d (0x444f bytes) 000630 1789052081.068570 write 0x038e6d thru 0x03ffff (0x7193 bytes) 000631 1789052081.146972 mapwrite 0x02f5b9 thru 0x030918 (0x1360 bytes) 000634 1789052081.269697 punch from 0x01e1fd to 0x0264c0 (0x82c4 bytes) 000635 1789052081.278449 punch from 0x004b35 to 0x00f20e (0xa6da bytes) 000636 1789052081.283823 read 0x01d884 thru 0x0291e0 (0xb95d bytes) 000640 1789052081.297426 mapread 0x03510d thru 0x039f05 (0x4df9 bytes) 000642 1789052081.299568 read 0x0306b4 thru 0x037e74 (0x77c1 bytes) 000643 1789052081.300812 read 0x036f00 thru 0x03e0b0 (0x71b1 bytes) 000644 1789052081.302022 punch from 0x0277de to 0x036f70 (0xf793 bytes) 000645 1789052081.312384 mapread 0x011286 thru 0x01a110 (0x8e8b bytes) 000648 1789052081.316638 read 0x010776 thru 0x01505a (0x48e5 bytes) 000649 1789052081.317692 mapread 0x02d231 thru 0x03aff2 (0xddc2 bytes) 000650 1789052081.334036 trunc from 0x040000 to 0x01aa85 (0x2557a bytes) 000651 1789052081.363697 read 0x013f2f thru 0x015d8b (0x1e5d bytes) 000652 1789052081.364437 mapwrite 0x00c383 thru 0x010ef8 (0x4b76 bytes) 000655 1789052081.425646 trunc from 0x01aa86 to 0x00c63f (0xe446 bytes) 000656 1789052081.449836 write 0x00e521 thru 0x01426b (0x5d4b bytes) 000658 1789052081.453958 write 0x02ae54 thru 0x0326c7 (0x7874 bytes) 000659 1789052081.456537 read 0x0287c6 thru 0x0326c7 (0x9f02 bytes) 000660 1789052081.464902 mapread 0x0164e4 thru 0x01e487 (0x7fa4 bytes) 000661 1789052081.467000 trunc from 0x0326c8 to 0x00d8ad (0x24e1a bytes) 000662 1789052081.490085 mapread 0x007236 thru 0x00d8ad (0x6678 bytes) 000663 1789052081.496427 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd8ae): Mirror 7: 0x75e84cbd (stale) Mirror 8: 0x75e84cbd (stale) Mirror 12: 0x75e84cbd (stale) Mirror 13: 0x75e84cbd (stale) Mirror 14: 0x75e84cbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd8ae): Mirror 7: 0x75e84cbd (stale) Mirror 8: 0x75e84cbd (stale) Mirror 12: 0x75e84cbd (stale) Mirror 13: 0x75e84cbd (stale) Mirror 14: 0x75e84cbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd8ae: skipped 000666 1789052081.832585 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd8ae): Mirror 7: 0x75e84cbd (stale) Mirror 12: 0x75e84cbd (stale) Mirror 13: 0x75e84cbd (stale) Mirror 14: 0x75e84cbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd8ae): Mirror 7: 0x75e84cbd (stale) Mirror 12: 0x75e84cbd (stale) Mirror 13: 0x75e84cbd (stale) Mirror 14: 0x75e84cbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd8ae: skipped 000667 1789052082.161227 trunc from 0x00d8ae to 0x0017fe (0xc0af bytes) 000668 1789052082.177422 mapread 0x001103 thru 0x0017fe (0x6fc bytes) 000669 1789052082.183039 write 0x0107d9 thru 0x01f208 (0xea30 bytes) 000670 1789052082.187866 trunc from 0x01f209 to 0x00c8ca (0x1293e bytes) 000671 1789052082.207591 punch from 0x009da4 to 0x00c8ca (0x2b27 bytes) 000672 1789052082.212787 punch from 0x008df1 to 0x00c8ca (0x3ada bytes) 000673 1789052082.217737 trunc from 0x00c8cb to 0x0061b8 (0x6712 bytes) 000674 1789052082.235627 write 0x00c982 thru 0x011654 (0x4cd3 bytes) 000675 1789052082.237027 write 0x005a24 thru 0x00ccf5 (0x72d2 bytes) 000676 1789052082.247410 trunc from 0x011655 to 0x023caa (0x12656 bytes) 000678 1789052082.266881 punch from 0x01e560 to 0x023caa (0x574b bytes) 000681 1789052082.280851 read 0x015cd2 thru 0x01c211 (0x6540 bytes) 000682 1789052082.295752 MIRROR_RESYNC 5 mirrors 000683 1789052083.614175 punch from 0x0107b6 to 0x014534 (0x3d7f bytes) 000684 1789052083.746434 write 0x01faed thru 0x020e0e (0x1322 bytes) 000685 1789052083.768754 mapwrite 0x0066d1 thru 0x0142de (0xdc0e bytes) 000687 1789052083.937309 trunc from 0x023cab to 0x033c65 (0xffbb bytes) 000688 1789052084.016484 punch from 0x028ee1 to 0x033c65 (0xad85 bytes) 000690 1789052084.028795 read 0x02c2a7 thru 0x032967 (0x66c1 bytes) 000693 1789052084.049247 mapwrite 0x0351d7 thru 0x03a82b (0x5655 bytes) 000694 1789052084.243784 write 0x01a50c thru 0x01ebac (0x46a1 bytes) 000695 1789052084.261461 mapwrite 0x001a1d thru 0x003e70 (0x2454 bytes) 000696 1789052084.393112 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a82c): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a82c): Mirror 7: 0x2ef000bd (stale) Mirror 12: 0x2ef000bd (stale) Mirror 13: 0x2ef000bd (stale) Mirror 14: 0x2ef000bd (stale) CRC-32 checksum value for chunk [0, 0x3a82c): Mirror 1: 0xc9659d65 Mirror 15: 0xc9659d65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a82c: skipped 000697 1789052085.475856 punch from 0x01886e to 0x027587 (0xed1a bytes) 000700 1789052085.566600 read 0x036bc1 thru 0x03a82b (0x3c6b bytes) 000702 1789052085.577171 read 0x00a11f thru 0x00bf82 (0x1e64 bytes) 000703 1789052085.589515 write 0x03cb51 thru 0x03ffff (0x34af bytes) 000704 1789052085.590733 write 0x00b9e3 thru 0x00f520 (0x3b3e bytes) 000708 1789052085.594657 mapread 0x002776 thru 0x010213 (0xda9e bytes) 000709 1789052085.595980 write 0x02f2d2 thru 0x035dff (0x6b2e bytes) 000710 1789052085.597069 read 0x03e58f thru 0x03f8d9 (0x134b bytes) 000711 1789052085.597688 mapwrite 0x02618b thru 0x032e0b (0xcc81 bytes) 000713 1789052085.652585 mapread 0x038127 thru 0x03ffff (0x7ed9 bytes) 000714 1789052085.658253 punch from 0x01f1b8 to 0x025c7a (0x6ac3 bytes) 000716 1789052085.667040 read 0x0342f7 thru 0x036d43 (0x2a4d bytes) 000717 1789052085.668551 mapwrite 0x00f39b thru 0x01c3b3 (0xd019 bytes) 000718 1789052085.731869 read 0x00245f thru 0x0071da (0x4d7c bytes) 000719 1789052085.738528 MIRROR_RESYNC 6 mirrors 000720 1789052087.446608 write 0x03bd31 thru 0x03ffff (0x42cf bytes) 000722 1789052087.635449 mapread 0x003af3 thru 0x009d86 (0x6294 bytes) 000723 1789052087.677579 write 0x002a3a thru 0x01029f (0xd866 bytes) 000724 1789052087.679083 read 0x007da4 thru 0x00a43c (0x2699 bytes) 000725 1789052087.679642 write 0x00a389 thru 0x015b21 (0xb799 bytes) 000726 1789052087.691775 punch from 0x01fbc6 to 0x02a990 (0xadcb bytes) 000727 1789052087.710122 mapread 0x02b3ca thru 0x0329d5 (0x760c bytes) 000728 1789052087.733751 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2d8537cd (stale) Mirror 13: 0x2d8537cd (stale) Mirror 14: 0x2d8537cd (stale) Mirror 15: 0x2d8537cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2d8537cd (stale) Mirror 13: 0x2d8537cd (stale) Mirror 14: 0x2d8537cd (stale) Mirror 15: 0x2d8537cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000729 1789052088.219602 write 0x03743f thru 0x03ffff (0x8bc1 bytes) 000731 1789052088.226857 mapwrite 0x03c870 thru 0x03ffff (0x3790 bytes) 000732 1789052088.299879 MIRROR_RESYNC 5 mirrors 000733 1789052089.532868 punch from 0x0030e1 to 0x004fbf (0x1edf bytes) 000735 1789052089.616121 mapwrite 0x02ec44 thru 0x0375ca (0x8987 bytes) 000736 1789052089.703809 punch from 0x03687f to 0x0369cb (0x14d bytes) 000739 1789052089.711126 punch from 0x019c21 to 0x0234f0 (0x98d0 bytes) 000740 1789052089.716502 write 0x01de72 thru 0x0299c5 (0xbb54 bytes) 000743 1789052089.725424 write 0x0361b7 thru 0x03ffff (0x9e49 bytes) 000744 1789052089.735402 mapread 0x0048c3 thru 0x00a932 (0x6070 bytes) 000745 1789052089.745848 trunc from 0x040000 to 0x03094d (0xf6b2 bytes) 000746 1789052089.770257 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3094e): Mirror 13: 0x1c7ff082 (stale) Mirror 14: 0x1c7ff082 (stale) Mirror 15: 0x1c7ff082 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3094e): Mirror 13: 0x1c7ff082 (stale) Mirror 14: 0x1c7ff082 (stale) Mirror 15: 0x1c7ff082 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3094e: skipped 000748 1789052090.223001 punch from 0x00889e to 0x00c06f (0x37d2 bytes) 000750 1789052090.230868 read 0x0280f0 thru 0x03094d (0x885e bytes) 000752 1789052090.246968 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3094e): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3094e): Mirror 13: 0x1c7ff082 (stale) Mirror 14: 0x1c7ff082 (stale) Mirror 15: 0x1c7ff082 (stale) CRC-32 checksum value for chunk [0, 0x3094e): Mirror 1: 0x55d6274a Mirror 16: 0x55d6274a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3094e: skipped 000753 1789052091.238704 mapread 0x02ea2d thru 0x03094d (0x1f21 bytes) 000754 1789052091.323081 mapwrite 0x0199d3 thru 0x024af2 (0xb120 bytes) 000755 1789052091.545919 write 0x0278be thru 0x036ac4 (0xf207 bytes) 000757 1789052091.552217 mapwrite 0x0356bb thru 0x03ffff (0xa945 bytes) 000758 1789052091.671359 mapread 0x034d9e thru 0x03d65b (0x88be bytes) 000759 1789052091.673558 mapwrite 0x03943f thru 0x03e75e (0x5320 bytes) 000760 1789052091.718666 write 0x01a4a1 thru 0x02a0f7 (0xfc57 bytes) 000761 1789052091.720857 trunc from 0x040000 to 0x01f714 (0x208eb bytes) 000762 1789052091.738572 write 0x009916 thru 0x013433 (0x9b1e bytes) 000764 1789052091.747793 write 0x02a95b thru 0x031753 (0x6df9 bytes) 000765 1789052091.755427 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31754): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 13: 0xf72a37c5 (stale) Mirror 14: 0xf72a37c5 (stale) Mirror 15: 0xf72a37c5 (stale) Mirror 16: 0xa5d33c6b (stale) CRC-32 checksum value for chunk [0, 0x31754): Mirror 1: 0x84295d22 Mirror 17: 0x84295d22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000766 1789052092.941833 mapwrite 0x01b28b thru 0x022b57 (0x78cd bytes) 000767 1789052093.118603 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31754): Mirror 13: 0xf72a37c5 (stale) Mirror 14: 0xf72a37c5 (stale) Mirror 15: 0xf72a37c5 (stale) Mirror 16: 0xa5d33c6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 13: 0xf72a37c5 (stale) Mirror 14: 0xf72a37c5 (stale) Mirror 15: 0xf72a37c5 (stale) Mirror 16: 0xa5d33c6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000768 1789052093.678718 mapwrite 0x012166 thru 0x0125c9 (0x464 bytes) 000770 1789052093.717411 trunc from 0x031754 to 0x03ba8b (0xa338 bytes) 000774 1789052093.737874 trunc from 0x03ba8c to 0x03b759 (0x332 bytes) 000775 1789052093.758452 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b75a): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b75a): Mirror 13: 0xdad8191a (stale) Mirror 14: 0xdad8191a (stale) Mirror 15: 0xdad8191a (stale) Mirror 16: 0xfd001f59 (stale) CRC-32 checksum value for chunk [0, 0x3b75a): Mirror 1: 0xf921ddc4 Mirror 17: 0xf921ddc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b75a: skipped 000776 1789052095.419798 punch from 0x023741 to 0x02ddfb (0xa6bb bytes) 000778 1789052095.499428 read 0x0269cb thru 0x030851 (0x9e87 bytes) 000779 1789052095.517934 mapread 0x037bf0 thru 0x03b759 (0x3b6a bytes) 000780 1789052095.557090 mapwrite 0x0168df thru 0x016a1b (0x13d bytes) 000781 1789052095.671691 MIRROR_RESYNC 6 mirrors 000782 1789052096.715716 mapwrite 0x02371d thru 0x029022 (0x5906 bytes) 000783 1789052096.993238 mapread 0x032cd7 thru 0x03875d (0x5a87 bytes) 000784 1789052097.015510 mapwrite 0x02a794 thru 0x036333 (0xbba0 bytes) 000785 1789052097.100698 mapwrite 0x00a937 thru 0x01751a (0xcbe4 bytes) 000788 1789052097.183506 MIRROR_RESYNC 6 mirrors 000790 1789052098.486032 read 0x02bf89 thru 0x032c4d (0x6cc5 bytes) 000791 1789052098.516789 mapwrite 0x020ea5 thru 0x02545b (0x45b7 bytes) 000792 1789052098.895641 punch from 0x0091ec to 0x00ffc3 (0x6dd8 bytes) 000793 1789052098.910616 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b75a): Mirror 14: 0xbdcd6a5d (stale) Mirror 15: 0xbdcd6a5d (stale) Mirror 16: 0xbdcd6a5d (stale) Mirror 17: 0xbdcd6a5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b75a): Mirror 14: 0xbdcd6a5d (stale) Mirror 15: 0xbdcd6a5d (stale) Mirror 16: 0xbdcd6a5d (stale) Mirror 17: 0xbdcd6a5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b75a: skipped 000794 1789052099.430549 punch from 0x00b470 to 0x01743b (0xbfcc bytes) 000796 1789052099.437977 write 0x0383b8 thru 0x03ffff (0x7c48 bytes) 000799 1789052099.446251 punch from 0x02f2cd to 0x03d2f6 (0xe02a bytes) 000801 1789052099.462474 read 0x0335e1 thru 0x0367e2 (0x3202 bytes) 000806 1789052099.470966 trunc from 0x040000 to 0x029803 (0x167fc bytes) 000807 1789052099.495749 mapread 0x00c383 thru 0x0146d0 (0x834e bytes) 000808 1789052099.521501 read 0x016f73 thru 0x01ff85 (0x9013 bytes) 000810 1789052099.533508 punch from 0x018889 to 0x01d297 (0x4a0f bytes) 000811 1789052099.539033 mapread 0x0207bb thru 0x029803 (0x9049 bytes) 000812 1789052099.540649 punch from 0x01ac2e to 0x0254c2 (0xa895 bytes) 000813 1789052099.546773 write 0x02526e thru 0x029b7b (0x490e bytes) 000814 1789052099.552900 mapwrite 0x025d27 thru 0x02ba40 (0x5d1a bytes) 000815 1789052099.639897 read 0x003960 thru 0x005ae2 (0x2183 bytes) 000816 1789052099.640556 write 0x009b0e thru 0x01346e (0x9961 bytes) 000817 1789052099.642496 punch from 0x008411 to 0x013014 (0xac04 bytes) 000819 1789052099.655167 write 0x01221a thru 0x01d42b (0xb212 bytes) 000820 1789052099.678873 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba41): Mirror 14: 0xf059d3f3 (stale) Mirror 15: 0xf059d3f3 (stale) Mirror 16: 0xf059d3f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba41): Mirror 14: 0xf059d3f3 (stale) Mirror 15: 0xf059d3f3 (stale) Mirror 16: 0xf059d3f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba41: skipped 000821 1789052100.059804 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba41): Mirror 15: 0xf059d3f3 (stale) Mirror 16: 0xf059d3f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba41): Mirror 15: 0xf059d3f3 (stale) Mirror 16: 0xf059d3f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba41: skipped 000822 1789052100.400343 write 0x035fc8 thru 0x03918d (0x31c6 bytes) 000824 1789052100.402141 write 0x02af0e thru 0x032ee2 (0x7fd5 bytes) 000826 1789052100.416079 write 0x030dd5 thru 0x039e82 (0x90ae bytes) 000827 1789052100.418073 mapwrite 0x012ebe thru 0x01cdb6 (0x9ef9 bytes) 000828 1789052100.513060 trunc from 0x039e83 to 0x019693 (0x207ef bytes) 000829 1789052100.533936 mapread 0x015c3e thru 0x019693 (0x3a56 bytes) 000830 1789052100.534885 mapread 0x00ff59 thru 0x0134fe (0x35a6 bytes) 000831 1789052100.535758 read 0x0030d7 thru 0x011e13 (0xed3d bytes) 000832 1789052100.546612 mapread 0x009251 thru 0x00a064 (0xe14 bytes) 000833 1789052100.547779 punch from 0x016741 to 0x018021 (0x18e1 bytes) 000835 1789052100.566189 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19694): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19694): Mirror 15: 0xfa70d063 (stale) Mirror 16: 0xfa70d063 (stale) CRC-32 checksum value for chunk [0, 0x19694): Mirror 1: 0xd3b12a9e Mirror 17: 0xd3b12a9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19694: skipped 000836 1789052102.023998 punch from 0x013cd2 to 0x019693 (0x59c2 bytes) 000838 1789052102.170628 write 0x03302b thru 0x03ffff (0xcfd5 bytes) 000839 1789052102.190922 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xdc051e90 (stale) Mirror 16: 0xdc051e90 (stale) Mirror 17: 0xf27f9627 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c8a6f30 Mirror 18: 0x8c8a6f30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000840 1789052103.951694 punch from 0x030898 to 0x039ed8 (0x9641 bytes) 000841 1789052104.057607 write 0x006f80 thru 0x013dfa (0xce7b bytes) 000842 1789052104.083163 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xdc051e90 (stale) Mirror 16: 0xdc051e90 (stale) Mirror 17: 0xf27f9627 (stale) Mirror 18: 0x8c8a6f30 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xed340a99 Mirror 19: 0xed340a99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1789052105.610816 write 0x0206b0 thru 0x02d408 (0xcd59 bytes) 000844 1789052105.696985 mapwrite 0x02bd12 thru 0x0355c9 (0x98b8 bytes) 000845 1789052105.772172 write 0x009584 thru 0x009823 (0x2a0 bytes) 000847 1789052105.780120 mapwrite 0x0183a3 thru 0x0264ef (0xe14d bytes) 000852 1789052105.879143 MIRROR_RESYNC 6 mirrors 000853 1789052107.244742 trunc from 0x040000 to 0x01d94e (0x226b1 bytes) 000854 1789052107.320914 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1d94f): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d94f): Mirror 15: 0xbaff4a3f (stale) Mirror 16: 0xbaff4a3f (stale) Mirror 17: 0xbaff4a3f (stale) Mirror 18: 0xbaff4a3f (stale) Mirror 19: 0xbaff4a3f (stale) CRC-32 checksum value for chunk [0, 0x1d94f): Mirror 1: 0xbaff4a3f Mirror 20: 0xbaff4a3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d94f: skipped 000856 1789052108.523895 read 0x00079a thru 0x00a24f (0x9ab6 bytes) 000857 1789052108.551083 trunc from 0x01d94f to 0x022d51 (0x5403 bytes) 000858 1789052108.649596 trunc from 0x022d52 to 0x0228c5 (0x48c bytes) 000860 1789052108.679439 mapread 0x010e5a thru 0x0193af (0x8556 bytes) 000862 1789052108.706199 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x228c6): Mirror 15: 0x331858c (stale) Mirror 17: 0x331858c (stale) Mirror 18: 0x331858c (stale) Mirror 19: 0x331858c (stale) Mirror 20: 0x31087037 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228c6): Mirror 15: 0x331858c (stale) Mirror 17: 0x331858c (stale) Mirror 18: 0x331858c (stale) Mirror 19: 0x331858c (stale) Mirror 20: 0x31087037 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228c6: skipped 000863 1789052109.086468 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x228c6): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228c6): Mirror 15: 0x331858c (stale) Mirror 17: 0x331858c (stale) Mirror 18: 0x331858c (stale) Mirror 19: 0x331858c (stale) Mirror 20: 0x31087037 (stale) CRC-32 checksum value for chunk [0, 0x228c6): Mirror 1: 0x31087037 Mirror 21: 0x31087037 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228c6: skipped 000865 1789052110.559200 write 0x01164e thru 0x017b7c (0x652f bytes) 000866 1789052110.665346 write 0x038c58 thru 0x03d7ac (0x4b55 bytes) 000868 1789052110.669021 punch from 0x02e779 to 0x036eda (0x8762 bytes) 000871 1789052110.691880 mapread 0x002c29 thru 0x01148d (0xe865 bytes) 000873 1789052110.709330 write 0x036c45 thru 0x03a1e0 (0x359c bytes) 000874 1789052110.714885 mapwrite 0x00deb4 thru 0x017cce (0x9e1b bytes) 000875 1789052110.775594 mapwrite 0x00bc05 thru 0x01af46 (0xf342 bytes) 000876 1789052110.859839 trunc from 0x03d7ad to 0x0304c1 (0xd2eb bytes) 000878 1789052110.897314 trunc from 0x0304c2 to 0x022c40 (0xd881 bytes) 000879 1789052110.916671 mapread 0x01670b thru 0x01a7ac (0x40a2 bytes) 000881 1789052110.923875 MIRROR_RESYNC 7 mirrors 000882 1789052112.711709 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000883 1789052113.329511 write 0x03b18e thru 0x03bc2f (0xaa2 bytes) 000884 1789052113.441163 punch from 0x0166ab to 0x023699 (0xcfef bytes) 000885 1789052113.460392 read 0x02e671 thru 0x032f46 (0x48d6 bytes) 000887 1789052113.475164 mapwrite 0x036c15 thru 0x03d2be (0x66aa bytes) 000888 1789052113.662025 read 0x025da1 thru 0x035251 (0xf4b1 bytes) 000890 1789052113.690428 mapread 0x033adc thru 0x034cdf (0x1204 bytes) 000892 1789052113.694600 punch from 0x02d5ec to 0x035b0d (0x8522 bytes) 000894 1789052113.714707 mapwrite 0x000654 thru 0x005ff4 (0x59a1 bytes) 000895 1789052113.834174 write 0x036568 thru 0x03ffff (0x9a98 bytes) 000896 1789052113.842035 punch from 0x01de58 to 0x0283d1 (0xa57a bytes) 000897 1789052113.870127 mapwrite 0x000c82 thru 0x00b148 (0xa4c7 bytes) 000898 1789052113.986953 trunc from 0x040000 to 0x03ed17 (0x12e8 bytes) 000901 1789052114.018513 write 0x027ef3 thru 0x02c8dd (0x49eb bytes) 000902 1789052114.030732 trunc from 0x03ed18 to 0x02c583 (0x12794 bytes) 000904 1789052114.075657 mapread 0x01a866 thru 0x01e549 (0x3ce4 bytes) 000905 1789052114.097640 mapread 0x00fbe8 thru 0x01b96f (0xbd88 bytes) 000906 1789052114.100383 read 0x028a7b thru 0x02c583 (0x3b09 bytes) 000908 1789052114.109953 read 0x002611 thru 0x00eb12 (0xc502 bytes) 000909 1789052114.111016 write 0x01f24b thru 0x021cba (0x2a70 bytes) 000911 1789052114.123484 read 0x024894 thru 0x02c583 (0x7cf0 bytes) 000912 1789052114.124333 mapread 0x0287ea thru 0x02c583 (0x3d9a bytes) 000913 1789052114.128679 write 0x00de66 thru 0x012cd3 (0x4e6e bytes) 000914 1789052114.138349 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c584): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c584): Mirror 15: 0xcfbb22a (stale) Mirror 17: 0xcfbb22a (stale) Mirror 18: 0xcfbb22a (stale) Mirror 19: 0xcfbb22a (stale) Mirror 20: 0xcfbb22a (stale) CRC-32 checksum value for chunk [0, 0x2c584): Mirror 1: 0xc3ddc59c Mirror 21: 0xc3ddc59c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c584: skipped 000915 1789052115.086616 punch from 0x02454c to 0x028e1d (0x48d2 bytes) 000916 1789052115.165631 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c584): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c584): Mirror 15: 0xcfbb22a (stale) Mirror 17: 0xcfbb22a (stale) Mirror 18: 0xcfbb22a (stale) Mirror 19: 0xcfbb22a (stale) Mirror 20: 0xcfbb22a (stale) Mirror 21: 0xc3ddc59c (stale) CRC-32 checksum value for chunk [0, 0x2c584): Mirror 1: 0x93c54a2 Mirror 22: 0x93c54a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c584: skipped 000917 1789052116.301831 write 0x03c03f thru 0x03ffff (0x3fc1 bytes) 000918 1789052116.433027 MIRROR_RESYNC 8 mirrors 000920 1789052117.764824 trunc from 0x040000 to 0x006604 (0x399fb bytes) 000921 1789052117.861142 mapread 0x005fb5 thru 0x006604 (0x650 bytes) 000922 1789052117.881436 mapwrite 0x02f0ba thru 0x03ef0c (0xfe53 bytes) 000924 1789052118.098095 mapread 0x038311 thru 0x03ef0c (0x6bfc bytes) 000925 1789052118.104763 MIRROR_RESYNC 8 mirrors 000928 1789052120.500262 punch from 0x031316 to 0x0377ba (0x64a5 bytes) 000929 1789052120.589542 MIRROR_RESYNC 8 mirrors 000930 1789052123.048231 trunc from 0x03ef0d to 0x00d55a (0x319b2 bytes) 000933 1789052123.213609 mapread 0x008371 thru 0x009ee7 (0x1b77 bytes) 000934 1789052123.233484 trunc from 0x00d55b to 0x001071 (0xc4e9 bytes) 000937 1789052123.266867 write 0x00c9ba thru 0x00dc4c (0x1293 bytes) 000938 1789052123.270008 punch from 0x00d3a2 to 0x00dc4c (0x8ab bytes) 000942 1789052123.292206 read 0x001c81 thru 0x0036b3 (0x1a33 bytes) 000943 1789052123.300905 read 0x00a3f9 thru 0x00dc4c (0x3854 bytes) 000944 1789052123.319187 write 0x024660 thru 0x031cfe (0xd69f bytes) 000947 1789052123.328302 mapread 0x01a3d4 thru 0x024043 (0x9c70 bytes) 000948 1789052123.373170 punch from 0x00112f to 0x00849d (0x736f bytes) 000950 1789052123.405843 mapwrite 0x019666 thru 0x01c4a6 (0x2e41 bytes) 000951 1789052123.523550 mapread 0x0206a1 thru 0x024e23 (0x4783 bytes) 000952 1789052123.528930 punch from 0x0105d0 to 0x01b24c (0xac7d bytes) 000953 1789052123.544881 MIRROR_RESYNC 8 mirrors 000954 1789052125.885719 punch from 0x003e20 to 0x013ade (0xfcbf bytes) 000955 1789052126.056785 trunc from 0x031cff to 0x02cc56 (0x50a8 bytes) 000956 1789052126.115185 mapwrite 0x03b2e1 thru 0x03ffff (0x4d1f bytes) 000958 1789052126.387675 write 0x03b79c thru 0x03ffff (0x4864 bytes) 000960 1789052126.392554 mapread 0x02bb2f thru 0x03a6eb (0xebbd bytes) 000961 1789052126.447102 read 0x03a001 thru 0x03ffff (0x5fff bytes) 000965 1789052126.461879 mapwrite 0x03dda4 thru 0x03e9df (0xc3c bytes) 000967 1789052126.588703 trunc from 0x040000 to 0x02184a (0x1e7b5 bytes) 000968 1789052126.670441 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2184b): Mirror 15: 0x7d6b6d89 (stale) Mirror 18: 0x7d6b6d89 (stale) Mirror 19: 0x7d6b6d89 (stale) Mirror 20: 0x7d6b6d89 (stale) Mirror 21: 0x7d6b6d89 (stale) Mirror 22: 0x7d6b6d89 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2184b): Mirror 15: 0x7d6b6d89 (stale) Mirror 18: 0x7d6b6d89 (stale) Mirror 19: 0x7d6b6d89 (stale) Mirror 20: 0x7d6b6d89 (stale) Mirror 21: 0x7d6b6d89 (stale) Mirror 22: 0x7d6b6d89 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2184b: skipped 000969 1789052127.343962 write 0x02711b thru 0x0302dc (0x91c2 bytes) 000970 1789052127.345340 mapread 0x008e5f thru 0x00f58f (0x6731 bytes) 000973 1789052127.401098 trunc from 0x0302dd to 0x03322f (0x2f53 bytes) 000974 1789052127.431922 mapwrite 0x036c51 thru 0x03c934 (0x5ce4 bytes) 000975 1789052127.555097 MIRROR_RESYNC 7 mirrors 000978 1789052129.042044 trunc from 0x03c935 to 0x0384bc (0x4478 bytes) 000979 1789052129.110717 write 0x00fd79 thru 0x01d385 (0xd60d bytes) 000981 1789052129.120184 punch from 0x02d679 to 0x02fb46 (0x24ce bytes) 000982 1789052129.134457 mapread 0x000993 thru 0x0015f8 (0xc66 bytes) 000983 1789052129.143763 read 0x0081ea thru 0x01208c (0x9ea3 bytes) 000984 1789052129.152972 read 0x0024f0 thru 0x009fc3 (0x7ad4 bytes) 000985 1789052129.161902 punch from 0x001f3a to 0x0118d4 (0xf99b bytes) 000986 1789052129.165953 trunc from 0x0384bd to 0x004d57 (0x33765 bytes) 000987 1789052129.187640 mapwrite 0x01d1e9 thru 0x027702 (0xa51a bytes) 000988 1789052129.287575 write 0x0057f5 thru 0x00fa22 (0xa22e bytes) 000989 1789052129.297199 mapread 0x01b918 thru 0x027702 (0xbdeb bytes) 000990 1789052129.309662 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27703): Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27703): Mirror 15: 0xcc1d3587 (stale) Mirror 18: 0xcc1d3587 (stale) Mirror 19: 0xcc1d3587 (stale) Mirror 20: 0xcc1d3587 (stale) Mirror 21: 0xcc1d3587 (stale) Mirror 22: 0xcc1d3587 (stale) CRC-32 checksum value for chunk [0, 0x27703): Mirror 1: 0x96d56df9 Mirror 23: 0x96d56df9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27703: skipped 000991 1789052129.955228 read 0x0005c4 thru 0x003949 (0x3386 bytes) 000992 1789052129.972041 MIRROR_RESYNC 8 mirrors 000993 1789052131.453226 MIRROR_EXTEND to 9 mirrors 000994 1789052132.284407 mapwrite 0x0096e4 thru 0x0189b0 (0xf2cd bytes) 000995 1789052132.494082 punch from 0x026643 to 0x027702 (0x10c0 bytes) 000997 1789052132.500045 mapwrite 0x038c51 thru 0x03ffff (0x73af bytes) 000998 1789052132.568391 punch from 0x028e1b to 0x030956 (0x7b3c bytes) 000999 1789052132.572723 write 0x01ea2e thru 0x01fdfd (0x13d0 bytes) 001000 1789052132.580771 trunc from 0x040000 to 0x01c105 (0x23efa bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 10:55:40 (1789052140) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 10:55:45 (1789052145) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.333277 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354123 s, 3.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 10:55:55 (1789052155) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6122560 .. Extending file size to 4755424 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5939520 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6852256 .. Extending file size to 9193984 .. Extending file size to 7944768 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8359904 .. Extending file size to 8402880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8470304 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57383 57384 57387 57388 57390 done 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:57:16 (1789052236) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3909248 .. Extending file size to 7769792 .. done Extending file size to 5354496 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6100352 .. Extending file size to 6616512 .. Extending file size to 8330528 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996640 .. Extending file size to 9127936 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8641408 .. Waiting 58696 58697 58700 58702 58704 done 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:58:35 (1789052315) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115026 s, 18.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283543 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5d11fb42 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:58:54 (1789052334) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109095 s, 19.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 10:59:00 (1789052340) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:59:07 (1789052347) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18745 s, 11.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:59:14 (1789052354) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237808 s, 8.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:59:23 (1789052363) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15842 s, 13.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:59:56 (1789052396) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118672 s, 8.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:00:58 (1789052458) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222693 s, 9.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:01:52 (1789052512) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. Creating new pool test_204f oleg414-server: Pool lustre.test_204f created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.451017 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg414-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:03:11 (1789052591) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:03:17 (1789052597) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:03:24 (1789052604) oleg414-server: Pool lustre.test_206 created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg414-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:03:47 (1789052627) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631245 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651152 s, 16.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xafcb5e16 Mirror 3: 0xafcb5e16 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:04:01 (1789052641) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:04:06 (1789052646) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:04:09 (1789052649) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.100487 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xe50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xe50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xe50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1550000 Read this (4 bytes): BBBB 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:04:44 (1789052684) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0233096 s, 351 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:04:51 (1789052691) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064627 s, 16.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:04:52.000000000 -0400 Modify: 2026-09-10 11:04:52.000000000 -0400 Change: 2026-09-10 11:04:54.131000000 -0400 Birth: 2026-09-10 11:04:52.721000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:05:00 (1789052700) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:05:03 (1789052703) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.02225 s, 40.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0502003 s, 81.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 11:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 11:05 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:05:12 (1789052712) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:05:19 (1789052719) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1827 sec =========== 11:05:28 (1789052728) === sanity-flr: start cleanup 11:05:30 (1789052730) === === sanity-flr: finish cleanup 11:05:36 (1789052736) ===