-----============= acceptance-small: sanity-flr ============----- Thu Aug 6 20:29:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg120-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:30:23 (1786062623) === oleg120-client.virtnet: executing check_config_client /mnt/lustre oleg120-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg120-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cc590a7e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cc590a7e800.idle_timeout=debug disable quota as required oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:31:03 (1786062663) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7663] == sanity-flr test 0a: lfs mirror create with -N option == 20:31:05 (1786062665) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:31:23 (1786062683) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:31:27 (1786062687) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:31:31 (1786062691) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:31:55 (1786062715) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:31:58 (1786062718) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:32:01 (1786062721) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cc590a7e800.stats=clear osc.lustre-OST0001-osc-ffff8cc590a7e800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:32:09 (1786062729) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:32:20 (1786062740) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:32:30 (1786062750) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:32:33 (1786062753) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:32:41 (1786062761) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 20:32:50 (1786062770) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:32:58 (1786062778) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:33:01 (1786062781) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:33:08 (1786062788) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:33:18 (1786062798) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:33:25 (1786062805) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.23631 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131408 s, 8.0 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:33:38 (1786062818) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.959298 s, 21.9 MB/s PASS 22 (14s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:33:52 (1786062832) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:34:04 (1786062844) Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:35:20 (1786062920) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.683483 s, 12.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:36:07 (1786062967) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:37:35 (1786063055) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.31249 s, 9.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 6 20:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.092s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.790s user 0m0.002s sys 0m0.201s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.101s user 0m0.001s sys 0m0.172s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cc590a7e800.ost_server_uuid in FULL state after 0 sec PASS 33b (111s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:39:26 (1786063166) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:39:30 (1786063170) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:39:34 (1786063174) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:39:37 (1786063177) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 20:39:46 (1786063186) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.643182 s, 19.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:39:59 (1786063199) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649036 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280964 s, 37.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:40:06 (1786063206) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:40:15 (1786063215) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135292 s, 757 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019184 s, 534 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00379687 s, 270 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000966127 s, 1.1 MB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 20:40:29 (1786063229) create test files with size 15759224 .. e51e63ad4fb5a6f7bcfbc2dde4c6624e - 4421b8a7535e2e7331608b27fcc86765 - 311fac7a74287c37cf63847c4eaf0226 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 20:40:54 (1786063254) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:41:09 (1786063269) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:41:16 (1786063276) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107989 s, 9.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355948 s, 29.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 20:41:25 (1786063285) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151832 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.361908 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228076 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 20:41:39 (1786063299) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:41:42 (1786063302) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:41:44 (1786063304) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496543 s, 21.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:41:52 (1786063312) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:41:55 (1786063315) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:42:06 (1786063326) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.380712 s, 27.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:42:19 (1786063339) llite.lustre-ffff8cc5875ce800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.379571 s, 27.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8cc5851c5800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:42:36 (1786063356) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:43:11 (1786063391) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:43:46 (1786063426) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:43:49 (1786063429) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.292193 s, 10.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127686 s, 24.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6ba59330 (nosync,stale) Mirror 2: 0x6ba59330 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5fc63d5590d6ea02bc3dab2d16129306 - mirror 1 checksum: 0cec1338e17ae2490bb2aa0fe6ab0772 - mirror 2 checksum: 5fc63d5590d6ea02bc3dab2d16129306 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T20:43:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T20:43:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5fc63d5590d6ea02bc3dab2d16129306 - mirror 1 checksum: 0cec1338e17ae2490bb2aa0fe6ab0772 - mirror 2 checksum: 0cec1338e17ae2490bb2aa0fe6ab0772 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T20:43:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T20:43:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:44:01 (1786063441) Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 20:44:09 (1786063449) shut down facet: mds1 facet_host: oleg120-server facet_failover_host: oleg120-server Failover mds1 to oleg120-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 20:44:46 (1786063486) targets are mounted 20:44:46 (1786063486) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434461 s, 24.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:45:06 (1786063506) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:45:10 (1786063510) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:45:13 (1786063513) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00687268 s, 2.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:45:22 (1786063522) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:45:26 (1786063526) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00458144 s, 894 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0632502 s, 64.8 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:45:34 (1786063534) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:45:58 (1786063558) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:46:28 (1786063588) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:46:52 (1786063612) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:47:17 (1786063637) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:47:24 (1786063644) mirror create pid 54357 mirror split pid 54358 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54357 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54358 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:48:31 (1786063711) Seed set to 3713 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786063713.551689 mapwrite 0x02bd01 thru 0x0347c6 (0x8ac6 bytes) 000003 1786063713.662517 MIRROR_EXTEND to 2 mirrors 000004 1786063714.184456 mapwrite 0x00bf9a thru 0x00cdaa (0xe11 bytes) 000007 1786063714.378253 mapread 0x011d02 thru 0x01bbe6 (0x9ee5 bytes) 000008 1786063714.386308 write 0x02b42c thru 0x0381b0 (0xcd85 bytes) truncating to largest ever: 0x2e75e 000009 1786063714.392548 trunc from 0x0381b1 to 0x02e75d (0x9a53 bytes) 000012 1786063714.437673 trunc from 0x02e75e to 0x021555 (0xd208 bytes) 000016 1786063714.466674 MIRROR_RESYNC 2 mirrors 000019 1786063715.111921 read 0x01f5c7 thru 0x021555 (0x1f8f bytes) 000020 1786063715.117055 read 0x019b1f thru 0x01c25c (0x273e bytes) 000021 1786063715.139358 mapwrite 0x01f997 thru 0x027d59 (0x83c3 bytes) 000022 1786063715.340683 mapwrite 0x01e250 thru 0x020501 (0x22b2 bytes) 000026 1786063715.418068 trunc from 0x027d5a to 0x0150f0 (0x12c69 bytes) 000027 1786063715.444249 mapread 0x0135c4 thru 0x0150f0 (0x1b2d bytes) 000028 1786063715.472687 MIRROR_RESYNC 2 mirrors 000029 1786063716.069732 mapwrite 0x035dcb thru 0x03ffff (0xa235 bytes) 000030 1786063716.249660 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7057cb42 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6cef7c74 Mirror 3: 0x6cef7c74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000031 1786063717.419933 MIRROR_RESYNC 3 mirrors 000035 1786063718.432830 read 0x02e556 thru 0x031df5 (0x38a0 bytes) 000036 1786063718.454298 mapread 0x0228d6 thru 0x024e55 (0x2580 bytes) 000038 1786063718.515396 MIRROR_RESYNC 3 mirrors 000039 1786063719.194778 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000040 1786063719.729736 trunc from 0x040000 to 0x022107 (0x1def8 bytes) 000041 1786063720.040735 write 0x028110 thru 0x02f91e (0x780f bytes) 000046 1786063720.049156 mapread 0x00fd82 thru 0x0111b2 (0x1431 bytes) 000047 1786063720.084318 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000049 1786063720.258438 mapwrite 0x01deb7 thru 0x02cef4 (0xf03e bytes) 000055 1786063720.409821 mapwrite 0x021611 thru 0x02af9b (0x998b bytes) 000057 1786063720.532446 read 0x004529 thru 0x010738 (0xc210 bytes) 000060 1786063720.550440 trunc from 0x02f91f to 0x0013d9 (0x2e545 bytes) 000062 1786063720.595251 mapread 0x0011bd thru 0x0013d9 (0x21d bytes) 000064 1786063720.618478 trunc from 0x0013da to 0x020462 (0x1f089 bytes) 000065 1786063720.641732 write 0x029e48 thru 0x0342f7 (0xa4b0 bytes) 000066 1786063720.643287 read 0x02b585 thru 0x0342f7 (0x8d73 bytes) 000069 1786063720.667643 mapread 0x024694 thru 0x03314b (0xeab8 bytes) 000070 1786063720.686197 write 0x024f4a thru 0x02d7e4 (0x889b bytes) 000074 1786063720.690531 read 0x01485b thru 0x023703 (0xeea9 bytes) 000078 1786063720.695911 write 0x03e571 thru 0x03ffff (0x1a8f bytes) 000079 1786063720.705972 trunc from 0x040000 to 0x00d30c (0x32cf3 bytes) 000080 1786063720.748262 mapwrite 0x03d0be thru 0x03ffff (0x2f42 bytes) 000084 1786063720.870654 read 0x0354d9 thru 0x03f999 (0xa4c1 bytes) 000085 1786063720.871524 mapwrite 0x03b7f7 thru 0x03ffff (0x4809 bytes) 000088 1786063721.012191 write 0x02003d thru 0x026df5 (0x6db9 bytes) 000091 1786063721.025271 mapread 0x027811 thru 0x036475 (0xec65 bytes) 000096 1786063721.042342 mapwrite 0x0219f1 thru 0x028c89 (0x7299 bytes) 000097 1786063721.141414 MIRROR_EXTEND to 2 mirrors 000098 1786063722.083511 mapread 0x00c30b thru 0x0104e3 (0x41d9 bytes) 000099 1786063722.145376 write 0x015a99 thru 0x01b9e3 (0x5f4b bytes) 000103 1786063722.282522 write 0x029733 thru 0x02e09b (0x4969 bytes) 000105 1786063722.303535 mapwrite 0x022c49 thru 0x02c1d7 (0x958f bytes) 000106 1786063722.397352 trunc from 0x040000 to 0x02bd34 (0x142cb bytes) 000107 1786063722.484880 mapread 0x02a37c thru 0x02bd34 (0x19b9 bytes) 000108 1786063722.486156 mapread 0x018818 thru 0x0234db (0xacc4 bytes) 000109 1786063722.494247 write 0x02ec65 thru 0x038414 (0x97b0 bytes) 000110 1786063722.506946 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38415): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38415): Mirror 2: 0xc9e01eb7 (stale) CRC-32 checksum value for chunk [0, 0x38415): Mirror 1: 0x85150dfd Mirror 3: 0x85150dfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38415: skipped 000111 1786063723.922200 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38415): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38415): Mirror 2: 0xc9e01eb7 (stale) CRC-32 checksum value for chunk [0, 0x38415): Mirror 1: 0x85150dfd Mirror 3: 0x85150dfd Mirror 4: 0x85150dfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38415: skipped 000112 1786063725.121320 mapread 0x01abd9 thru 0x01e8ec (0x3d14 bytes) 000113 1786063725.176406 mapwrite 0x00becd thru 0x00e9ae (0x2ae2 bytes) 000114 1786063725.526563 trunc from 0x038415 to 0x006e53 (0x315c1 bytes) 000115 1786063725.556784 mapread 0x00150f thru 0x006e53 (0x5945 bytes) 000116 1786063725.570738 mapwrite 0x01e424 thru 0x02a172 (0xbd4f bytes) 000117 1786063725.743001 mapread 0x0054f0 thru 0x00dfa8 (0x8ab9 bytes) 000118 1786063725.761393 read 0x01ceb6 thru 0x023ada (0x6c25 bytes) 000119 1786063725.769964 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a173): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a173): Mirror 2: 0xd49ad3a1 (stale) Mirror 3: 0xa2513477 (stale) Mirror 4: 0xa2513477 (stale) CRC-32 checksum value for chunk [0, 0x2a173): Mirror 1: 0x1d9fd0ec Mirror 5: 0x1d9fd0ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a173: skipped 000121 1786063727.569326 mapread 0x019964 thru 0x02387e (0x9f1b bytes) 000122 1786063727.681051 MIRROR_RESYNC 5 mirrors 000124 1786063729.155400 write 0x004ed0 thru 0x008316 (0x3447 bytes) 000125 1786063729.311086 mapwrite 0x034faf thru 0x03ffff (0xb051 bytes) 000126 1786063729.518553 read 0x000fe9 thru 0x007bd2 (0x6bea bytes) 000127 1786063729.544305 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ac4fad7 (stale) Mirror 4: 0x4ac4fad7 (stale) Mirror 5: 0x4ac4fad7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ac4fad7 (stale) Mirror 4: 0x4ac4fad7 (stale) Mirror 5: 0x4ac4fad7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000128 1786063730.189615 read 0x01265c thru 0x02239e (0xfd43 bytes) 000130 1786063730.207078 read 0x03cb4a thru 0x03ffff (0x34b6 bytes) 000132 1786063730.251152 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ac4fad7 (stale) Mirror 4: 0x4ac4fad7 (stale) Mirror 5: 0x4ac4fad7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9bebf4e0 Mirror 6: 0x9bebf4e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000133 1786063732.349838 trunc from 0x040000 to 0x009f58 (0x360a7 bytes) 000134 1786063732.606087 read 0x001648 thru 0x009f58 (0x8911 bytes) 000135 1786063732.638933 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x9f59): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f59): Mirror 3: 0x76f2292 (stale) Mirror 4: 0x76f2292 (stale) Mirror 5: 0x76f2292 (stale) Mirror 6: 0xdab609bb (stale) CRC-32 checksum value for chunk [0, 0x9f59): Mirror 1: 0xdab609bb Mirror 7: 0xdab609bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f59: skipped 000136 1786063734.841134 mapread 0x008970 thru 0x009f58 (0x15e9 bytes) 000138 1786063734.978074 write 0x019451 thru 0x01d4d2 (0x4082 bytes) 000141 1786063735.149364 read 0x01cf8b thru 0x01d4d2 (0x548 bytes) 000142 1786063735.150272 mapwrite 0x00c73c thru 0x0162e8 (0x9bad bytes) 000143 1786063735.211867 mapwrite 0x01d834 thru 0x02906b (0xb838 bytes) 000145 1786063735.382046 mapwrite 0x00a484 thru 0x0140e8 (0x9c65 bytes) 000147 1786063735.479283 mapwrite 0x03cd09 thru 0x03fefb (0x31f3 bytes) truncating to largest ever: 0x3e864 000148 1786063735.551541 trunc from 0x03fefc to 0x03e863 (0x1698 bytes) 000152 1786063735.569709 mapread 0x0125f7 thru 0x018a80 (0x648a bytes) 000153 1786063735.571204 trunc from 0x03e864 to 0x026ef5 (0x1796e bytes) 000154 1786063735.583522 mapread 0x01f226 thru 0x0252d8 (0x60b3 bytes) 000155 1786063735.589541 MIRROR_RESYNC 6 mirrors 000157 1786063739.063546 MIRROR_EXTEND to 7 mirrors 000158 1786063740.308524 read 0x01c8d0 thru 0x01d38a (0xabb bytes) 000159 1786063740.321174 trunc from 0x026ef6 to 0x039590 (0x1269b bytes) 000161 1786063740.421810 mapwrite 0x027958 thru 0x035789 (0xde32 bytes) 000162 1786063740.511022 mapread 0x01fa10 thru 0x025451 (0x5a42 bytes) 000163 1786063740.519601 write 0x033bbd thru 0x038a6d (0x4eb1 bytes) 000165 1786063740.526720 MIRROR_RESYNC 7 mirrors 000167 1786063745.427553 read 0x00bb58 thru 0x00d9e5 (0x1e8e bytes) 000168 1786063745.446025 read 0x018944 thru 0x0252d9 (0xc996 bytes) 000169 1786063745.477740 mapread 0x0084b1 thru 0x00c26e (0x3dbe bytes) 000171 1786063745.478845 mapread 0x03686b thru 0x039590 (0x2d26 bytes) 000172 1786063745.479515 write 0x00af40 thru 0x00e8a1 (0x3962 bytes) 000173 1786063745.677908 write 0x028c6b thru 0x02ed08 (0x609e bytes) 000174 1786063745.687816 read 0x019366 thru 0x02632b (0xcfc6 bytes) 000176 1786063745.698689 trunc from 0x039591 to 0x0297a5 (0xfdeb bytes) 000177 1786063745.726952 mapwrite 0x01bebb thru 0x0275bc (0xb702 bytes) 000178 1786063745.830079 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x297a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297a6): Mirror 3: 0x5549978e (stale) Mirror 4: 0x5549978e (stale) Mirror 5: 0x5549978e (stale) Mirror 6: 0x5549978e (stale) Mirror 7: 0x5549978e (stale) Mirror 8: 0x5549978e (stale) CRC-32 checksum value for chunk [0, 0x297a6): Mirror 1: 0x9cf9ceb6 Mirror 9: 0x9cf9ceb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297a6: skipped 000179 1786063746.689107 MIRROR_RESYNC 8 mirrors 000180 1786063749.407073 read 0x01056e thru 0x014baa (0x463d bytes) 000182 1786063749.434033 trunc from 0x0297a6 to 0x028136 (0x166f bytes) 000184 1786063749.604494 read 0x0074d8 thru 0x00aa21 (0x354a bytes) 000187 1786063749.626548 read 0x01b826 thru 0x024735 (0x8f10 bytes) 000189 1786063749.628891 trunc from 0x028137 to 0x039645 (0x1150f bytes) 000190 1786063749.682366 write 0x01b9d9 thru 0x021d84 (0x63ac bytes) 000191 1786063749.685567 read 0x00e086 thru 0x01450f (0x648a bytes) 000193 1786063749.687528 trunc from 0x039646 to 0x0394af (0x196 bytes) 000194 1786063749.719299 MIRROR_RESYNC 8 mirrors 000197 1786063753.412751 MIRROR_EXTEND to 9 mirrors 000201 1786063754.373008 mapwrite 0x006003 thru 0x00eb54 (0x8b52 bytes) 000202 1786063754.675085 read 0x008505 thru 0x015033 (0xcb2f bytes) 000204 1786063754.685607 write 0x02909c thru 0x02b1f1 (0x2156 bytes) 000205 1786063754.699419 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394b0): Mirror 3: 0x3a18e6c3 (stale) Mirror 4: 0x3a18e6c3 (stale) Mirror 5: 0x3a18e6c3 (stale) Mirror 6: 0x3a18e6c3 (stale) Mirror 7: 0x3a18e6c3 (stale) Mirror 9: 0x3a18e6c3 (stale) Mirror 10: 0x3a18e6c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394b0): Mirror 3: 0x3a18e6c3 (stale) Mirror 4: 0x3a18e6c3 (stale) Mirror 5: 0x3a18e6c3 (stale) Mirror 6: 0x3a18e6c3 (stale) Mirror 7: 0x3a18e6c3 (stale) Mirror 9: 0x3a18e6c3 (stale) Mirror 10: 0x3a18e6c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394b0: skipped 000206 1786063755.519622 write 0x02651f thru 0x033f98 (0xda7a bytes) 000207 1786063755.544596 mapwrite 0x0061c5 thru 0x00a512 (0x434e bytes) 000209 1786063755.829249 trunc from 0x0394b0 to 0x0344e6 (0x4fc9 bytes) 000210 1786063755.866962 mapread 0x00414a thru 0x00e010 (0x9ec7 bytes) 000212 1786063755.874329 read 0x00637b thru 0x014c8d (0xe913 bytes) 000214 1786063755.895517 write 0x037713 thru 0x039e53 (0x2741 bytes) 000216 1786063755.907239 read 0x008d36 thru 0x0185e3 (0xf8ae bytes) 000217 1786063755.958869 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e54): Mirror 4: 0x209728a4 (stale) Mirror 5: 0x209728a4 (stale) Mirror 6: 0x209728a4 (stale) Mirror 7: 0x209728a4 (stale) Mirror 9: 0x209728a4 (stale) Mirror 10: 0x209728a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e54): Mirror 4: 0x209728a4 (stale) Mirror 5: 0x209728a4 (stale) Mirror 6: 0x209728a4 (stale) Mirror 7: 0x209728a4 (stale) Mirror 9: 0x209728a4 (stale) Mirror 10: 0x209728a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e54: skipped 000220 1786063756.748070 mapread 0x009f47 thru 0x011d48 (0x7e02 bytes) 000222 1786063756.768022 mapread 0x01dba2 thru 0x01f222 (0x1681 bytes) 000223 1786063756.780085 trunc from 0x039e54 to 0x025f4b (0x13f08 bytes) 000224 1786063756.813363 read 0x0112f4 thru 0x01270e (0x141b bytes) 000226 1786063756.814736 mapread 0x000b10 thru 0x00a52a (0x9a1b bytes) 000227 1786063756.843065 write 0x026b58 thru 0x0354eb (0xe994 bytes) 000231 1786063756.864440 MIRROR_RESYNC 7 mirrors 000232 1786063758.776030 read 0x0300f9 thru 0x0354eb (0x53f3 bytes) 000233 1786063758.784846 trunc from 0x0354ec to 0x00b078 (0x2a473 bytes) 000235 1786063758.867914 mapread 0x00026c thru 0x00b078 (0xae0d bytes) 000236 1786063758.897112 mapwrite 0x03034d thru 0x030fcb (0xc7f bytes) 000237 1786063759.083053 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30fcc): Mirror 4: 0x584accf7 (stale) Mirror 5: 0x584accf7 (stale) Mirror 6: 0x584accf7 (stale) Mirror 9: 0x584accf7 (stale) Mirror 10: 0x584accf7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fcc): Mirror 4: 0x584accf7 (stale) Mirror 5: 0x584accf7 (stale) Mirror 6: 0x584accf7 (stale) Mirror 9: 0x584accf7 (stale) Mirror 10: 0x584accf7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fcc: skipped 000242 1786063759.593980 MIRROR_RESYNC 6 mirrors 000243 1786063761.439770 mapread 0x023f16 thru 0x030fcb (0xd0b6 bytes) 000244 1786063761.505723 mapread 0x02bbeb thru 0x02eaa7 (0x2ebd bytes) 000245 1786063761.509237 write 0x02565d thru 0x02ef74 (0x9918 bytes) 000246 1786063761.588402 read 0x0248ad thru 0x030115 (0xb869 bytes) 000247 1786063761.598103 read 0x01de11 thru 0x0202fd (0x24ed bytes) 000248 1786063761.607946 mapwrite 0x00be84 thru 0x016220 (0xa39d bytes) 000249 1786063761.674078 trunc from 0x030fcc to 0x03e85f (0xd894 bytes) 000250 1786063761.698253 mapread 0x002688 thru 0x0067ae (0x4127 bytes) 000253 1786063761.701336 read 0x03424b thru 0x03e85f (0xa615 bytes) 000254 1786063761.710612 mapread 0x03a4c2 thru 0x03e85f (0x439e bytes) 000255 1786063761.712448 mapwrite 0x0305c2 thru 0x031d04 (0x1743 bytes) 000258 1786063761.764603 MIRROR_RESYNC 6 mirrors 000259 1786063764.200233 write 0x03bb98 thru 0x03ffff (0x4468 bytes) 000262 1786063764.272927 mapwrite 0x03da75 thru 0x03ffff (0x258b bytes) 000263 1786063764.306512 trunc from 0x040000 to 0x0053d1 (0x3ac2e bytes) 000264 1786063764.336551 trunc from 0x0053d2 to 0x0217b0 (0x1c3df bytes) 000265 1786063764.362004 trunc from 0x0217b1 to 0x027b12 (0x6362 bytes) 000269 1786063764.383453 mapread 0x00a245 thru 0x0155ba (0xb376 bytes) 000270 1786063764.414859 write 0x009613 thru 0x00da26 (0x4414 bytes) 000271 1786063764.421842 MIRROR_RESYNC 6 mirrors 000274 1786063766.608119 read 0x00007e thru 0x001f08 (0x1e8b bytes) 000276 1786063766.614086 mapread 0x013508 thru 0x017cc5 (0x47be bytes) 000278 1786063766.635734 write 0x02962d thru 0x0395c8 (0xff9c bytes) 000281 1786063766.690444 mapwrite 0x00b988 thru 0x01b173 (0xf7ec bytes) 000282 1786063766.747956 read 0x014b5d thru 0x02423c (0xf6e0 bytes) 000283 1786063766.761633 MIRROR_RESYNC 6 mirrors 000284 1786063769.078725 mapwrite 0x02ab9d thru 0x036206 (0xb66a bytes) 000285 1786063769.246068 mapread 0x01931a thru 0x027895 (0xe57c bytes) 000286 1786063769.269385 read 0x01e827 thru 0x028b6a (0xa344 bytes) 000289 1786063769.278504 mapwrite 0x00ac87 thru 0x018ef3 (0xe26d bytes) 000290 1786063769.380992 read 0x02ee23 thru 0x036c4b (0x7e29 bytes) 000291 1786063769.394207 MIRROR_RESYNC 6 mirrors 000292 1786063772.560647 read 0x005058 thru 0x00bc92 (0x6c3b bytes) 000293 1786063772.568433 mapwrite 0x00d5c2 thru 0x012736 (0x5175 bytes) 000294 1786063772.716351 mapread 0x02aac9 thru 0x03355b (0x8a93 bytes) 000295 1786063772.738512 mapwrite 0x036e26 thru 0x038dab (0x1f86 bytes) 000302 1786063772.807570 mapread 0x0078a2 thru 0x011502 (0x9c61 bytes) 000303 1786063772.814276 MIRROR_RESYNC 6 mirrors 000304 1786063775.185802 mapread 0x00d75a thru 0x011c33 (0x44da bytes) 000305 1786063775.209084 trunc from 0x0395c9 to 0x01388e (0x25d3a bytes) 000307 1786063775.298362 trunc from 0x01388f to 0x01466f (0xde1 bytes) 000308 1786063775.321511 write 0x03650a thru 0x03bb36 (0x562d bytes) 000309 1786063775.323318 write 0x015301 thru 0x0251ae (0xfeae bytes) 000310 1786063775.340719 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bb37): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb37): Mirror 4: 0xbed4d60a (stale) Mirror 5: 0xbed4d60a (stale) Mirror 6: 0xbed4d60a (stale) Mirror 9: 0xbed4d60a (stale) Mirror 10: 0xbed4d60a (stale) CRC-32 checksum value for chunk [0, 0x3bb37): Mirror 1: 0x10b6b178 Mirror 11: 0x10b6b178 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb37: skipped 000316 1786063777.120917 mapread 0x014b6b thru 0x022c90 (0xe126 bytes) 000318 1786063777.240582 mapwrite 0x036fbd thru 0x03ffff (0x9043 bytes) 000319 1786063777.630972 write 0x030f63 thru 0x034a72 (0x3b10 bytes) 000320 1786063777.632165 read 0x005d36 thru 0x00c732 (0x69fd bytes) 000323 1786063777.662549 trunc from 0x040000 to 0x0395cf (0x6a30 bytes) 000326 1786063777.732881 read 0x025232 thru 0x02d26c (0x803b bytes) 000328 1786063777.747114 write 0x01d686 thru 0x02a3d1 (0xcd4c bytes) 000330 1786063777.754855 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395d0): Mirror 5: 0x24bfa1e8 (stale) Mirror 6: 0x24bfa1e8 (stale) Mirror 9: 0x24bfa1e8 (stale) Mirror 10: 0x24bfa1e8 (stale) Mirror 11: 0x1b51f054 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395d0): Mirror 5: 0x24bfa1e8 (stale) Mirror 6: 0x24bfa1e8 (stale) Mirror 9: 0x24bfa1e8 (stale) Mirror 10: 0x24bfa1e8 (stale) Mirror 11: 0x1b51f054 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395d0: skipped 000331 1786063778.373240 mapwrite 0x02a3bd thru 0x034c4c (0xa890 bytes) 000332 1786063778.491267 mapread 0x008eb7 thru 0x00e8d8 (0x5a22 bytes) 000333 1786063778.504487 write 0x0096e2 thru 0x00da63 (0x4382 bytes) 000334 1786063778.508404 read 0x02a151 thru 0x02ec5b (0x4b0b bytes) 000336 1786063778.513137 read 0x00f28d thru 0x01bac9 (0xc83d bytes) 000337 1786063778.523835 mapwrite 0x021c3d thru 0x028841 (0x6c05 bytes) 000338 1786063778.567606 write 0x007659 thru 0x00a98a (0x3332 bytes) 000339 1786063778.570836 trunc from 0x0395d0 to 0x0240de (0x154f1 bytes) 000340 1786063778.603256 mapread 0x010785 thru 0x01ad01 (0xa57d bytes) 000342 1786063778.605038 read 0x00a890 thru 0x0150bc (0xa82d bytes) 000343 1786063778.615649 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x240df): Mirror 5: 0xb68b033d (stale) Mirror 9: 0xb68b033d (stale) Mirror 10: 0xb68b033d (stale) Mirror 11: 0xfb8e0ed5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x240df): Mirror 5: 0xb68b033d (stale) Mirror 9: 0xb68b033d (stale) Mirror 10: 0xb68b033d (stale) Mirror 11: 0xfb8e0ed5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x240df: skipped 000344 1786063779.130788 read 0x020d3f thru 0x0240de (0x33a0 bytes) 000345 1786063779.143987 trunc from 0x0240df to 0x037f01 (0x13e23 bytes) 000346 1786063779.168671 mapwrite 0x004b42 thru 0x010dcb (0xc28a bytes) 000347 1786063779.301549 read 0x0089ba thru 0x010a15 (0x805c bytes) 000349 1786063779.303021 write 0x03a644 thru 0x03f10a (0x4ac7 bytes) 000350 1786063779.306432 trunc from 0x03f10b to 0x023c84 (0x1b486 bytes) 000351 1786063779.325543 mapwrite 0x00ba0d thru 0x01432a (0x891e bytes) 000354 1786063779.405541 write 0x027188 thru 0x035f0f (0xed88 bytes) 000355 1786063779.408714 mapwrite 0x034c01 thru 0x036058 (0x1458 bytes) 000356 1786063779.505500 trunc from 0x036059 to 0x003ced (0x3236b bytes) 000358 1786063779.526152 trunc from 0x003cee to 0x023210 (0x1f523 bytes) 000361 1786063779.553642 read 0x01f0f6 thru 0x023210 (0x411b bytes) 000362 1786063779.579178 mapread 0x015e2e thru 0x01a9d8 (0x4bab bytes) 000364 1786063779.588694 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23211): Mirror 9: 0x30d9e7f1 (stale) Mirror 10: 0x30d9e7f1 (stale) Mirror 11: 0x94450d2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23211): Mirror 9: 0x30d9e7f1 (stale) Mirror 10: 0x30d9e7f1 (stale) Mirror 11: 0x94450d2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23211: skipped 000365 1786063780.033346 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23211): Mirror 9: 0x30d9e7f1 (stale) Mirror 10: 0x30d9e7f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23211): Mirror 9: 0x30d9e7f1 (stale) Mirror 10: 0x30d9e7f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23211: skipped 000366 1786063780.429337 trunc from 0x023211 to 0x02c3a5 (0x9195 bytes) 000367 1786063780.449283 mapread 0x024c8f thru 0x02ae36 (0x61a8 bytes) 000369 1786063780.480198 read 0x02a90d thru 0x02c3a5 (0x1a99 bytes) 000370 1786063780.488407 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c3a6): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3a6): Mirror 9: 0xed60376b (stale) Mirror 10: 0xed60376b (stale) CRC-32 checksum value for chunk [0, 0x2c3a6): Mirror 1: 0x59f260ef Mirror 11: 0x59f260ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3a6: skipped 000371 1786063781.736449 trunc from 0x02c3a6 to 0x010f15 (0x1b490 bytes) 000372 1786063781.852382 mapwrite 0x018112 thru 0x0190ad (0xf9c bytes) 000373 1786063781.972790 trunc from 0x0190ae to 0x02bd91 (0x12ce4 bytes) 000374 1786063782.001651 write 0x01909a thru 0x01e278 (0x51df bytes) 000383 1786063782.017438 write 0x0267cb thru 0x02fc9d (0x94d3 bytes) 000384 1786063782.024698 write 0x028adf thru 0x02ba43 (0x2f65 bytes) 000385 1786063782.032905 MIRROR_RESYNC 4 mirrors 000386 1786063783.448644 MIRROR_RESYNC 4 mirrors 000389 1786063783.869538 read 0x011f79 thru 0x017e1b (0x5ea3 bytes) 000391 1786063783.878230 trunc from 0x02fc9e to 0x031b0b (0x1e6e bytes) 000393 1786063784.012007 mapread 0x01ea71 thru 0x024c67 (0x61f7 bytes) 000394 1786063784.046512 mapwrite 0x036935 thru 0x03ffff (0x96cb bytes) 000396 1786063784.178973 read 0x03c4a9 thru 0x03ffff (0x3b57 bytes) 000399 1786063784.181635 read 0x00034b thru 0x00fe20 (0xfad6 bytes) 000400 1786063784.196118 read 0x015fcb thru 0x01fd7c (0x9db2 bytes) 000401 1786063784.200334 MIRROR_RESYNC 4 mirrors 000403 1786063786.125823 read 0x001c08 thru 0x00b47e (0x9877 bytes) 000404 1786063786.148838 read 0x0007e8 thru 0x007116 (0x692f bytes) 000405 1786063786.166318 write 0x02af50 thru 0x02f284 (0x4335 bytes) 000406 1786063786.297477 write 0x01ca94 thru 0x025424 (0x8991 bytes) 000407 1786063786.325568 mapwrite 0x033e59 thru 0x03905f (0x5207 bytes) 000409 1786063786.446846 read 0x020be8 thru 0x02f917 (0xed30 bytes) 000410 1786063786.461045 write 0x01d934 thru 0x028fe7 (0xb6b4 bytes) 000413 1786063786.463645 mapwrite 0x0280e0 thru 0x031c92 (0x9bb3 bytes) 000415 1786063786.534646 mapwrite 0x014104 thru 0x01aaf2 (0x69ef bytes) 000416 1786063786.588054 trunc from 0x040000 to 0x01e8c5 (0x2173a bytes) 000417 1786063786.614610 mapread 0x00ba7b thru 0x019e60 (0xe3e6 bytes) 000418 1786063786.618516 mapread 0x010040 thru 0x0187d1 (0x8792 bytes) 000422 1786063786.627371 mapwrite 0x010b45 thru 0x017ccf (0x718b bytes) 000432 1786063786.698201 mapread 0x013bc1 thru 0x0179e4 (0x3e24 bytes) 000434 1786063786.700739 read 0x014892 thru 0x01e8c5 (0xa034 bytes) 000437 1786063786.703794 trunc from 0x01e8c6 to 0x0207dd (0x1f18 bytes) 000440 1786063786.728756 mapwrite 0x01d97a thru 0x02c7e9 (0xee70 bytes) 000442 1786063786.923180 mapread 0x007559 thru 0x010624 (0x90cc bytes) 000443 1786063786.928278 mapread 0x016aad thru 0x01ae9c (0x43f0 bytes) 000446 1786063786.956033 MIRROR_RESYNC 4 mirrors 000447 1786063788.255913 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000448 1786063788.768381 mapread 0x0109e9 thru 0x0171d6 (0x67ee bytes) 000449 1786063788.804504 trunc from 0x02c7ea to 0x010ab8 (0x1bd31 bytes) 000450 1786063788.926027 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10ab9): Mirror 10: 0xc5acce8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10ab9): Mirror 10: 0xc5acce8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10ab9: skipped 000453 1786063789.257888 read 0x00946f thru 0x010ab8 (0x764a bytes) 000455 1786063789.270162 write 0x02ab0e thru 0x037f3c (0xd42f bytes) 000457 1786063789.275662 mapwrite 0x03d841 thru 0x03ffff (0x27bf bytes) 000459 1786063789.486068 mapwrite 0x01a78b thru 0x0233ef (0x8c65 bytes) 000461 1786063789.576367 read 0x00ce8f thru 0x011ac0 (0x4c32 bytes) 000462 1786063789.587181 mapwrite 0x009de0 thru 0x018ca3 (0xeec4 bytes) 000463 1786063789.645753 mapwrite 0x03b5cb thru 0x03ffff (0x4a35 bytes) 000464 1786063789.696483 MIRROR_RESYNC 2 mirrors 000466 1786063791.060964 MIRROR_RESYNC 2 mirrors 000467 1786063791.446577 MIRROR_RESYNC 2 mirrors 000468 1786063791.939874 mapwrite 0x01a0a9 thru 0x024cf4 (0xac4c bytes) 000469 1786063792.177517 read 0x00562d thru 0x005c53 (0x627 bytes) 000470 1786063792.184581 trunc from 0x040000 to 0x001fe6 (0x3e019 bytes) 000471 1786063792.227191 mapwrite 0x02a8bc thru 0x031757 (0x6e9c bytes) 000474 1786063792.327502 trunc from 0x031758 to 0x00f129 (0x2262e bytes) 000476 1786063792.348260 mapwrite 0x004750 thru 0x0121d2 (0xda83 bytes) 000477 1786063792.446084 mapread 0x010b17 thru 0x0121d2 (0x16bc bytes) 000478 1786063792.453500 MIRROR_RESYNC 2 mirrors 000479 1786063793.076728 write 0x015c20 thru 0x021937 (0xbd18 bytes) 000480 1786063793.144090 trunc from 0x021938 to 0x018bac (0x8d8b bytes) 000481 1786063793.196157 write 0x006223 thru 0x00cbdc (0x69ba bytes) 000482 1786063793.206206 write 0x00940b thru 0x0103ee (0x6fe4 bytes) 000484 1786063793.227914 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x18bad): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18bad): Mirror 10: 0x89852fe6 (stale) CRC-32 checksum value for chunk [0, 0x18bad): Mirror 1: 0x4a4fa42f Mirror 11: 0x4a4fa42f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18bad: skipped 000485 1786063794.322007 read 0x004bdb thru 0x00851f (0x3945 bytes) 000486 1786063794.353201 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000487 1786063794.769057 read 0x012607 thru 0x018bac (0x65a6 bytes) 000488 1786063794.799559 write 0x013a04 thru 0x01fc98 (0xc295 bytes) 000489 1786063794.938680 read 0x00f5a0 thru 0x018fce (0x9a2f bytes) 000490 1786063794.952796 trunc from 0x01fc99 to 0x00c811 (0x13487 bytes) 000491 1786063794.984760 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc812): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc812): Mirror 11: 0xaf5e9852 (stale) CRC-32 checksum value for chunk [0, 0xc812): Mirror 1: 0xaf5e9852 Mirror 12: 0xaf5e9852 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc812: skipped 000493 1786063796.048876 MIRROR_RESYNC 3 mirrors 000494 1786063796.843584 mapwrite 0x039224 thru 0x03ffff (0x6ddc bytes) 000498 1786063796.996176 mapread 0x01ac07 thru 0x02635a (0xb754 bytes) 000499 1786063797.035013 mapwrite 0x02962a thru 0x02b770 (0x2147 bytes) 000500 1786063797.126581 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1f6fc0d2 (stale) Mirror 12: 0x1f6fc0d2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73d170f8 Mirror 13: 0x73d170f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000502 1786063798.458318 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1f6fc0d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1f6fc0d2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73d170f8 Mirror 13: 0x73d170f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000503 1786063799.416479 write 0x00d085 thru 0x017678 (0xa5f4 bytes) 000504 1786063799.523821 read 0x02b1b8 thru 0x0361b6 (0xafff bytes) 000506 1786063799.536859 trunc from 0x040000 to 0x02461a (0x1b9e5 bytes) 000507 1786063799.618189 mapread 0x00f032 thru 0x0116b5 (0x2684 bytes) 000509 1786063799.624884 write 0x002d28 thru 0x00d955 (0xac2e bytes) 000510 1786063799.654276 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2461b): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2461b): Mirror 12: 0x26587a52 (stale) Mirror 13: 0x26587a52 (stale) CRC-32 checksum value for chunk [0, 0x2461b): Mirror 1: 0xd838e14d Mirror 14: 0xd838e14d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2461b: skipped 000511 1786063800.795410 trunc from 0x02461b to 0x02b559 (0x6f3f bytes) truncating to largest ever: 0x3ecc4 000512 1786063800.876504 trunc from 0x02b55a to 0x03ecc3 (0x1376a bytes) 000514 1786063800.908815 write 0x0369c2 thru 0x03ffff (0x963e bytes) 000515 1786063800.914417 trunc from 0x040000 to 0x03a207 (0x5df8 bytes) 000517 1786063800.934653 write 0x006be2 thru 0x0119df (0xadfe bytes) 000522 1786063800.952263 read 0x01d36f thru 0x02c57d (0xf20f bytes) 000524 1786063800.965384 write 0x008e23 thru 0x00c1ba (0x3398 bytes) 000525 1786063800.967714 mapwrite 0x00cbf9 thru 0x018a53 (0xbe5b bytes) 000526 1786063801.082041 mapwrite 0x0072d9 thru 0x013342 (0xc06a bytes) 000529 1786063801.140076 mapread 0x0376c8 thru 0x038d51 (0x168a bytes) 000533 1786063801.145044 trunc from 0x03a208 to 0x014f42 (0x252c5 bytes) 000534 1786063801.165043 trunc from 0x014f43 to 0x0247b0 (0xf86e bytes) 000536 1786063801.185621 mapread 0x00ae36 thru 0x012d73 (0x7f3e bytes) 000538 1786063801.188860 mapread 0x018eca thru 0x01ec6f (0x5da6 bytes) 000539 1786063801.199896 trunc from 0x0247b1 to 0x01e92c (0x5e84 bytes) 000540 1786063801.213110 write 0x03ba75 thru 0x03f127 (0x36b3 bytes) 000541 1786063801.214639 trunc from 0x03f128 to 0x038882 (0x68a5 bytes) 000544 1786063801.235093 mapread 0x009ba5 thru 0x017558 (0xd9b4 bytes) 000547 1786063801.238078 write 0x00a252 thru 0x00d1eb (0x2f9a bytes) 000550 1786063801.242062 write 0x034590 thru 0x03ffff (0xba70 bytes) 000553 1786063801.253389 write 0x0391cf thru 0x03ffff (0x6e31 bytes) 000555 1786063801.256247 mapwrite 0x019c63 thru 0x0258ce (0xbc6c bytes) 000559 1786063801.320970 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1f6fc0d2 (stale) Mirror 13: 0x73d170f8 (stale) Mirror 14: 0xb2c3f587 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b43cda5 Mirror 15: 0x4b43cda5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000560 1786063802.536493 mapread 0x027d74 thru 0x036bcb (0xee58 bytes) 000561 1786063802.601895 trunc from 0x040000 to 0x000ebc (0x3f143 bytes) 000562 1786063802.854528 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xebd): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebd): Mirror 12: 0xe47788cf (stale) Mirror 13: 0xe47788cf (stale) Mirror 14: 0xe47788cf (stale) Mirror 15: 0xe47788cf (stale) CRC-32 checksum value for chunk [0, 0xebd): Mirror 1: 0xe47788cf Mirror 16: 0xe47788cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebd: skipped 000563 1786063804.117689 trunc from 0x000ebd to 0x002b1e (0x1c62 bytes) 000564 1786063804.195947 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b1f): Mirror 12: 0x75f84d41 (stale) Mirror 13: 0x75f84d41 (stale) Mirror 14: 0x75f84d41 (stale) Mirror 16: 0x75f84d41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1f): Mirror 12: 0x75f84d41 (stale) Mirror 13: 0x75f84d41 (stale) Mirror 14: 0x75f84d41 (stale) Mirror 16: 0x75f84d41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1f: skipped 000567 1786063804.520877 read 0x0010db thru 0x002b1e (0x1a44 bytes) 000569 1786063804.531044 mapwrite 0x0185b1 thru 0x01d31a (0x4d6a bytes) 000570 1786063804.620436 write 0x01d5a3 thru 0x02c8f4 (0xf352 bytes) 000574 1786063804.626104 write 0x012568 thru 0x01f946 (0xd3df bytes) 000575 1786063804.639110 MIRROR_RESYNC 5 mirrors 000576 1786063805.577421 mapread 0x00d989 thru 0x01ac6d (0xd2e5 bytes) 000577 1786063805.629580 mapread 0x007b18 thru 0x00b99b (0x3e84 bytes) 000578 1786063805.648085 MIRROR_RESYNC 5 mirrors 000580 1786063806.173585 mapread 0x00fc84 thru 0x01e67a (0xe9f7 bytes) 000583 1786063806.178832 trunc from 0x02c8f5 to 0x03587e (0x8f8a bytes) 000584 1786063806.244757 write 0x0151bc thru 0x022ce7 (0xdb2c bytes) 000585 1786063806.255864 read 0x01acd9 thru 0x024bf7 (0x9f1f bytes) 000586 1786063806.261339 write 0x03920c thru 0x03d7af (0x45a4 bytes) 000591 1786063806.265007 mapwrite 0x01bf99 thru 0x01c3aa (0x412 bytes) 000594 1786063806.351265 write 0x0174f9 thru 0x01f6da (0x81e2 bytes) 000597 1786063806.354534 read 0x0290ca thru 0x029f6a (0xea1 bytes) 000598 1786063806.367956 mapwrite 0x009bea thru 0x015898 (0xbcaf bytes) 000602 1786063806.460686 trunc from 0x03d7b0 to 0x03e803 (0x1054 bytes) 000604 1786063806.492385 mapwrite 0x02c802 thru 0x02c865 (0x64 bytes) 000606 1786063806.537738 mapread 0x00989d thru 0x00a8d9 (0x103d bytes) 000607 1786063806.544828 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e804): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e804): Mirror 12: 0xd57f1c1c (stale) Mirror 13: 0xd57f1c1c (stale) Mirror 14: 0xd57f1c1c (stale) Mirror 16: 0xd57f1c1c (stale) CRC-32 checksum value for chunk [0, 0x3e804): Mirror 1: 0x39361d84 Mirror 17: 0x39361d84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e804: skipped 000608 1786063807.758767 read 0x022d1f thru 0x024ce4 (0x1fc6 bytes) 000609 1786063807.772730 read 0x03c337 thru 0x03e803 (0x24cd bytes) 000611 1786063807.802994 read 0x002be0 thru 0x00501c (0x243d bytes) 000615 1786063807.845926 read 0x017421 thru 0x01cd24 (0x5904 bytes) 000617 1786063807.860315 trunc from 0x03e804 to 0x010dcd (0x2da36 bytes) 000623 1786063807.966802 mapwrite 0x02bf4e thru 0x0369e0 (0xaa93 bytes) truncating to largest ever: 0x3f020 000624 1786063808.073958 trunc from 0x0369e1 to 0x03f01f (0x863f bytes) 000625 1786063808.099812 mapread 0x028730 thru 0x033be5 (0xb4b6 bytes) 000626 1786063808.101088 mapwrite 0x022fd2 thru 0x02990e (0x693d bytes) 000628 1786063808.187841 mapread 0x0259ba thru 0x026d8c (0x13d3 bytes) 000632 1786063808.190949 trunc from 0x03f020 to 0x00a802 (0x3481d bytes) 000633 1786063808.209000 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xa803): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa803): Mirror 12: 0x140593f5 (stale) Mirror 13: 0x140593f5 (stale) Mirror 14: 0x140593f5 (stale) Mirror 16: 0x140593f5 (stale) Mirror 17: 0x9d9e1e0c (stale) CRC-32 checksum value for chunk [0, 0xa803): Mirror 1: 0x9d9e1e0c Mirror 18: 0x9d9e1e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa803: skipped 000635 1786063809.000560 mapread 0x00874e thru 0x00a802 (0x20b5 bytes) 000636 1786063809.050109 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xa803): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa803): Mirror 12: 0x140593f5 (stale) Mirror 13: 0x140593f5 (stale) Mirror 14: 0x140593f5 (stale) Mirror 16: 0x140593f5 (stale) Mirror 17: 0x9d9e1e0c (stale) CRC-32 checksum value for chunk [0, 0xa803): Mirror 1: 0x9d9e1e0c Mirror 18: 0x9d9e1e0c Mirror 19: 0x9d9e1e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa803: skipped 000637 1786063809.878739 mapwrite 0x0139b4 thru 0x01dc20 (0xa26d bytes) 000639 1786063810.026485 write 0x01d8f3 thru 0x023c1f (0x632d bytes) 000640 1786063810.027875 mapwrite 0x01bc79 thru 0x025b8c (0x9f14 bytes) 000641 1786063810.108403 write 0x03edf1 thru 0x03ffff (0x120f bytes) 000642 1786063810.114660 read 0x003fd2 thru 0x00e62f (0xa65e bytes) 000643 1786063810.147592 read 0x00a864 thru 0x00a91f (0xbc bytes) 000645 1786063810.151537 read 0x0056e8 thru 0x014f85 (0xf89e bytes) 000647 1786063810.165242 read 0x025092 thru 0x025918 (0x887 bytes) 000650 1786063810.171585 trunc from 0x040000 to 0x03cadd (0x3522 bytes) 000652 1786063810.224247 read 0x0382a7 thru 0x03bd21 (0x3a7b bytes) 000654 1786063810.241199 write 0x0349c3 thru 0x037b16 (0x3154 bytes) 000655 1786063810.259261 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cade): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cade): Mirror 12: 0xc2f8fca0 (stale) Mirror 13: 0xc2f8fca0 (stale) Mirror 14: 0xc2f8fca0 (stale) Mirror 16: 0xc2f8fca0 (stale) Mirror 17: 0xfcdd33f4 (stale) Mirror 18: 0xbd52d722 (stale) Mirror 19: 0xbd52d722 (stale) CRC-32 checksum value for chunk [0, 0x3cade): Mirror 1: 0x7a055bee Mirror 20: 0x7a055bee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cade: skipped 000656 1786063811.834876 read 0x00dc5c thru 0x0178f8 (0x9c9d bytes) 000657 1786063811.866624 MIRROR_RESYNC 9 mirrors 000658 1786063815.936919 mapread 0x039b66 thru 0x03cadd (0x2f78 bytes) 000659 1786063815.966641 write 0x02504b thru 0x02eb78 (0x9b2e bytes) 000661 1786063816.075810 mapread 0x0034ed thru 0x00eb48 (0xb65c bytes) 000662 1786063816.089550 mapread 0x02fb57 thru 0x03b69d (0xbb47 bytes) 000665 1786063816.119611 MIRROR_RESYNC 9 mirrors 000669 1786063819.598971 mapread 0x0195a4 thru 0x02177d (0x81da bytes) 000670 1786063819.626376 write 0x000c69 thru 0x006a1c (0x5db4 bytes) 000672 1786063819.701745 MIRROR_RESYNC 9 mirrors 000673 1786063823.495528 trunc from 0x03cade to 0x005fa8 (0x36b35 bytes) 000675 1786063823.554003 mapwrite 0x02035c thru 0x020e1d (0xac2 bytes) 000676 1786063823.635374 mapwrite 0x01ff28 thru 0x0241bb (0x4294 bytes) 000677 1786063823.722080 write 0x02ca6e thru 0x032ba6 (0x6139 bytes) 000678 1786063823.724192 mapread 0x005b5b thru 0x014ebf (0xf365 bytes) truncating to largest ever: 0x3f57c 000679 1786063823.739951 trunc from 0x032ba7 to 0x03f57b (0xc9d5 bytes) 000680 1786063823.750358 write 0x021d86 thru 0x026b22 (0x4d9d bytes) 000682 1786063823.759650 read 0x004d98 thru 0x0090bd (0x4326 bytes) 000683 1786063823.760475 write 0x01ed92 thru 0x01f9e6 (0xc55 bytes) 000684 1786063823.761653 mapread 0x01e395 thru 0x027c36 (0x98a2 bytes) 000685 1786063823.768203 trunc from 0x03f57c to 0x03a26d (0x530e bytes) 000687 1786063823.781537 MIRROR_RESYNC 9 mirrors 000688 1786063826.881819 trunc from 0x03a26e to 0x0219c4 (0x188a9 bytes) 000689 1786063826.951064 write 0x002b40 thru 0x012453 (0xf914 bytes) 000691 1786063826.963141 trunc from 0x0219c5 to 0x0123f0 (0xf5d4 bytes) 000693 1786063826.987411 mapread 0x002e58 thru 0x00ea03 (0xbbac bytes) 000694 1786063826.988930 mapread 0x0107b7 thru 0x0123f0 (0x1c3a bytes) 000696 1786063826.994555 mapread 0x00a0c6 thru 0x00d6f3 (0x362e bytes) 000697 1786063826.995906 read 0x00ed88 thru 0x0123f0 (0x3669 bytes) 000698 1786063826.997127 trunc from 0x0123f1 to 0x00bcb8 (0x6738 bytes) 000700 1786063827.018642 write 0x019ee3 thru 0x01a07f (0x19d bytes) 000701 1786063827.020956 write 0x0204fb thru 0x028d4d (0x8853 bytes) 000704 1786063827.024318 mapread 0x0005e6 thru 0x003bb4 (0x35cf bytes) 000706 1786063827.044635 MIRROR_RESYNC 9 mirrors 000708 1786063829.866105 read 0x0026a3 thru 0x011338 (0xec96 bytes) 000709 1786063829.883279 MIRROR_RESYNC 9 mirrors 000711 1786063830.544914 trunc from 0x028d4e to 0x00ab09 (0x1e244 bytes) 000713 1786063830.621597 trunc from 0x00ab0a to 0x02ca2a (0x21f21 bytes) 000714 1786063830.642734 read 0x00332f thru 0x0110fa (0xddcc bytes) 000716 1786063830.658926 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ca2b): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca2b): Mirror 12: 0xabf83674 (stale) Mirror 13: 0xabf83674 (stale) Mirror 14: 0xabf83674 (stale) Mirror 16: 0xabf83674 (stale) Mirror 17: 0xabf83674 (stale) Mirror 18: 0xabf83674 (stale) Mirror 19: 0xabf83674 (stale) Mirror 20: 0xabf83674 (stale) CRC-32 checksum value for chunk [0, 0x2ca2b): Mirror 1: 0x891c8d00 Mirror 21: 0x891c8d00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca2b: skipped 000720 1786063832.046285 MIRROR_RESYNC 10 mirrors 000721 1786063834.054578 mapwrite 0x003679 thru 0x012a80 (0xf408 bytes) 000723 1786063834.282012 write 0x023529 thru 0x0272f7 (0x3dcf bytes) 000724 1786063834.299594 write 0x023d1d thru 0x0332dd (0xf5c1 bytes) 000725 1786063834.302823 read 0x0073c8 thru 0x010b6a (0x97a3 bytes) 000728 1786063834.305917 read 0x02c058 thru 0x0332dd (0x7286 bytes) 000729 1786063834.307240 mapread 0x009756 thru 0x009dcf (0x67a bytes) 000730 1786063834.309179 write 0x020c97 thru 0x026ecd (0x6237 bytes) 000731 1786063834.324621 MIRROR_RESYNC 10 mirrors 000732 1786063837.410919 read 0x02e1d0 thru 0x032530 (0x4361 bytes) 000733 1786063837.430130 trunc from 0x0332de to 0x037a2e (0x4751 bytes) 000734 1786063837.571922 mapread 0x02f765 thru 0x03538b (0x5c27 bytes) 000741 1786063837.609304 mapwrite 0x00bc61 thru 0x017fc2 (0xc362 bytes) 000742 1786063837.832305 read 0x0118f7 thru 0x01c550 (0xac5a bytes) 000743 1786063837.833298 write 0x0319d8 thru 0x032a68 (0x1091 bytes) 000744 1786063837.834888 write 0x0360c6 thru 0x03c713 (0x664e bytes) 000748 1786063837.838594 trunc from 0x03c714 to 0x000689 (0x3c08a bytes) 000749 1786063837.885552 mapread 0x000329 thru 0x000689 (0x361 bytes) 000750 1786063837.898365 MIRROR_RESYNC 10 mirrors 000751 1786063839.593014 MIRROR_RESYNC 10 mirrors 000752 1786063839.933887 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000753 1786063840.469283 write 0x03aea7 thru 0x03ffff (0x5159 bytes) 000754 1786063840.594152 write 0x03e8b9 thru 0x03ffff (0x1747 bytes) 000755 1786063840.595276 read 0x029962 thru 0x031c91 (0x8330 bytes) 000757 1786063840.604193 trunc from 0x040000 to 0x0114c9 (0x2eb36 bytes) 000761 1786063840.653824 mapwrite 0x02e41e thru 0x036ad6 (0x86b9 bytes) 000763 1786063840.764154 mapread 0x01581d thru 0x016db6 (0x159a bytes) 000764 1786063840.772757 mapread 0x0090d1 thru 0x01851e (0xf44e bytes) 000766 1786063840.784423 mapread 0x017f8d thru 0x01b2ed (0x3361 bytes) 000767 1786063840.790572 mapwrite 0x02b93f thru 0x031c62 (0x6324 bytes) 000768 1786063840.874810 mapread 0x00051a thru 0x00db33 (0xd61a bytes) 000769 1786063840.895674 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36ad7): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ad7): Mirror 12: 0x88986484 (stale) Mirror 13: 0x88986484 (stale) Mirror 14: 0x88986484 (stale) Mirror 16: 0x88986484 (stale) Mirror 17: 0x88986484 (stale) Mirror 18: 0x88986484 (stale) Mirror 19: 0x88986484 (stale) Mirror 21: 0x88986484 (stale) CRC-32 checksum value for chunk [0, 0x36ad7): Mirror 1: 0x74a6f933 Mirror 22: 0x74a6f933 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ad7: skipped 000770 1786063842.322887 mapread 0x0194e4 thru 0x01f72c (0x6249 bytes) 000772 1786063842.379975 read 0x00efb3 thru 0x010bdd (0x1c2b bytes) 000773 1786063842.404062 read 0x02ae0c thru 0x036ad6 (0xbccb bytes) 000777 1786063842.476076 write 0x02c302 thru 0x02fa0a (0x3709 bytes) 000778 1786063842.623325 read 0x0051bd thru 0x014acf (0xf913 bytes) 000779 1786063842.644449 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ad7): Mirror 12: 0x88986484 (stale) Mirror 13: 0x88986484 (stale) Mirror 14: 0x88986484 (stale) Mirror 16: 0x88986484 (stale) Mirror 18: 0x88986484 (stale) Mirror 19: 0x88986484 (stale) Mirror 21: 0x88986484 (stale) Mirror 22: 0x74a6f933 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ad7): Mirror 12: 0x88986484 (stale) Mirror 13: 0x88986484 (stale) Mirror 14: 0x88986484 (stale) Mirror 16: 0x88986484 (stale) Mirror 18: 0x88986484 (stale) Mirror 19: 0x88986484 (stale) Mirror 21: 0x88986484 (stale) Mirror 22: 0x74a6f933 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ad7: skipped 000780 1786063843.185107 trunc from 0x036ad7 to 0x00a86a (0x2c26c bytes) 000781 1786063843.217584 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa86b): Mirror 12: 0xeb69c993 (stale) Mirror 13: 0xeb69c993 (stale) Mirror 14: 0xeb69c993 (stale) Mirror 16: 0xeb69c993 (stale) Mirror 18: 0xeb69c993 (stale) Mirror 19: 0xeb69c993 (stale) Mirror 21: 0xeb69c993 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa86b): Mirror 12: 0xeb69c993 (stale) Mirror 13: 0xeb69c993 (stale) Mirror 14: 0xeb69c993 (stale) Mirror 16: 0xeb69c993 (stale) Mirror 18: 0xeb69c993 (stale) Mirror 19: 0xeb69c993 (stale) Mirror 21: 0xeb69c993 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa86b: skipped 000782 1786063843.921360 mapwrite 0x026cc3 thru 0x02bf2d (0x526b bytes) 000783 1786063844.043298 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf2e): Mirror 12: 0x69f7c78b (stale) Mirror 13: 0x69f7c78b (stale) Mirror 14: 0x69f7c78b (stale) Mirror 16: 0x69f7c78b (stale) Mirror 19: 0x69f7c78b (stale) Mirror 21: 0x69f7c78b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf2e): Mirror 12: 0x69f7c78b (stale) Mirror 13: 0x69f7c78b (stale) Mirror 14: 0x69f7c78b (stale) Mirror 16: 0x69f7c78b (stale) Mirror 19: 0x69f7c78b (stale) Mirror 21: 0x69f7c78b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf2e: skipped 000784 1786063844.476059 write 0x006471 thru 0x00b0f3 (0x4c83 bytes) 000785 1786063844.487948 write 0x00030c thru 0x0075f9 (0x72ee bytes) 000792 1786063844.500619 mapread 0x00fc8d thru 0x0150c4 (0x5438 bytes) 000793 1786063844.521324 read 0x025c3d thru 0x02bf2d (0x62f1 bytes) 000796 1786063844.532000 mapwrite 0x00f21b thru 0x00f628 (0x40e bytes) 000798 1786063844.581042 trunc from 0x02bf2e to 0x03ef3e (0x13011 bytes) 000801 1786063844.606694 MIRROR_RESYNC 7 mirrors 000802 1786063847.540345 mapread 0x031924 thru 0x0375a0 (0x5c7d bytes) 000803 1786063847.567158 write 0x031f97 thru 0x039248 (0x72b2 bytes) 000804 1786063847.659622 mapwrite 0x02f332 thru 0x033c86 (0x4955 bytes) 000806 1786063847.760990 mapread 0x0304f0 thru 0x036217 (0x5d28 bytes) 000807 1786063847.780510 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef3f): Mirror 13: 0x3d94bc9a (stale) Mirror 14: 0x3d94bc9a (stale) Mirror 16: 0x3d94bc9a (stale) Mirror 19: 0x3d94bc9a (stale) Mirror 21: 0x3d94bc9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef3f): Mirror 13: 0x3d94bc9a (stale) Mirror 14: 0x3d94bc9a (stale) Mirror 16: 0x3d94bc9a (stale) Mirror 19: 0x3d94bc9a (stale) Mirror 21: 0x3d94bc9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef3f: skipped 000808 1786063848.296838 mapread 0x020fb1 thru 0x02c2ae (0xb2fe bytes) 000809 1786063848.326928 trunc from 0x03ef3f to 0x025974 (0x195ca bytes) 000812 1786063848.348915 mapread 0x00df71 thru 0x010245 (0x22d5 bytes) 000813 1786063848.357715 trunc from 0x025975 to 0x032723 (0xcdaf bytes) 000814 1786063848.374644 read 0x02a7e6 thru 0x032723 (0x7f3e bytes) 000818 1786063848.404590 trunc from 0x032724 to 0x010137 (0x225ec bytes) 000819 1786063848.425925 mapwrite 0x03db3f thru 0x03ffff (0x24c1 bytes) 000821 1786063848.536856 trunc from 0x040000 to 0x02810d (0x17ef2 bytes) 000822 1786063848.560820 write 0x00dada thru 0x015f49 (0x8470 bytes) 000823 1786063848.570899 mapwrite 0x015971 thru 0x018d35 (0x33c5 bytes) 000824 1786063848.641007 mapread 0x006940 thru 0x009609 (0x2cca bytes) 000825 1786063848.642467 mapwrite 0x0345a3 thru 0x03ffff (0xba5d bytes) 000826 1786063848.745721 MIRROR_RESYNC 6 mirrors 000827 1786063850.740702 mapwrite 0x0256b1 thru 0x02962d (0x3f7d bytes) 000829 1786063850.909884 mapread 0x031e19 thru 0x035fb5 (0x419d bytes) 000832 1786063850.919465 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe3673bab (stale) Mirror 14: 0xe3673bab (stale) Mirror 16: 0xe3673bab (stale) Mirror 19: 0xe3673bab (stale) Mirror 21: 0xe3673bab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x573bfd4f Mirror 22: 0x573bfd4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000833 1786063852.642141 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe3673bab (stale) Mirror 14: 0xe3673bab (stale) Mirror 19: 0xe3673bab (stale) Mirror 21: 0xe3673bab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe3673bab (stale) Mirror 14: 0xe3673bab (stale) Mirror 19: 0xe3673bab (stale) Mirror 21: 0xe3673bab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x573bfd4f Mirror 22: 0x573bfd4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000836 1786063853.934377 read 0x00df8d thru 0x0144b2 (0x6526 bytes) 000837 1786063853.965007 mapread 0x0046af thru 0x00515e (0xab0 bytes) 000838 1786063854.008955 mapwrite 0x03f2c4 thru 0x03ffff (0xd3c bytes) 000839 1786063854.284099 mapread 0x014a23 thru 0x01a56f (0x5b4d bytes) 000841 1786063854.306190 read 0x00fd18 thru 0x014058 (0x4341 bytes) 000843 1786063854.313155 mapread 0x02887e thru 0x03761a (0xed9d bytes) 000846 1786063854.337079 read 0x007552 thru 0x014c98 (0xd747 bytes) 000847 1786063854.338268 mapread 0x03557b thru 0x0395f3 (0x4079 bytes) 000848 1786063854.342107 mapread 0x00d9cb thru 0x015411 (0x7a47 bytes) 000851 1786063854.361099 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe3673bab (stale) Mirror 19: 0xe3673bab (stale) Mirror 21: 0xe3673bab (stale) Mirror 22: 0x573bfd4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe3673bab (stale) Mirror 19: 0xe3673bab (stale) Mirror 21: 0xe3673bab (stale) Mirror 22: 0x573bfd4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000853 1786063854.973669 read 0x0353c4 thru 0x037f33 (0x2b70 bytes) 000854 1786063854.980535 mapwrite 0x024215 thru 0x02dfa4 (0x9d90 bytes) truncating to largest ever: 0x3fc25 000855 1786063855.103851 trunc from 0x040000 to 0x03fc24 (0x3db bytes) 000861 1786063855.145607 mapread 0x037987 thru 0x03f586 (0x7c00 bytes) 000862 1786063855.169254 read 0x01f25d thru 0x02a293 (0xb037 bytes) 000868 1786063855.184963 read 0x010de4 thru 0x0209d1 (0xfbee bytes) 000869 1786063855.213598 mapwrite 0x02e97b thru 0x032e2c (0x44b2 bytes) 000870 1786063855.314806 trunc from 0x03fc25 to 0x01832d (0x278f7 bytes) 000871 1786063855.345139 mapwrite 0x00719f thru 0x00f6d1 (0x8533 bytes) 000874 1786063855.427003 MIRROR_RESYNC 5 mirrors 000877 1786063856.657462 trunc from 0x01832e to 0x023a69 (0xb73c bytes) 000878 1786063856.742435 mapwrite 0x012445 thru 0x022038 (0xfbf4 bytes) 000879 1786063856.833427 read 0x01cc89 thru 0x023a69 (0x6de1 bytes) 000880 1786063856.839469 MIRROR_RESYNC 5 mirrors 000881 1786063858.115142 read 0x01becc thru 0x023a69 (0x7b9e bytes) 000882 1786063858.123436 mapread 0x00240d thru 0x006d76 (0x496a bytes) 000884 1786063858.163587 write 0x033c36 thru 0x03ffff (0xc3ca bytes) 000885 1786063858.252842 read 0x00dbcc thru 0x016582 (0x89b7 bytes) 000888 1786063858.274109 write 0x01c3fe thru 0x029346 (0xcf49 bytes) 000893 1786063858.280804 mapwrite 0x006f38 thru 0x00e0b2 (0x717b bytes) 000894 1786063858.385032 write 0x025800 thru 0x033d59 (0xe55a bytes) 000897 1786063858.388133 read 0x03fdff thru 0x03ffff (0x201 bytes) 000898 1786063858.389457 mapread 0x006496 thru 0x0076a6 (0x1211 bytes) 000899 1786063858.391107 trunc from 0x040000 to 0x018090 (0x27f6f bytes) 000905 1786063858.424342 mapread 0x00dce4 thru 0x00faba (0x1dd7 bytes) 000907 1786063858.428911 mapread 0x012c72 thru 0x018090 (0x541f bytes) 000909 1786063858.439173 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18091): Mirror 14: 0x51b8c1cf (stale) Mirror 21: 0x51b8c1cf (stale) Mirror 22: 0x51b8c1cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18091): Mirror 14: 0x51b8c1cf (stale) Mirror 21: 0x51b8c1cf (stale) Mirror 22: 0x51b8c1cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18091: skipped 000910 1786063858.956993 MIRROR_RESYNC 4 mirrors 000911 1786063859.847466 mapwrite 0x01efb3 thru 0x0263b8 (0x7406 bytes) 000913 1786063859.997707 mapread 0x0071b8 thru 0x016cf3 (0xfb3c bytes) 000915 1786063860.022765 mapread 0x00adfc thru 0x00ca9e (0x1ca3 bytes) 000916 1786063860.024422 trunc from 0x0263b9 to 0x006f8b (0x1f42d bytes) 000917 1786063860.048693 mapwrite 0x00ba39 thru 0x00c636 (0xbfe bytes) 000918 1786063860.144681 trunc from 0x00c637 to 0x00b42e (0x1208 bytes) 000919 1786063860.168261 write 0x01e2c0 thru 0x027eff (0x9c40 bytes) 000920 1786063860.170548 read 0x01016d thru 0x012680 (0x2514 bytes) 000921 1786063860.178612 mapwrite 0x018b70 thru 0x019326 (0x7b7 bytes) 000922 1786063860.231884 mapread 0x01011e thru 0x010b27 (0xa0a bytes) 000923 1786063860.233179 mapwrite 0x03df6b thru 0x03ffff (0x2095 bytes) 000924 1786063860.342401 read 0x030edf thru 0x035a1e (0x4b40 bytes) 000925 1786063860.358373 trunc from 0x040000 to 0x013161 (0x2ce9e bytes) 000926 1786063860.379879 read 0x0092c6 thru 0x012529 (0x9264 bytes) 000927 1786063860.386129 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13162): Mirror 21: 0x60ff17c2 (stale) Mirror 22: 0x60ff17c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13162): Mirror 21: 0x60ff17c2 (stale) Mirror 22: 0x60ff17c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13162: skipped 000928 1786063860.771954 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13162): Mirror 22: 0x60ff17c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13162): Mirror 22: 0x60ff17c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13162: skipped 000929 1786063861.187972 write 0x010fd9 thru 0x0140d1 (0x30f9 bytes) 000930 1786063861.196049 trunc from 0x0140d2 to 0x034ce0 (0x20c0f bytes) 000931 1786063861.233829 write 0x028585 thru 0x034a00 (0xc47c bytes) 000932 1786063861.259629 write 0x00ee77 thru 0x0138a6 (0x4a30 bytes) 000934 1786063861.271470 read 0x02bd2c thru 0x034ce0 (0x8fb5 bytes) 000935 1786063861.272455 mapwrite 0x02afcd thru 0x02b2dd (0x311 bytes) 000936 1786063861.406559 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000940 1786063861.575743 mapwrite 0x00d67f thru 0x01be65 (0xe7e7 bytes) 000941 1786063861.855696 mapwrite 0x02e398 thru 0x03912d (0xad96 bytes) 000942 1786063862.119996 trunc from 0x03912e to 0x0100c9 (0x29064 bytes) 000943 1786063862.181531 write 0x01a069 thru 0x02592d (0xb8c5 bytes) 000944 1786063862.185989 read 0x009c32 thru 0x00a715 (0xae4 bytes) 000947 1786063862.188595 mapread 0x005fae thru 0x006ac3 (0xb16 bytes) 000948 1786063862.208284 write 0x028e39 thru 0x02b831 (0x29f9 bytes) 000950 1786063862.210295 mapwrite 0x00c1d6 thru 0x0163d1 (0xa1fc bytes) 000954 1786063862.360129 mapwrite 0x00b3e2 thru 0x00d57c (0x219b bytes) 000956 1786063862.471703 trunc from 0x02b832 to 0x00e218 (0x1d619 bytes) 000958 1786063862.504529 mapread 0x0078eb thru 0x00ab78 (0x328e bytes) 000960 1786063862.512774 mapread 0x00c449 thru 0x00e218 (0x1dd0 bytes) 000961 1786063862.516510 trunc from 0x00e219 to 0x003c3d (0xa5db bytes) 000962 1786063862.544682 mapread 0x001281 thru 0x003c3d (0x29bd bytes) 000967 1786063862.553005 mapwrite 0x00a5e3 thru 0x018205 (0xdc23 bytes) 000970 1786063862.703627 write 0x039a61 thru 0x03e119 (0x46b9 bytes) 000972 1786063862.706820 mapread 0x01a7f6 thru 0x022f9f (0x87aa bytes) 000973 1786063862.719483 mapread 0x01af53 thru 0x02976e (0xe81c bytes) 000976 1786063862.730986 mapwrite 0x02c59b thru 0x039de1 (0xd847 bytes) 000977 1786063862.846373 trunc from 0x03e11a to 0x0002e3 (0x3de36 bytes) 000978 1786063862.865385 trunc from 0x0002e4 to 0x02d7de (0x2d4fb bytes) 000981 1786063862.886357 trunc from 0x02d7df to 0x01c0c2 (0x1171c bytes) 000982 1786063862.903585 trunc from 0x01c0c3 to 0x027864 (0xb7a2 bytes) 000984 1786063862.923518 trunc from 0x027865 to 0x00d7b3 (0x1a0b1 bytes) 000986 1786063862.941060 mapwrite 0x03a704 thru 0x03b73f (0x103c bytes) 000988 1786063863.069517 MIRROR_EXTEND to 2 mirrors 000991 1786063864.114417 mapwrite 0x033eb5 thru 0x03ffff (0xc14b bytes) 000992 1786063864.493664 mapwrite 0x02bcca thru 0x0381f8 (0xc52f bytes) 000993 1786063864.612974 trunc from 0x040000 to 0x01cf64 (0x2309b bytes) 000995 1786063864.652469 MIRROR_RESYNC 2 mirrors 000996 1786063865.498175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c9:0x0]: Invalid argument (22) 000997 1786063865.654924 trunc from 0x01cf65 to 0x01bfd0 (0xf94 bytes) 000999 1786063865.702799 write 0x03b4bb thru 0x03ffff (0x4b45 bytes) All operations completed A-OK! PASS 70a (159s) == sanity-flr test 71: check mirror extend parent fid ==== 20:51:10 (1786063870) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 20:51:13 (1786063873) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.368232 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258519 s, 4.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 20:51:23 (1786063883) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8398560 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7174560 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7583968 .. Extending file size to 8421856 .. Extending file size to 7839520 .. Extending file size to 7964352 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8552256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8725952 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9194528 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8756288 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57476 57477 57480 57481 57484 done 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) 192.168.201.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:52:51 (1786063971) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1403424 .. Extending file size to 8594432 .. Extending file size to 5198016 .. Extending file size to 6511008 .. Extending file size to 8639328 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7272672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7580224 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9108992 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9392832 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59231 59232 59234 59235 59237 failed 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) 192.168.201.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:54:13 (1786064053) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162639 s, 12.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190793 s, 11.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5c095c2d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:54:34 (1786064074) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115104 s, 18.2 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 20:54:44 (1786064084) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:54:54 (1786064094) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232905 s, 9.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:55:07 (1786064107) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189251 s, 11.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:55:18 (1786064118) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.471609 s, 4.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:55:53 (1786064153) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180558 s, 5.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:56:59 (1786064219) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283708 s, 7.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:58:00 (1786064280) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. Creating new pool test_204f oleg120-server: Pool lustre.test_204f created Adding targets to pool oleg120-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg120-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152905 s, 13.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg120-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg120-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg120-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:59:22 (1786064362) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:59:30 (1786064370) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:59:40 (1786064380) oleg120-server: Pool lustre.test_206 created Adding targets to pool oleg120-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg120-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg120-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg120-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg120-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:00:07 (1786064407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114029 s, 9.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515714 s, 20.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7a16d3e1 Mirror 3: 0x7a16d3e1 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:00:20 (1786064420) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:00:24 (1786064424) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:00:28 (1786064428) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.131834 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1df0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1df0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1df0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1510000 Read this (4 bytes): BBBB 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:01:10 (1786064470) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0484977 s, 169 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:01:19 (1786064479) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731471 s, 14.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x235 lmm_fid: [0x200000404:0x235:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x235 lmm_fid: [0x200000404:0x235:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834485 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-06 21:01:21.000000000 -0400 Modify: 2026-08-06 21:01:21.000000000 -0400 Change: 2026-08-06 21:01:23.000000000 -0400 Birth: 2026-08-06 21:01:21.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:01:29 (1786064489) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:01:34 (1786064494) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.461475 s, 88.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0558064 s, 73.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 6 21:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 6 21:01 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:01:45 (1786064505) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:01:55 (1786064515) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1927 sec =========== 21:02:04 (1786064524) === sanity-flr: start cleanup 21:02:06 (1786064526) === === sanity-flr: finish cleanup 21:02:12 (1786064532) ===