-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 14:33:49 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg649-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:34:09 (1787855649) === oleg649-client.virtnet: executing check_config_client /mnt/lustre oleg649-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg649-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff896249659800.idle_timeout=debug osc.lustre-OST0001-osc-ffff896249659800.idle_timeout=debug disable quota as required oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:34:43 (1787855683) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7640] == sanity-flr test 0a: lfs mirror create with -N option == 14:34:46 (1787855686) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:35:02 (1787855702) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:35:05 (1787855705) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:35:09 (1787855709) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:35:37 (1787855737) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:35:40 (1787855740) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:35:43 (1787855743) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff896249659800.stats=clear osc.lustre-OST0001-osc-ffff896249659800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:35:52 (1787855752) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:36:04 (1787855764) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:36:13 (1787855773) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:36:17 (1787855777) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:36:25 (1787855785) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 14:36:33 (1787855793) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:36:40 (1787855800) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:36:44 (1787855804) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:36:52 (1787855812) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:37:03 (1787855823) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:37:10 (1787855830) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.85769 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145548 s, 7.2 MB/s PASS 21 (16s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:37:26 (1787855846) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.370513 s, 22.6 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:37:37 (1787855857) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:37:47 (1787855867) Stopping /mnt/lustre-ost1 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg649-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg649-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:39:02 (1787855942) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.235359 s, 17.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg649-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:39:40 (1787855980) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg649-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg649-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:41:03 (1787856063) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.82254 s, 9.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 14:41 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.159s user 0m0.000s sys 0m0.200s Stopping /mnt/lustre-ost1 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.390s user 0m0.002s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg649-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg649-server oleg649-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.607s user 0m0.002s sys 0m0.226s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg649-server: oleg649-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg649-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg649-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff896249659800.ost_server_uuid in FULL state after 0 sec PASS 33b (106s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:42:49 (1787856169) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:42:53 (1787856173) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:42:57 (1787856177) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:43:00 (1787856180) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 14:43:08 (1787856188) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.936108 s, 13.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:43:22 (1787856202) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624388 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832061 s, 12.6 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:43:31 (1787856211) PASS 36c (11s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:43:42 (1787856222) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00164141 s, 624 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122209 s, 838 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000801369 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141853 s, 722 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 14:43:53 (1787856233) create test files with size 15752412 .. a5928954a4893974a5231a07bdf7a9ba - 5a148ee7f71f42a05211ccd4d11e8405 - e81e39971cea1f106c8567e4618855d9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg649-client.virtnet /mnt/lustre (opts:) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 14:44:18 (1787856258) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:44:32 (1787856272) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:44:38 (1787856278) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135092 s, 7.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032089 s, 32.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 14:44:48 (1787856288) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205556 s, 10.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.497235 s, 8.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330492 s, 6.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 14:45:08 (1787856308) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:45:12 (1787856312) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:45:15 (1787856315) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431347 s, 24.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:45:23 (1787856323) llite.lustre-ffff896243075800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:45:30 (1787856330) llite.lustre-ffff896243075800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335301 s, 31.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535314 s, 19.6 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:45:38 (1787856338) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:45:43 (1787856343) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc0:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:45:54 (1787856354) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.375114 s, 28.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg649-client.virtnet /mnt/lustre (opts:) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:46:08 (1787856368) llite.lustre-ffff896244af8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.404146 s, 25.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg649-client.virtnet /mnt/lustre (opts:) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff896245444800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:46:24 (1787856384) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:47:10 (1787856430) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:47:48 (1787856468) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:47:51 (1787856471) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.284712 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.193548 s, 16.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9d0a02ba (nosync,stale) Mirror 2: 0x9d0a02ba (nosync,stale) ** verify mirror 2 doesn't change original checksum: fd9f9f799ba5f3270604f04ad9738024 - mirror 1 checksum: 844330b5706b6d1da23aaa7ec9fb131a - mirror 2 checksum: fd9f9f799ba5f3270604f04ad9738024 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:47:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:47:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fd9f9f799ba5f3270604f04ad9738024 - mirror 1 checksum: 844330b5706b6d1da23aaa7ec9fb131a - mirror 2 checksum: 844330b5706b6d1da23aaa7ec9fb131a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:47:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:47:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:48:04 (1787856484) Failing mds1 on oleg649-server Stopping /mnt/lustre-mds1 (opts:) on oleg649-server 14:48:12 (1787856492) shut down facet: mds1 facet_host: oleg649-server facet_failover_host: oleg649-server Failover mds1 to oleg649-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg649-server: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-MDT0000 14:48:46 (1787856526) targets are mounted 14:48:46 (1787856526) facet_failover done oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344588 s, 30.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:49:08 (1787856548) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:49:11 (1787856551) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:49:14 (1787856554) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0128752 s, 1.3 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:49:23 (1787856563) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:49:27 (1787856567) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0310819 s, 132 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.080656 s, 50.8 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:49:36 (1787856576) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:49:59 (1787856599) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:50:28 (1787856628) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:50:53 (1787856653) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:51:17 (1787856677) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 14:51:24 (1787856684) mirror create pid 55597 mirror split pid 55599 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55597 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55599 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 14:52:31 (1787856751) Seed set to 6752 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787856753.013989 write 0x03d8b7 thru 0x03ffff (0x2749 bytes) 000002 1787856753.034519 write 0x0245fe thru 0x0287e4 (0x41e7 bytes) 000003 1787856753.056404 mapwrite 0x02aebf thru 0x039d52 (0xee94 bytes) truncating to largest ever: 0x33db4 000004 1787856753.174961 trunc from 0x040000 to 0x033db3 (0xc24c bytes) 000005 1787856753.197834 mapread 0x002101 thru 0x00c2ed (0xa1ed bytes) 000006 1787856753.210312 mapwrite 0x029532 thru 0x037497 (0xdf66 bytes) 000009 1787856753.293208 mapwrite 0x00be25 thru 0x017004 (0xb1e0 bytes) 000012 1787856753.366209 write 0x0305b2 thru 0x0360ef (0x5b3e bytes) 000013 1787856753.368969 trunc from 0x037498 to 0x0277e0 (0xfcb7 bytes) 000014 1787856753.392389 mapread 0x01ec20 thru 0x02594f (0x6d30 bytes) 000015 1787856753.419272 trunc from 0x0277e1 to 0x01353f (0x142a1 bytes) 000016 1787856753.456319 write 0x015d32 thru 0x02430e (0xe5dd bytes) 000017 1787856753.459490 mapread 0x01d37a thru 0x02430e (0x6f95 bytes) 000018 1787856753.461192 mapread 0x011ba3 thru 0x012d02 (0x1160 bytes) 000020 1787856753.467875 mapwrite 0x023e68 thru 0x025ad4 (0x1c6d bytes) 000021 1787856753.577898 read 0x01aa03 thru 0x01d2fb (0x28f9 bytes) 000022 1787856753.578939 write 0x02696a thru 0x02a308 (0x399f bytes) 000025 1787856753.581766 mapread 0x006f7d thru 0x01625f (0xf2e3 bytes) 000026 1787856753.587343 write 0x03e0e8 thru 0x03ffff (0x1f18 bytes) 000028 1787856753.589605 trunc from 0x040000 to 0x007eca (0x38135 bytes) 000029 1787856753.619994 trunc from 0x007ecb to 0x02b70b (0x23841 bytes) 000030 1787856753.650556 MIRROR_EXTEND to 2 mirrors 000031 1787856754.494591 trunc from 0x02b70c to 0x02bc8e (0x583 bytes) 000034 1787856754.637363 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000039 1787856754.772667 mapread 0x00848f thru 0x01304a (0xabbc bytes) 000042 1787856754.820318 mapread 0x00263e thru 0x0105a4 (0xdf67 bytes) 000043 1787856754.828184 trunc from 0x02bc8f to 0x006d60 (0x24f2e bytes) 000044 1787856754.844005 write 0x00c59f thru 0x01115f (0x4bc1 bytes) 000046 1787856754.845791 read 0x003dd0 thru 0x00eb01 (0xad32 bytes) 000049 1787856754.855691 mapread 0x00db7c thru 0x010bd2 (0x3057 bytes) 000050 1787856754.857770 mapwrite 0x017709 thru 0x01d433 (0x5d2b bytes) 000051 1787856754.917677 read 0x00d8ae thru 0x01bb73 (0xe2c6 bytes) 000057 1787856754.923122 read 0x00adf7 thru 0x0191da (0xe3e4 bytes) 000058 1787856754.924758 trunc from 0x01d434 to 0x023b21 (0x66ee bytes) 000062 1787856754.943440 write 0x003981 thru 0x003993 (0x13 bytes) 000064 1787856754.949745 MIRROR_EXTEND to 2 mirrors 000067 1787856755.953787 trunc from 0x023b22 to 0x028f41 (0x5420 bytes) 000068 1787856756.033010 mapwrite 0x018a1b thru 0x019d1a (0x1300 bytes) 000069 1787856756.118710 trunc from 0x028f42 to 0x01f42e (0x9b13 bytes) 000070 1787856756.165589 mapread 0x0035be thru 0x00fa5b (0xc49e bytes) 000071 1787856756.199683 trunc from 0x01f42f to 0x0131a7 (0xc287 bytes) 000072 1787856756.229139 mapread 0x0123a4 thru 0x0131a7 (0xe04 bytes) 000073 1787856756.237024 mapwrite 0x0384f3 thru 0x03ffff (0x7b0d bytes) 000074 1787856756.318415 read 0x02868b thru 0x02c44b (0x3dc1 bytes) 000076 1787856756.328375 trunc from 0x040000 to 0x025b50 (0x1a4af bytes) truncating to largest ever: 0x3f6b3 000077 1787856756.345111 trunc from 0x025b51 to 0x03f6b2 (0x19b62 bytes) 000078 1787856756.365395 trunc from 0x03f6b3 to 0x018090 (0x27622 bytes) 000079 1787856756.389976 trunc from 0x018091 to 0x03a3aa (0x2231a bytes) 000080 1787856756.420400 mapwrite 0x036ffb thru 0x03c973 (0x5979 bytes) 000081 1787856756.518839 mapwrite 0x00bfe4 thru 0x00c217 (0x234 bytes) 000083 1787856756.626707 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c974): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c974): Mirror 2: 0xf2c3eba3 (stale) CRC-32 checksum value for chunk [0, 0x3c974): Mirror 1: 0xe711c2f2 Mirror 3: 0xe711c2f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c974: skipped 000086 1787856758.021704 mapread 0x02c2e7 thru 0x03bc35 (0xf94f bytes) 000088 1787856758.071312 mapread 0x039fa7 thru 0x03c973 (0x29cd bytes) 000090 1787856758.072471 write 0x03c362 thru 0x03ffff (0x3c9e bytes) 000091 1787856758.233124 trunc from 0x040000 to 0x031b42 (0xe4bd bytes) 000092 1787856758.287304 mapwrite 0x0019ea thru 0x008bc8 (0x71df bytes) 000093 1787856758.393548 mapwrite 0x014db8 thru 0x0194b3 (0x46fc bytes) 000094 1787856758.507064 read 0x0100cd thru 0x01284d (0x2781 bytes) 000095 1787856758.507753 write 0x02cf0f thru 0x02e30e (0x1400 bytes) 000096 1787856758.530627 mapread 0x0147f8 thru 0x0224b2 (0xdcbb bytes) 000097 1787856758.560908 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b43): Mirror 3: 0x52fc3f15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b43): Mirror 3: 0x52fc3f15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b43: skipped 000098 1787856758.932097 trunc from 0x031b43 to 0x013712 (0x1e430 bytes) 000099 1787856758.958100 trunc from 0x013713 to 0x027d90 (0x1467e bytes) 000101 1787856758.984971 write 0x01fc79 thru 0x025f7b (0x6303 bytes) 000102 1787856758.999325 mapwrite 0x02d4d6 thru 0x03a5ce (0xd0f9 bytes) 000103 1787856759.140634 trunc from 0x03a5cf to 0x022de7 (0x177e7 bytes) 000108 1787856759.172174 mapread 0x019b05 thru 0x022de7 (0x92e3 bytes) 000111 1787856759.183399 write 0x0370a8 thru 0x03d82f (0x6788 bytes) 000112 1787856759.186377 trunc from 0x03d830 to 0x032099 (0xb796 bytes) 000115 1787856759.209264 mapwrite 0x03aa37 thru 0x03ffff (0x55c9 bytes) 000117 1787856759.314372 mapread 0x0082ec thru 0x014e06 (0xcb1b bytes) 000119 1787856759.346471 mapwrite 0x0026e7 thru 0x0119bc (0xf2d6 bytes) 000120 1787856759.540293 write 0x01efbf thru 0x02d849 (0xe88b bytes) 000121 1787856759.551399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa442f07b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2b1043d5 Mirror 4: 0x2b1043d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1787856761.265233 read 0x00ea74 thru 0x015177 (0x6704 bytes) 000124 1787856761.291428 trunc from 0x040000 to 0x0288c7 (0x17738 bytes) 000128 1787856761.459166 mapread 0x00c2cd thru 0x00f994 (0x36c8 bytes) 000129 1787856761.483147 write 0x03f312 thru 0x03ffff (0xcee bytes) 000130 1787856761.485005 mapread 0x028ac7 thru 0x03720a (0xe744 bytes) 000131 1787856761.506981 mapread 0x00978f thru 0x017369 (0xdbdb bytes) 000133 1787856761.517968 mapread 0x01c3b8 thru 0x02011a (0x3d63 bytes) 000134 1787856761.525162 write 0x0045f2 thru 0x00df4f (0x995e bytes) 000136 1787856761.538157 MIRROR_RESYNC 3 mirrors 000138 1787856762.943126 MIRROR_RESYNC 3 mirrors 000139 1787856763.353983 mapread 0x01644a thru 0x0228ce (0xc485 bytes) 000143 1787856763.391416 mapread 0x01908f thru 0x01b51a (0x248c bytes) 000147 1787856763.400310 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000149 1787856763.834702 MIRROR_RESYNC 2 mirrors 000151 1787856764.270149 mapread 0x02531e thru 0x02b28d (0x5f70 bytes) 000152 1787856764.313607 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000157 1787856764.538137 read 0x033d3c thru 0x03ef31 (0xb1f6 bytes) 000159 1787856764.549917 read 0x027645 thru 0x02abee (0x35aa bytes) 000161 1787856764.563258 read 0x0295d2 thru 0x032a3d (0x946c bytes) 000162 1787856764.564831 write 0x03726f thru 0x03ffff (0x8d91 bytes) 000166 1787856764.601657 write 0x0337ae thru 0x03ed0a (0xb55d bytes) 000167 1787856764.603473 write 0x01554d thru 0x019e88 (0x493c bytes) 000168 1787856764.604550 write 0x03da11 thru 0x03ffff (0x25ef bytes) 000174 1787856764.608119 write 0x020745 thru 0x02a9dc (0xa298 bytes) 000177 1787856764.610685 read 0x01370c thru 0x01b94f (0x8244 bytes) 000179 1787856764.611725 read 0x01fdb8 thru 0x024c96 (0x4edf bytes) 000180 1787856764.612446 mapread 0x00ea83 thru 0x01014e (0x16cc bytes) 000183 1787856764.618658 MIRROR_EXTEND to 2 mirrors 000184 1787856765.576270 write 0x029b69 thru 0x037538 (0xd9d0 bytes) 000185 1787856765.673637 trunc from 0x040000 to 0x030414 (0xfbeb bytes) 000187 1787856765.705381 write 0x02ea46 thru 0x032c05 (0x41c0 bytes) 000190 1787856765.708906 trunc from 0x032c06 to 0x00643c (0x2c7c9 bytes) 000194 1787856765.728359 trunc from 0x00643d to 0x022be3 (0x1c7a7 bytes) 000197 1787856765.747295 mapwrite 0x027a06 thru 0x029144 (0x173f bytes) 000198 1787856765.830479 trunc from 0x029145 to 0x00467f (0x24ac5 bytes) 000199 1787856765.859275 mapread 0x002041 thru 0x00467f (0x263f bytes) 000201 1787856765.890243 mapwrite 0x011309 thru 0x018c3e (0x7936 bytes) 000202 1787856765.963299 read 0x017dc3 thru 0x018c3e (0xe7c bytes) 000203 1787856765.965062 write 0x030633 thru 0x034496 (0x3e64 bytes) 000204 1787856765.970007 mapwrite 0x01262b thru 0x01c76c (0xa142 bytes) 000205 1787856766.058778 trunc from 0x034497 to 0x01f377 (0x1511f bytes) 000206 1787856766.086620 read 0x01a8fa thru 0x01aa95 (0x19c bytes) 000207 1787856766.087477 write 0x00a81b thru 0x01307f (0x8865 bytes) 000210 1787856766.091219 mapwrite 0x01b232 thru 0x02619b (0xaf6a bytes) 000212 1787856766.182647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000214 1787856766.375322 mapread 0x01fa92 thru 0x022e55 (0x33c4 bytes) 000217 1787856766.393355 mapwrite 0x0290b3 thru 0x02cf82 (0x3ed0 bytes) 000219 1787856766.467622 read 0x00ab8a thru 0x00ba86 (0xefd bytes) 000222 1787856766.474066 read 0x01e191 thru 0x0257d3 (0x7643 bytes) 000223 1787856766.474704 read 0x0026a1 thru 0x00d0d1 (0xaa31 bytes) 000226 1787856766.482168 mapread 0x01bbfb thru 0x020eda (0x52e0 bytes) 000227 1787856766.483214 read 0x01875a thru 0x01989b (0x1142 bytes) 000229 1787856766.484324 trunc from 0x02cf83 to 0x01f97f (0xd603 bytes) 000230 1787856766.498418 mapread 0x01c7a1 thru 0x01f97f (0x31df bytes) 000231 1787856766.499410 trunc from 0x01f980 to 0x010fc8 (0xe9b7 bytes) 000234 1787856766.513326 mapwrite 0x0387ad thru 0x03ffff (0x7853 bytes) 000235 1787856766.579312 mapread 0x00a714 thru 0x018d4a (0xe637 bytes) 000236 1787856766.581458 mapread 0x02d64e thru 0x036d03 (0x96b6 bytes) 000239 1787856766.584064 mapwrite 0x0197ba thru 0x01e174 (0x49bb bytes) 000240 1787856766.639076 write 0x00c5ed thru 0x015bc2 (0x95d6 bytes) 000242 1787856766.642773 read 0x00a322 thru 0x00a75d (0x43c bytes) 000246 1787856766.646592 mapwrite 0x03787c thru 0x03ffff (0x8784 bytes) 000248 1787856766.712161 read 0x0289e8 thru 0x02f564 (0x6b7d bytes) 000253 1787856766.716239 read 0x0392d3 thru 0x03f7e1 (0x650f bytes) 000254 1787856766.716878 write 0x01858e thru 0x018ff4 (0xa67 bytes) 000257 1787856766.718942 mapwrite 0x0086bd thru 0x017fc9 (0xf90d bytes) 000261 1787856766.770313 read 0x03d9d5 thru 0x03ffff (0x262b bytes) 000262 1787856766.771123 mapread 0x00425c thru 0x00ffcc (0xbd71 bytes) 000265 1787856766.773637 read 0x034615 thru 0x03af82 (0x696e bytes) 000266 1787856766.774443 read 0x00fd98 thru 0x0144ed (0x4756 bytes) 000269 1787856766.776471 mapwrite 0x02e2d3 thru 0x03d2f9 (0xf027 bytes) 000270 1787856766.827333 mapread 0x0114b6 thru 0x0206a9 (0xf1f4 bytes) 000272 1787856766.829691 mapread 0x003898 thru 0x00c573 (0x8cdc bytes) 000274 1787856766.831472 mapread 0x019c01 thru 0x0244b4 (0xa8b4 bytes) 000275 1787856766.832959 write 0x03ef37 thru 0x03ffff (0x10c9 bytes) 000278 1787856766.839774 MIRROR_EXTEND to 2 mirrors 000279 1787856767.711591 trunc from 0x040000 to 0x016686 (0x29979 bytes) 000280 1787856767.797253 mapwrite 0x03e3c9 thru 0x03ffff (0x1c37 bytes) 000281 1787856767.879056 read 0x01e1e5 thru 0x0245cf (0x63eb bytes) 000283 1787856767.891459 MIRROR_RESYNC 2 mirrors 000287 1787856768.692038 read 0x019f8c thru 0x029bf2 (0xfc67 bytes) 000289 1787856768.701875 write 0x014e38 thru 0x015040 (0x209 bytes) 000290 1787856768.767854 mapread 0x024e7b thru 0x0312d1 (0xc457 bytes) 000291 1787856768.807595 MIRROR_RESYNC 2 mirrors 000295 1787856769.927291 MIRROR_RESYNC 2 mirrors 000298 1787856770.241872 write 0x02b3fa thru 0x039d9a (0xe9a1 bytes) 000300 1787856770.330821 mapwrite 0x02739a thru 0x0343bc (0xd023 bytes) 000301 1787856770.413004 trunc from 0x040000 to 0x027568 (0x18a97 bytes) 000302 1787856770.458305 mapwrite 0x00cb66 thru 0x01b883 (0xed1e bytes) 000304 1787856770.555686 write 0x026778 thru 0x0328b5 (0xc13e bytes) 000306 1787856770.565248 read 0x003048 thru 0x007617 (0x45d0 bytes) 000307 1787856770.577590 read 0x00f91b thru 0x01e80e (0xeef4 bytes) 000310 1787856770.598473 mapwrite 0x027b86 thru 0x02b5ec (0x3a67 bytes) 000312 1787856770.664987 mapread 0x0268fd thru 0x0328b5 (0xbfb9 bytes) 000313 1787856770.666381 mapwrite 0x031431 thru 0x03f1a7 (0xdd77 bytes) 000314 1787856770.749678 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000315 1787856770.911059 read 0x000c1a thru 0x00f2b9 (0xe6a0 bytes) 000317 1787856770.929845 trunc from 0x03f1a8 to 0x00ad77 (0x34430 bytes) 000318 1787856770.950678 mapread 0x0020b8 thru 0x00ad77 (0x8cc0 bytes) 000319 1787856770.951750 read 0x005dd2 thru 0x00ad77 (0x4fa6 bytes) 000320 1787856770.953203 read 0x000ef6 thru 0x0073ef (0x64fa bytes) 000325 1787856770.956840 write 0x03719d thru 0x037d5f (0xbc3 bytes) 000327 1787856770.958394 mapread 0x028e1a thru 0x031d0a (0x8ef1 bytes) 000333 1787856770.970863 trunc from 0x037d60 to 0x03f538 (0x77d9 bytes) 000334 1787856770.997639 mapread 0x02701c thru 0x02d0bc (0x60a1 bytes) 000335 1787856770.998700 mapread 0x035e56 thru 0x0381fb (0x23a6 bytes) 000338 1787856771.015423 read 0x032647 thru 0x03703d (0x49f7 bytes) 000339 1787856771.016018 write 0x01212f thru 0x018e26 (0x6cf8 bytes) 000340 1787856771.022866 read 0x0337d9 thru 0x035f72 (0x279a bytes) 000343 1787856771.036595 read 0x00d89d thru 0x0117d4 (0x3f38 bytes) 000345 1787856771.038633 write 0x00faa8 thru 0x0135d8 (0x3b31 bytes) 000346 1787856771.040060 write 0x008358 thru 0x00f253 (0x6efc bytes) 000347 1787856771.041856 mapread 0x00983a thru 0x00db3d (0x4304 bytes) 000349 1787856771.044310 mapwrite 0x031d71 thru 0x03a150 (0x83e0 bytes) 000350 1787856771.155868 trunc from 0x03f539 to 0x0309ad (0xeb8b bytes) 000353 1787856771.178197 trunc from 0x0309ae to 0x01339a (0x1d613 bytes) 000355 1787856771.204181 mapwrite 0x01301c thru 0x016370 (0x3355 bytes) 000356 1787856771.276462 read 0x012787 thru 0x016370 (0x3bea bytes) 000358 1787856771.279557 read 0x00c2fd thru 0x00de6d (0x1b71 bytes) 000361 1787856771.283548 write 0x005365 thru 0x00e954 (0x95f0 bytes) 000362 1787856771.285449 read 0x00146d thru 0x01069f (0xf233 bytes) 000365 1787856771.288801 write 0x00a565 thru 0x01491a (0xa3b6 bytes) 000368 1787856771.292274 mapwrite 0x00f676 thru 0x017381 (0x7d0c bytes) 000369 1787856771.388248 trunc from 0x017382 to 0x03035d (0x18fdc bytes) 000370 1787856771.418547 mapwrite 0x02cb08 thru 0x039a00 (0xcef9 bytes) 000372 1787856771.574694 mapread 0x00ee25 thru 0x01ce40 (0xe01c bytes) 000374 1787856771.577677 write 0x03ae0d thru 0x03ffff (0x51f3 bytes) 000376 1787856771.580404 mapwrite 0x00acdd thru 0x00e681 (0x39a5 bytes) 000378 1787856771.645682 read 0x005847 thru 0x011322 (0xbadc bytes) 000379 1787856771.646531 trunc from 0x040000 to 0x0354b7 (0xab48 bytes) 000380 1787856771.673548 MIRROR_EXTEND to 2 mirrors 000381 1787856772.583719 trunc from 0x0354b8 to 0x013eb3 (0x21604 bytes) 000383 1787856772.676389 mapwrite 0x029571 thru 0x030f53 (0x79e3 bytes) 000385 1787856772.769640 mapwrite 0x02f4f1 thru 0x0333d7 (0x3ee7 bytes) 000386 1787856772.836757 read 0x019d8b thru 0x029104 (0xf37a bytes) 000387 1787856772.848414 MIRROR_RESYNC 2 mirrors 000388 1787856774.117378 mapwrite 0x017ecb thru 0x01f6c4 (0x77fa bytes) 000391 1787856774.486595 read 0x0261e6 thru 0x03061e (0xa439 bytes) 000397 1787856774.507806 mapread 0x0230de thru 0x023c9b (0xbbe bytes) 000398 1787856774.544180 read 0x00d979 thru 0x013997 (0x601f bytes) 000399 1787856774.577263 MIRROR_RESYNC 2 mirrors 000400 1787856775.541978 read 0x03288b thru 0x032ea3 (0x619 bytes) 000404 1787856775.548401 write 0x01e6f9 thru 0x0213cf (0x2cd7 bytes) 000405 1787856775.661098 mapwrite 0x03508e thru 0x03ffff (0xaf72 bytes) 000407 1787856775.803197 read 0x02eca3 thru 0x0331c1 (0x451f bytes) 000408 1787856775.828172 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x43c2cb2a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x13a2ddbd Mirror 3: 0x13a2ddbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1787856776.927485 mapread 0x011810 thru 0x012f9e (0x178f bytes) 000410 1787856776.948770 trunc from 0x040000 to 0x0319fe (0xe601 bytes) 000412 1787856777.097036 trunc from 0x0319ff to 0x02cd72 (0x4c8c bytes) 000413 1787856777.126900 mapwrite 0x0144d8 thru 0x022f24 (0xea4d bytes) 000416 1787856777.213358 read 0x015da9 thru 0x01fbc6 (0x9e1e bytes) 000417 1787856777.214900 read 0x019224 thru 0x01a4f5 (0x12d2 bytes) 000418 1787856777.216334 read 0x0033f3 thru 0x0076b6 (0x42c4 bytes) 000419 1787856777.230472 mapread 0x021019 thru 0x02cd72 (0xbd5a bytes) 000421 1787856777.232363 write 0x02604f thru 0x02a37a (0x432c bytes) 000422 1787856777.237433 MIRROR_RESYNC 3 mirrors 000424 1787856778.264033 mapwrite 0x017b54 thru 0x020745 (0x8bf2 bytes) 000426 1787856778.492890 write 0x039a39 thru 0x03ffff (0x65c7 bytes) 000428 1787856778.496157 write 0x02a0b6 thru 0x02fdfd (0x5d48 bytes) 000430 1787856778.521284 write 0x02a8d4 thru 0x03379c (0x8ec9 bytes) 000431 1787856778.536836 mapwrite 0x03c80e thru 0x03ffff (0x37f2 bytes) 000434 1787856778.656469 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbf37f39e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbf37f39e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1787856779.298760 mapread 0x000974 thru 0x0043dd (0x3a6a bytes) 000436 1787856779.325479 read 0x03cfa0 thru 0x03ffff (0x3060 bytes) 000438 1787856779.334649 read 0x037cec thru 0x03ffff (0x8314 bytes) 000441 1787856779.347470 write 0x02eb5b thru 0x031abe (0x2f64 bytes) 000442 1787856779.348609 read 0x030305 thru 0x03e8b9 (0xe5b5 bytes) 000443 1787856779.349699 read 0x033efb thru 0x037a41 (0x3b47 bytes) 000445 1787856779.351048 trunc from 0x040000 to 0x036f62 (0x909d bytes) 000447 1787856779.394381 mapread 0x007bd5 thru 0x00c088 (0x44b4 bytes) 000448 1787856779.395398 mapwrite 0x0297cd thru 0x038884 (0xf0b8 bytes) 000449 1787856779.544079 mapwrite 0x01453f thru 0x01bd74 (0x7836 bytes) 000450 1787856779.693611 mapwrite 0x00469f thru 0x01120f (0xcb71 bytes) 000451 1787856779.932360 mapwrite 0x026cce thru 0x026f2d (0x260 bytes) 000453 1787856780.067463 trunc from 0x038885 to 0x01c45e (0x1c426 bytes) 000455 1787856780.091416 read 0x00c404 thru 0x00dc12 (0x180f bytes) 000463 1787856780.103843 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c45f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c45f): Mirror 3: 0xc9ec24cf (stale) CRC-32 checksum value for chunk [0, 0x1c45f): Mirror 1: 0x98cd03f7 Mirror 4: 0x98cd03f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c45f: skipped 000464 1787856781.316163 read 0x0109f8 thru 0x0193e6 (0x89ef bytes) 000466 1787856781.326553 mapread 0x01763e thru 0x01c45e (0x4e21 bytes) 000467 1787856781.343299 write 0x036e21 thru 0x03ffff (0x91df bytes) 000468 1787856781.530643 mapwrite 0x02bf94 thru 0x02e790 (0x27fd bytes) 000469 1787856781.602715 mapwrite 0x039e34 thru 0x03ffff (0x61cc bytes) 000471 1787856781.665438 mapwrite 0x01d33b thru 0x020df3 (0x3ab9 bytes) 000473 1787856781.746538 write 0x036bd3 thru 0x03e190 (0x75be bytes) 000474 1787856781.755668 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbf37f39e (stale) Mirror 4: 0xc71e22f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x362b4485 Mirror 5: 0x362b4485 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1787856783.449219 trunc from 0x040000 to 0x0269ec (0x19613 bytes) 000476 1787856783.550081 trunc from 0x0269ed to 0x005f62 (0x20a8a bytes) 000477 1787856783.597964 mapwrite 0x0381a3 thru 0x03c316 (0x4174 bytes) 000481 1787856783.779338 write 0x034e1f thru 0x035b64 (0xd46 bytes) 000482 1787856783.782211 trunc from 0x03c317 to 0x004c20 (0x376f6 bytes) 000483 1787856783.807216 mapwrite 0x0091b8 thru 0x0159f7 (0xc840 bytes) 000484 1787856783.886822 trunc from 0x0159f8 to 0x014496 (0x1561 bytes) 000486 1787856783.909143 write 0x018ce0 thru 0x0268a0 (0xdbc1 bytes) 000488 1787856783.914437 mapread 0x020861 thru 0x0268a0 (0x6040 bytes) 000489 1787856783.916275 read 0x022af6 thru 0x0268a0 (0x3dab bytes) 000490 1787856783.917284 mapread 0x004520 thru 0x00c308 (0x7de9 bytes) 000493 1787856783.920352 mapwrite 0x01cf5a thru 0x0286b0 (0xb757 bytes) 000494 1787856783.994261 read 0x00a1cb thru 0x012e51 (0x8c87 bytes) 000495 1787856783.995262 trunc from 0x0286b1 to 0x022a26 (0x5c8a bytes) 000496 1787856784.012329 mapread 0x00bbfa thru 0x014e14 (0x921b bytes) 000497 1787856784.013808 write 0x01aba2 thru 0x01caca (0x1f29 bytes) 000498 1787856784.021098 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22a27): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a27): Mirror 3: 0x3c4bfbae (stale) Mirror 4: 0xae42ffc3 (stale) Mirror 5: 0xd3eabaae (stale) CRC-32 checksum value for chunk [0, 0x22a27): Mirror 1: 0x14670cd3 Mirror 6: 0x14670cd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a27: skipped 000500 1787856784.869077 write 0x024a5c thru 0x030aa4 (0xc049 bytes) 000501 1787856784.953728 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30aa5): Mirror 3: 0xfc50e7e7 (stale) Mirror 4: 0xbfae93f2 (stale) Mirror 6: 0xd4f37f42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30aa5): Mirror 3: 0xfc50e7e7 (stale) Mirror 4: 0xbfae93f2 (stale) Mirror 6: 0xd4f37f42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30aa5: skipped 000502 1787856785.391612 trunc from 0x030aa5 to 0x039103 (0x865f bytes) 000504 1787856785.445461 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39104): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39104): Mirror 3: 0xbb94acd3 (stale) Mirror 4: 0xa8bb01f2 (stale) Mirror 6: 0x3536ce50 (stale) CRC-32 checksum value for chunk [0, 0x39104): Mirror 1: 0x137b6631 Mirror 7: 0x137b6631 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39104: skipped 000507 1787856787.002790 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39104): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39104): Mirror 3: 0xbb94acd3 (stale) Mirror 4: 0xa8bb01f2 (stale) Mirror 6: 0x3536ce50 (stale) CRC-32 checksum value for chunk [0, 0x39104): Mirror 1: 0x137b6631 Mirror 7: 0x137b6631 Mirror 8: 0x137b6631 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39104: skipped truncating to largest ever: 0x3f8e6 000508 1787856788.216179 trunc from 0x039104 to 0x03f8e5 (0x67e2 bytes) 000509 1787856788.361783 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8e6): Mirror 4: 0x762ba8b7 (stale) Mirror 6: 0x5cefb7f9 (stale) Mirror 7: 0x673f847f (stale) Mirror 8: 0x673f847f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8e6): Mirror 4: 0x762ba8b7 (stale) Mirror 6: 0x5cefb7f9 (stale) Mirror 7: 0x673f847f (stale) Mirror 8: 0x673f847f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8e6: skipped 000510 1787856789.032988 trunc from 0x03f8e6 to 0x006f26 (0x389bf bytes) 000511 1787856789.062786 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x6f27): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6f27): Mirror 4: 0x7b191025 (stale) Mirror 6: 0x62fdd45d (stale) Mirror 7: 0x62fdd45d (stale) Mirror 8: 0x62fdd45d (stale) CRC-32 checksum value for chunk [0, 0x6f27): Mirror 1: 0x62fdd45d Mirror 9: 0x62fdd45d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6f27: skipped 000513 1787856790.357329 read 0x005f36 thru 0x006f26 (0xff1 bytes) 000517 1787856790.364373 trunc from 0x006f27 to 0x0274d5 (0x205af bytes) 000518 1787856790.459659 read 0x018e63 thru 0x02556b (0xc709 bytes) 000519 1787856790.476262 write 0x03243b thru 0x03ffff (0xdbc5 bytes) 000525 1787856790.486349 MIRROR_RESYNC 6 mirrors 000528 1787856792.549363 write 0x01eac4 thru 0x0211c7 (0x2704 bytes) 000530 1787856792.625279 trunc from 0x040000 to 0x02003c (0x1ffc3 bytes) 000532 1787856792.654557 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2003d): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2003d): Mirror 4: 0xf1ca056d (stale) Mirror 6: 0xf1ca056d (stale) Mirror 7: 0xf1ca056d (stale) Mirror 8: 0xf1ca056d (stale) Mirror 9: 0xf1ca056d (stale) CRC-32 checksum value for chunk [0, 0x2003d): Mirror 1: 0x13452ffb Mirror 10: 0x13452ffb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2003d: skipped 000533 1787856794.121762 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2003d): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2003d): Mirror 4: 0xf1ca056d (stale) Mirror 6: 0xf1ca056d (stale) Mirror 7: 0xf1ca056d (stale) Mirror 8: 0xf1ca056d (stale) Mirror 9: 0xf1ca056d (stale) CRC-32 checksum value for chunk [0, 0x2003d): Mirror 1: 0x13452ffb Mirror 10: 0x13452ffb Mirror 11: 0x13452ffb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2003d: skipped 000536 1787856795.488555 read 0x017de8 thru 0x02003c (0x8255 bytes) 000537 1787856795.497744 read 0x01e144 thru 0x02003c (0x1ef9 bytes) 000539 1787856795.499125 mapwrite 0x03ea86 thru 0x03ffff (0x157a bytes) 000540 1787856795.717546 mapwrite 0x01e338 thru 0x0288f1 (0xa5ba bytes) 000541 1787856795.804233 mapwrite 0x03dcd8 thru 0x03ffff (0x2328 bytes) 000543 1787856795.874604 read 0x017357 thru 0x025770 (0xe41a bytes) 000545 1787856795.875775 trunc from 0x040000 to 0x03dc9d (0x2362 bytes) 000548 1787856795.914457 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc9e): Mirror 4: 0x13fae145 (stale) Mirror 6: 0x13fae145 (stale) Mirror 8: 0x13fae145 (stale) Mirror 9: 0x13fae145 (stale) Mirror 10: 0x86607b53 (stale) Mirror 11: 0x86607b53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc9e): Mirror 4: 0x13fae145 (stale) Mirror 6: 0x13fae145 (stale) Mirror 8: 0x13fae145 (stale) Mirror 9: 0x13fae145 (stale) Mirror 10: 0x86607b53 (stale) Mirror 11: 0x86607b53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc9e: skipped 000549 1787856796.509951 write 0x02e3ac thru 0x030f2a (0x2b7f bytes) 000550 1787856796.523286 trunc from 0x03dc9e to 0x036436 (0x7867 bytes) 000556 1787856796.549294 mapread 0x010ef2 thru 0x016330 (0x543f bytes) 000558 1787856796.567314 mapwrite 0x004814 thru 0x00fe7b (0xb668 bytes) 000562 1787856796.641242 write 0x01da5b thru 0x02c31f (0xe8c5 bytes) 000563 1787856796.652880 write 0x020ed8 thru 0x02caa0 (0xbbc9 bytes) 000564 1787856796.659771 MIRROR_RESYNC 7 mirrors 000565 1787856799.167102 trunc from 0x036437 to 0x028aca (0xd96c bytes) 000566 1787856799.282284 mapwrite 0x03d478 thru 0x03ffff (0x2b88 bytes) 000567 1787856799.414049 read 0x00c561 thru 0x016b7a (0xa61a bytes) 000568 1787856799.428083 read 0x0144b5 thru 0x014c34 (0x780 bytes) 000572 1787856799.432708 write 0x002150 thru 0x0029cb (0x87c bytes) 000573 1787856799.450786 read 0x03ada6 thru 0x03ffff (0x525a bytes) 000574 1787856799.451369 mapread 0x00f4d0 thru 0x0190d4 (0x9c05 bytes) 000575 1787856799.452331 write 0x039f85 thru 0x03ffff (0x607b bytes) 000577 1787856799.455390 read 0x0340cc thru 0x03e98d (0xa8c2 bytes) 000579 1787856799.463726 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe03f0ee7 (stale) Mirror 8: 0xe03f0ee7 (stale) Mirror 9: 0xe03f0ee7 (stale) Mirror 10: 0xe03f0ee7 (stale) Mirror 11: 0xe03f0ee7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe03f0ee7 (stale) Mirror 8: 0xe03f0ee7 (stale) Mirror 9: 0xe03f0ee7 (stale) Mirror 10: 0xe03f0ee7 (stale) Mirror 11: 0xe03f0ee7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1787856799.999823 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe03f0ee7 (stale) Mirror 8: 0xe03f0ee7 (stale) Mirror 9: 0xe03f0ee7 (stale) Mirror 10: 0xe03f0ee7 (stale) Mirror 11: 0xe03f0ee7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x442897f9 Mirror 12: 0x442897f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1787856801.428482 MIRROR_RESYNC 7 mirrors 000584 1787856803.566224 trunc from 0x040000 to 0x031dee (0xe211 bytes) 000585 1787856803.654439 read 0x000dcc thru 0x00696c (0x5ba1 bytes) 000587 1787856803.668670 trunc from 0x031def to 0x028653 (0x979b bytes) 000588 1787856803.695242 read 0x001820 thru 0x00313d (0x191e bytes) 000592 1787856803.701544 write 0x012be8 thru 0x0142c6 (0x16df bytes) 000593 1787856803.714070 trunc from 0x028654 to 0x0361e8 (0xdb95 bytes) 000594 1787856803.743125 MIRROR_RESYNC 7 mirrors 000595 1787856805.796721 write 0x03141b thru 0x032ab3 (0x1699 bytes) 000596 1787856805.871365 mapread 0x000103 thru 0x000f21 (0xe1f bytes) 000597 1787856805.876656 read 0x0043b9 thru 0x0069e0 (0x2628 bytes) 000598 1787856805.898145 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x361e9): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361e9): Mirror 4: 0x2550ad1c (stale) Mirror 8: 0x2550ad1c (stale) Mirror 9: 0x2550ad1c (stale) Mirror 10: 0x2550ad1c (stale) Mirror 11: 0x2550ad1c (stale) Mirror 12: 0x2550ad1c (stale) CRC-32 checksum value for chunk [0, 0x361e9): Mirror 1: 0xa4cbf435 Mirror 13: 0xa4cbf435 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361e9: skipped 000600 1787856807.359379 trunc from 0x0361e9 to 0x00e185 (0x28063 bytes) 000601 1787856807.459738 trunc from 0x00e186 to 0x0398fa (0x2b775 bytes) 000603 1787856807.484151 mapwrite 0x01bda2 thru 0x023b72 (0x7dd1 bytes) 000604 1787856807.555344 read 0x03642b thru 0x0398fa (0x34d0 bytes) 000605 1787856807.565839 read 0x02bc14 thru 0x0398fa (0xdce7 bytes) 000606 1787856807.580250 mapwrite 0x0354e2 thru 0x03c5b2 (0x70d1 bytes) 000609 1787856807.671215 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5b3): Mirror 4: 0xd03aff69 (stale) Mirror 8: 0xd03aff69 (stale) Mirror 9: 0xd03aff69 (stale) Mirror 10: 0xd03aff69 (stale) Mirror 11: 0xd03aff69 (stale) Mirror 12: 0xd03aff69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5b3): Mirror 4: 0xd03aff69 (stale) Mirror 8: 0xd03aff69 (stale) Mirror 9: 0xd03aff69 (stale) Mirror 10: 0xd03aff69 (stale) Mirror 11: 0xd03aff69 (stale) Mirror 12: 0xd03aff69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5b3: skipped 000610 1787856808.238385 mapwrite 0x02770a thru 0x033705 (0xbffc bytes) 000611 1787856808.385071 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5b3): Mirror 4: 0xd03aff69 (stale) Mirror 8: 0xd03aff69 (stale) Mirror 9: 0xd03aff69 (stale) Mirror 10: 0xd03aff69 (stale) Mirror 12: 0xd03aff69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5b3): Mirror 4: 0xd03aff69 (stale) Mirror 8: 0xd03aff69 (stale) Mirror 9: 0xd03aff69 (stale) Mirror 10: 0xd03aff69 (stale) Mirror 12: 0xd03aff69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5b3: skipped 000612 1787856808.793517 mapwrite 0x033424 thru 0x03f48c (0xc069 bytes) 000613 1787856808.928183 write 0x01b895 thru 0x01eafe (0x326a bytes) 000615 1787856808.949243 trunc from 0x03f48d to 0x0160da (0x293b2 bytes) 000617 1787856808.992123 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160db): Mirror 4: 0xc2ac7e0a (stale) Mirror 8: 0xc2ac7e0a (stale) Mirror 9: 0xc2ac7e0a (stale) Mirror 12: 0xc2ac7e0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160db): Mirror 4: 0xc2ac7e0a (stale) Mirror 8: 0xc2ac7e0a (stale) Mirror 9: 0xc2ac7e0a (stale) Mirror 12: 0xc2ac7e0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160db: skipped 000618 1787856809.378751 write 0x021141 thru 0x02becc (0xad8c bytes) 000619 1787856809.380385 mapread 0x02279f thru 0x02becc (0x972e bytes) 000620 1787856809.382348 mapwrite 0x0298f2 thru 0x032f36 (0x9645 bytes) 000621 1787856809.478411 mapread 0x02d0b9 thru 0x0312f9 (0x4241 bytes) 000622 1787856809.480661 read 0x00c6fd thru 0x0131c3 (0x6ac7 bytes) 000624 1787856809.491010 write 0x02076e thru 0x023d9a (0x362d bytes) 000625 1787856809.496082 mapread 0x0248d1 thru 0x0258f0 (0x1020 bytes) 000627 1787856809.502828 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: 0xf69c55be (stale) Mirror 8: 0xf69c55be (stale) Mirror 9: 0xf69c55be (stale) Mirror 12: 0xf69c55be (stale) CRC-32 checksum value for chunk [0, 0x32f37): Mirror 1: 0x686c69f Mirror 13: 0x686c69f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f37: skipped 000628 1787856810.604314 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: 0xf69c55be (stale) Mirror 8: 0xf69c55be (stale) Mirror 9: 0xf69c55be (stale) Mirror 12: 0xf69c55be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: 0xf69c55be (stale) Mirror 8: 0xf69c55be (stale) Mirror 9: 0xf69c55be (stale) Mirror 12: 0xf69c55be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f37: skipped 000629 1787856810.992544 read 0x01eeab thru 0x02102d (0x2183 bytes) 000632 1787856811.013379 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: 0xf69c55be (stale) Mirror 8: 0xf69c55be (stale) Mirror 9: 0xf69c55be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f37): Mirror 4: 0xf69c55be (stale) Mirror 8: 0xf69c55be (stale) Mirror 9: 0xf69c55be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f37: skipped 000633 1787856811.381081 mapwrite 0x01cf3c thru 0x022c7a (0x5d3f bytes) 000634 1787856811.535352 write 0x0350c5 thru 0x03ffff (0xaf3b bytes) 000638 1787856811.540211 trunc from 0x040000 to 0x03a32c (0x5cd3 bytes) 000640 1787856811.583450 mapwrite 0x01d49e thru 0x022e4a (0x59ad bytes) 000641 1787856811.652088 read 0x02a9bb thru 0x038058 (0xd69e bytes) 000643 1787856811.672281 mapwrite 0x0156bd thru 0x01a75e (0x50a2 bytes) 000644 1787856811.814177 write 0x0296bd thru 0x033460 (0x9da4 bytes) 000645 1787856811.831274 trunc from 0x03a32d to 0x008968 (0x319c4 bytes) 000646 1787856811.852229 mapwrite 0x038bea thru 0x03ffff (0x7416 bytes) 000649 1787856811.929861 write 0x01fb5a thru 0x01fcfe (0x1a5 bytes) 000650 1787856811.937260 mapwrite 0x004f33 thru 0x00a891 (0x595f bytes) 000652 1787856811.998386 mapread 0x003c95 thru 0x010ebc (0xd228 bytes) 000653 1787856812.009229 MIRROR_RESYNC 4 mirrors 000654 1787856813.660408 MIRROR_RESYNC 4 mirrors 000656 1787856814.187027 read 0x004628 thru 0x00b62f (0x7008 bytes) 000660 1787856814.211453 write 0x00a005 thru 0x00cfaa (0x2fa6 bytes) 000661 1787856814.337399 trunc from 0x040000 to 0x002d94 (0x3d26b bytes) 000662 1787856814.363427 write 0x00e834 thru 0x00f9d6 (0x11a3 bytes) 000663 1787856814.365308 write 0x033528 thru 0x03a88a (0x7363 bytes) 000664 1787856814.367570 trunc from 0x03a88b to 0x02b79b (0xf0ef bytes) 000665 1787856814.384121 mapread 0x00f5b7 thru 0x0178c3 (0x830d bytes) 000667 1787856814.403720 mapread 0x01e17d thru 0x024ce9 (0x6b6d bytes) 000670 1787856814.423304 MIRROR_RESYNC 4 mirrors 000671 1787856816.066205 trunc from 0x02b79c to 0x014149 (0x17652 bytes) 000672 1787856816.166148 write 0x014c21 thru 0x01d2f3 (0x86d3 bytes) 000673 1787856816.172094 mapread 0x0181a1 thru 0x01d2f3 (0x5153 bytes) 000674 1787856816.179126 MIRROR_RESYNC 4 mirrors 000675 1787856817.661350 trunc from 0x01d2f4 to 0x029fa3 (0xccb0 bytes) 000676 1787856817.764448 mapread 0x00edad thru 0x01bdea (0xd03e bytes) 000677 1787856817.817920 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29fa4): Mirror 8: 0xd5858407 (stale) Mirror 9: 0xd5858407 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fa4): Mirror 8: 0xd5858407 (stale) Mirror 9: 0xd5858407 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fa4: skipped 000679 1787856818.134407 trunc from 0x029fa4 to 0x01967a (0x10929 bytes) 000680 1787856818.167587 read 0x0005b9 thru 0x00af68 (0xa9b0 bytes) 000682 1787856818.177582 write 0x01bbb0 thru 0x0232a2 (0x76f3 bytes) 000683 1787856818.184549 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x232a3): Mirror 8: 0x7a5dfd7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232a3): Mirror 8: 0x7a5dfd7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232a3: skipped 000687 1787856818.643809 mapread 0x010f76 thru 0x019ea9 (0x8f34 bytes) 000688 1787856818.663985 mapwrite 0x00e4fc thru 0x01ac62 (0xc767 bytes) 000690 1787856818.764621 read 0x016113 thru 0x0185b7 (0x24a5 bytes) 000692 1787856818.765637 read 0x01f25b thru 0x0232a2 (0x4048 bytes) 000696 1787856818.786012 trunc from 0x0232a3 to 0x03dafe (0x1a85c bytes) 000697 1787856818.818565 mapwrite 0x011fd2 thru 0x0154b9 (0x34e8 bytes) 000698 1787856818.896512 write 0x013d4b thru 0x023ab7 (0xfd6d bytes) 000699 1787856818.907976 MIRROR_RESYNC 2 mirrors 000700 1787856819.842528 trunc from 0x03daff to 0x021ae3 (0x1c01b bytes) 000703 1787856819.974908 trunc from 0x021ae4 to 0x035cb5 (0x141d2 bytes) 000708 1787856820.008192 trunc from 0x035cb6 to 0x01cf92 (0x18d23 bytes) 000709 1787856820.036163 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000710 1787856820.230851 write 0x0356a9 thru 0x0384b8 (0x2e10 bytes) 000713 1787856820.242541 write 0x02ad37 thru 0x039cf8 (0xefc2 bytes) 000714 1787856820.263337 mapread 0x0206ad thru 0x02e792 (0xe0e6 bytes) 000716 1787856820.310484 mapread 0x0244ce thru 0x02ac20 (0x6753 bytes) 000717 1787856820.312298 read 0x014bf5 thru 0x017f09 (0x3315 bytes) 000718 1787856820.329749 mapread 0x02f225 thru 0x03590e (0x66ea bytes) 000719 1787856820.331032 write 0x00fcb9 thru 0x00fea9 (0x1f1 bytes) 000720 1787856820.347294 trunc from 0x039cf9 to 0x02012f (0x19bc9 bytes) 000721 1787856820.369011 mapread 0x01c319 thru 0x02012f (0x3e17 bytes) 000722 1787856820.371260 read 0x01ef94 thru 0x02012f (0x119c bytes) 000724 1787856820.373754 mapread 0x006a8f thru 0x0086bc (0x1c2e bytes) 000725 1787856820.390860 read 0x00ed9f thru 0x016b55 (0x7db7 bytes) 000726 1787856820.391985 mapread 0x011a76 thru 0x01a77f (0x8d0a bytes) 000728 1787856820.394809 mapwrite 0x0374c8 thru 0x03acb7 (0x37f0 bytes) 000735 1787856820.507782 trunc from 0x03acb8 to 0x0055c5 (0x356f2 bytes) 000736 1787856820.534336 read 0x002434 thru 0x0046bb (0x2288 bytes) 000738 1787856820.537136 write 0x016af9 thru 0x01fa3f (0x8f47 bytes) 000740 1787856820.542543 write 0x02748e thru 0x029923 (0x2496 bytes) 000741 1787856820.544948 mapread 0x00f927 thru 0x01f7bc (0xfe96 bytes) 000742 1787856820.554720 write 0x000165 thru 0x00456e (0x440a bytes) 000745 1787856820.559031 mapwrite 0x01c3ef thru 0x01f18c (0x2d9e bytes) 000746 1787856820.628593 trunc from 0x029924 to 0x00ad1c (0x1ec07 bytes) 000748 1787856820.654404 write 0x003c13 thru 0x007153 (0x3541 bytes) 000752 1787856820.662858 write 0x033d95 thru 0x03f70e (0xb97a bytes) 000753 1787856820.665771 trunc from 0x03f70f to 0x03023f (0xf4cf bytes) truncating to largest ever: 0x3fc8c 000754 1787856820.694800 trunc from 0x030240 to 0x03fc8b (0xfa4c bytes) 000756 1787856820.713458 mapread 0x007670 thru 0x0123ee (0xad7f bytes) 000757 1787856820.801600 write 0x00ba41 thru 0x0152bb (0x987b bytes) 000759 1787856820.809361 trunc from 0x03fc8c to 0x02c0e7 (0x13ba4 bytes) 000763 1787856820.822738 trunc from 0x02c0e8 to 0x00d79e (0x1e949 bytes) 000764 1787856820.845119 write 0x02a9f2 thru 0x03477b (0x9d8a bytes) 000766 1787856820.852104 write 0x02c8aa thru 0x035155 (0x88ac bytes) 000769 1787856820.856619 read 0x033355 thru 0x035155 (0x1e01 bytes) 000770 1787856820.857545 write 0x031aa2 thru 0x039a6b (0x7fca bytes) 000773 1787856820.861220 write 0x030519 thru 0x03a0e2 (0x9bca bytes) 000774 1787856820.869657 mapwrite 0x03c5bb thru 0x03ffff (0x3a45 bytes) 000776 1787856820.978604 write 0x0350bf thru 0x036718 (0x165a bytes) 000778 1787856820.981937 write 0x03cd85 thru 0x03ffff (0x327b bytes) 000780 1787856820.984572 mapread 0x026487 thru 0x031393 (0xaf0d bytes) 000782 1787856820.992327 mapread 0x02016b thru 0x023f8e (0x3e24 bytes) 000784 1787856820.995161 read 0x01f8ef thru 0x02c9e9 (0xd0fb bytes) 000785 1787856820.995998 trunc from 0x040000 to 0x01a3b1 (0x25c4e bytes) 000790 1787856821.026922 mapwrite 0x00a6d3 thru 0x00f9f3 (0x5321 bytes) 000791 1787856821.087989 mapwrite 0x03cfca thru 0x03ffff (0x3036 bytes) 000795 1787856821.173154 trunc from 0x040000 to 0x030688 (0xf977 bytes) 000796 1787856821.186183 mapwrite 0x01ae56 thru 0x01f39c (0x4547 bytes) 000799 1787856821.267896 write 0x017f26 thru 0x0245db (0xc6b6 bytes) 000800 1787856821.278863 MIRROR_EXTEND to 2 mirrors 000802 1787856822.153048 write 0x00ec08 thru 0x01e794 (0xfb8d bytes) 000804 1787856822.286628 write 0x02cc77 thru 0x031037 (0x43c1 bytes) 000805 1787856822.301732 trunc from 0x031038 to 0x035d40 (0x4d09 bytes) 000806 1787856822.358428 MIRROR_RESYNC 2 mirrors 000807 1787856823.347031 trunc from 0x035d41 to 0x03be87 (0x6147 bytes) 000809 1787856823.445338 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000810 1787856823.605602 read 0x02a1cc thru 0x02ea81 (0x48b6 bytes) 000811 1787856823.616277 read 0x01f8b8 thru 0x02129c (0x19e5 bytes) 000813 1787856823.633719 trunc from 0x03be88 to 0x0050e0 (0x36da7 bytes) 000814 1787856823.651089 write 0x01aeb4 thru 0x01dc52 (0x2d9f bytes) 000817 1787856823.654855 mapread 0x0007b9 thru 0x0012e4 (0xb2c bytes) 000819 1787856823.657131 mapwrite 0x034167 thru 0x037592 (0x342c bytes) 000820 1787856823.781855 trunc from 0x037593 to 0x03d9f6 (0x6464 bytes) 000823 1787856823.796984 read 0x001568 thru 0x00fcb9 (0xe752 bytes) 000824 1787856823.815579 mapread 0x02b819 thru 0x034240 (0x8a28 bytes) 000825 1787856823.817032 trunc from 0x03d9f7 to 0x0356c0 (0x8336 bytes) 000826 1787856823.842920 write 0x02e2fb thru 0x03498b (0x6691 bytes) 000827 1787856823.845472 mapwrite 0x02e38a thru 0x038217 (0x9e8e bytes) 000828 1787856823.934897 write 0x031efa thru 0x035480 (0x3587 bytes) 000830 1787856823.936543 trunc from 0x038218 to 0x03c881 (0x466a bytes) 000831 1787856823.959382 trunc from 0x03c882 to 0x00bd92 (0x30aef bytes) 000832 1787856823.981092 read 0x001b77 thru 0x00bd92 (0xa21c bytes) 000836 1787856823.985058 trunc from 0x00bd93 to 0x001c64 (0xa12e bytes) 000837 1787856824.012118 mapwrite 0x004b63 thru 0x008184 (0x3622 bytes) 000838 1787856824.139235 read 0x007ea7 thru 0x008184 (0x2de bytes) 000843 1787856824.146226 mapread 0x00675f thru 0x008184 (0x1a26 bytes) 000844 1787856824.156224 mapwrite 0x014067 thru 0x014850 (0x7ea bytes) 000846 1787856824.252476 read 0x000baa thru 0x008bf2 (0x8049 bytes) 000848 1787856824.253719 mapwrite 0x02327a thru 0x026013 (0x2d9a bytes) 000849 1787856824.331583 MIRROR_EXTEND to 2 mirrors 000852 1787856825.286539 trunc from 0x026014 to 0x005115 (0x20efe bytes) 000853 1787856825.443861 mapread 0x00031b thru 0x005115 (0x4dfb bytes) 000854 1787856825.476395 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000858 1787856825.677885 write 0x03f038 thru 0x03ffff (0xfc8 bytes) 000863 1787856825.684263 trunc from 0x040000 to 0x03d406 (0x2bf9 bytes) 000866 1787856825.699390 read 0x01835e thru 0x025e4f (0xdaf2 bytes) 000874 1787856825.714879 mapwrite 0x0142ab thru 0x01c68e (0x83e4 bytes) 000876 1787856825.799365 write 0x0192e7 thru 0x0261e5 (0xceff bytes) 000879 1787856825.809846 trunc from 0x03d407 to 0x001628 (0x3bdde bytes) 000883 1787856825.831853 read 0x000b75 thru 0x001628 (0xab4 bytes) 000884 1787856825.840225 mapread 0x00140b thru 0x001628 (0x21e bytes) 000885 1787856825.841969 read 0x00136a thru 0x001628 (0x2bf bytes) 000888 1787856825.844184 read 0x00002d thru 0x001628 (0x15fc bytes) 000890 1787856825.845981 write 0x0391ae thru 0x03ffff (0x6e52 bytes) 000893 1787856825.850946 mapwrite 0x007d29 thru 0x011f48 (0xa220 bytes) 000894 1787856825.962567 trunc from 0x040000 to 0x02ce76 (0x13189 bytes) 000895 1787856826.003786 trunc from 0x02ce77 to 0x00cc30 (0x20246 bytes) 000897 1787856826.049585 write 0x013590 thru 0x01c35d (0x8dce bytes) 000898 1787856826.051964 write 0x008985 thru 0x00dcd2 (0x534e bytes) 000901 1787856826.059532 trunc from 0x01c35e to 0x02cf15 (0x10bb8 bytes) 000902 1787856826.096034 trunc from 0x02cf16 to 0x008e1b (0x240fa bytes) 000905 1787856826.122216 write 0x00abbc thru 0x016210 (0xb655 bytes) 000906 1787856826.131871 mapread 0x00387b thru 0x006a1b (0x31a1 bytes) 000907 1787856826.133136 mapread 0x0107d2 thru 0x0126b4 (0x1ee3 bytes) 000910 1787856826.155487 trunc from 0x016211 to 0x02f56d (0x1935d bytes) 000915 1787856826.178958 write 0x00a004 thru 0x00af57 (0xf54 bytes) 000916 1787856826.180141 mapwrite 0x02f8d5 thru 0x03f3cf (0xfafb bytes) 000917 1787856826.393800 read 0x0055dd thru 0x010dae (0xb7d2 bytes) 000921 1787856826.419676 write 0x01eb8e thru 0x024f62 (0x63d5 bytes) 000922 1787856826.421924 write 0x010a03 thru 0x01b483 (0xaa81 bytes) 000924 1787856826.425853 write 0x02f8dc thru 0x03d0a3 (0xd7c8 bytes) 000925 1787856826.433747 trunc from 0x03f3d0 to 0x0274ea (0x17ee5 bytes) 000926 1787856826.475610 write 0x035242 thru 0x03ab3d (0x58fc bytes) 000927 1787856826.478292 write 0x0177e6 thru 0x0254d5 (0xdcf0 bytes) 000931 1787856826.483845 read 0x023ba0 thru 0x02471c (0xb7d bytes) 000935 1787856826.486688 mapread 0x038d3e thru 0x03ab3d (0x1e00 bytes) 000937 1787856826.487896 mapread 0x027574 thru 0x02aedc (0x3969 bytes) 000939 1787856826.511031 read 0x03a807 thru 0x03ab3d (0x337 bytes) 000942 1787856826.512486 mapread 0x0253ed thru 0x02a538 (0x514c bytes) 000943 1787856826.513346 mapread 0x021953 thru 0x02f5de (0xdc8c bytes) 000944 1787856826.514169 write 0x0220d6 thru 0x0239ca (0x18f5 bytes) 000945 1787856826.514907 read 0x022147 thru 0x028012 (0x5ecc bytes) 000946 1787856826.520682 MIRROR_EXTEND to 2 mirrors 000949 1787856827.763634 mapwrite 0x039e75 thru 0x03ffff (0x618b bytes) 000951 1787856827.984219 trunc from 0x040000 to 0x023434 (0x1cbcb bytes) 000952 1787856828.024966 read 0x000351 thru 0x00869f (0x834f bytes) 000953 1787856828.049078 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000958 1787856828.270631 write 0x00279a thru 0x007f3d (0x57a4 bytes) 000960 1787856828.299928 write 0x02c08e thru 0x033dbd (0x7d30 bytes) 000962 1787856828.306483 mapread 0x0004b4 thru 0x0064de (0x602b bytes) 000964 1787856828.337440 read 0x020d12 thru 0x024924 (0x3c13 bytes) 000965 1787856828.356260 read 0x03216d thru 0x033dbd (0x1c51 bytes) 000966 1787856828.359179 trunc from 0x033dbe to 0x029d77 (0xa046 bytes) 000967 1787856828.389212 read 0x0011f2 thru 0x008677 (0x7486 bytes) 000968 1787856828.410127 read 0x00ee35 thru 0x01da73 (0xec3f bytes) 000969 1787856828.411019 trunc from 0x029d78 to 0x01fe3e (0x9f39 bytes) 000971 1787856828.443662 read 0x01db80 thru 0x01fe3e (0x22bf bytes) 000973 1787856828.445161 mapwrite 0x014626 thru 0x016634 (0x200f bytes) 000975 1787856828.525114 mapwrite 0x01f2dc thru 0x023c2e (0x4953 bytes) 000977 1787856828.630855 mapread 0x0143ba thru 0x022721 (0xe368 bytes) 000980 1787856828.639092 read 0x003bbc thru 0x004432 (0x877 bytes) 000982 1787856828.642750 mapwrite 0x00116b thru 0x005d58 (0x4bee bytes) 000983 1787856828.722490 MIRROR_EXTEND to 2 mirrors 000984 1787856829.529840 read 0x0124a4 thru 0x01e6b9 (0xc216 bytes) 000986 1787856829.541985 write 0x0264cc thru 0x02e57f (0x80b4 bytes) 000988 1787856829.627074 mapwrite 0x0384cf thru 0x03ffff (0x7b31 bytes) 000991 1787856829.746222 mapwrite 0x00d90c thru 0x00e815 (0xf0a bytes) 000992 1787856829.826572 read 0x010795 thru 0x01453a (0x3da6 bytes) 000995 1787856829.827992 write 0x02bd99 thru 0x03690f (0xab77 bytes) 001000 1787856829.834963 trunc from 0x040000 to 0x033aac (0xc553 bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 14:53:54 (1787856834) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 14:53:56 (1787856836) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265522 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164372 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 14:54:05 (1787856845) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre2 Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2040864 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7678144 .. failed Extending file size to 3237888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5070944 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8498592 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6339456 .. Extending file size to 8583616 .. Extending file size to 6832192 .. Extending file size to 8258016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8486848 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8794528 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8861280 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8938880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9252864 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58596 58597 58598 58599 58603 192.168.206.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre2 (opts:) 192.168.206.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:55:24 (1787856924) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre2 Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5388448 .. Extending file size to 2991456 .. done Extending file size to 5658400 .. Extending file size to 6496096 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8451808 .. Extending file size to 8377504 .. Extending file size to 8675008 .. done Extending file size to 9051488 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9251968 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9116416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60078 60079 60081 60082 60085 done 192.168.206.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre2 (opts:) 192.168.206.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:56:41 (1787857001) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102598 s, 20.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.413961 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x48749fae (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:57:00 (1787857020) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112634 s, 18.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:57:08 (1787857028) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1db:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:57:16 (1787857036) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202392 s, 10.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:57:23 (1787857043) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220006 s, 9.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:57:31 (1787857051) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.366784 s, 5.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:58:07 (1787857087) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg649-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122229 s, 8.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:59:04 (1787857144) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg649-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174329 s, 12.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:00:04 (1787857204) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg649-server mds-ost sync done. Creating new pool test_204f oleg649-server: Pool lustre.test_204f created Adding targets to pool oleg649-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg649-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187448 s, 11.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg649-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg649-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg649-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:01:23 (1787857283) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:01:30 (1787857290) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:01:36 (1787857296) oleg649-server: Pool lustre.test_206 created Adding targets to pool oleg649-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg649-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg649-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg649-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg649-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:02:01 (1787857321) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094057 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558058 s, 18.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x56bd5746 Mirror 3: 0x56bd5746 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:02:13 (1787857333) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:02:17 (1787857337) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:02:21 (1787857341) Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.168998 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xab0000 Read this (4 bytes): BBBB 192.168.206.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg649-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:02:57 (1787857377) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0257414 s, 318 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:03:04 (1787857384) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609103 s, 17.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834432 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 15:03:06.000000000 -0400 Modify: 2026-08-27 15:03:06.000000000 -0400 Change: 2026-08-27 15:03:08.000000000 -0400 Birth: 2026-08-27 15:03:06.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:03:13 (1787857393) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:03:17 (1787857397) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.471649 s, 86.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0493795 s, 82.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 15:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 15:03 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:03:25 (1787857405) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:03:34 (1787857414) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1793 sec =========== 15:03:44 (1787857424) === sanity-flr: start cleanup 15:03:45 (1787857425) === === sanity-flr: finish cleanup 15:03:52 (1787857432) ===