-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 10:44:52 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg130-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:45:16 (1787841916) === oleg130-client.virtnet: executing check_config_client /mnt/lustre oleg130-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg130-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90b0d1822000.idle_timeout=debug osc.lustre-OST0001-osc-ffff90b0d1822000.idle_timeout=debug disable quota as required oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:45:58 (1787841958) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 10:46:01 (1787841961) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:46:22 (1787841982) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:46:25 (1787841985) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:46:30 (1787841990) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:46:58 (1787842018) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:47:03 (1787842023) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:47:07 (1787842027) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90b0d1822000.stats=clear osc.lustre-OST0001-osc-ffff90b0d1822000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:47:15 (1787842035) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:47:27 (1787842047) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:47:37 (1787842057) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:47:41 (1787842061) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:47:50 (1787842070) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 10:48:01 (1787842081) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:48:09 (1787842089) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:48:13 (1787842093) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:48:20 (1787842100) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:48:32 (1787842112) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:48:41 (1787842121) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.431707 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130457 s, 8.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:48:51 (1787842131) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.107338 s, 29.3 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:49:00 (1787842140) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:49:12 (1787842152) Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:50:32 (1787842232) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.445995 s, 14.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:51:17 (1787842277) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:52:38 (1787842358) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.42153 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 10:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.617s user 0m0.000s sys 0m0.233s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.437s user 0m0.000s sys 0m0.214s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.398s user 0m0.001s sys 0m0.186s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90b0d1822000.ost_server_uuid in FULL state after 0 sec PASS 33b (112s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:54:31 (1787842471) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:54:36 (1787842476) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:54:41 (1787842481) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:54:45 (1787842485) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 10:54:55 (1787842495) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.682393 s, 18.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:55:10 (1787842510) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743781 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717639 s, 14.6 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:55:19 (1787842519) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:55:29 (1787842529) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00191492 s, 535 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176666 s, 580 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303351 s, 338 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134565 s, 761 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 10:55:41 (1787842541) create test files with size 15737485 .. 604b7ab7dfdb2996e5ce1b9f8aac5603 - 0631a06ba1b230ff2a10158bcabb0203 - 394f8d07e872d6e2d430df4b5c6c6056 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 10:56:09 (1787842569) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:56:24 (1787842584) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:56:31 (1787842591) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107125 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024959 s, 42.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 10:56:41 (1787842601) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246895 s, 8.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.484757 s, 8.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252772 s, 8.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 10:56:58 (1787842618) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:57:01 (1787842621) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:57:05 (1787842625) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04413 s, 23.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 10:57:13 (1787842633) llite.lustre-ffff90b0d06f6000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 10:57:20 (1787842640) llite.lustre-ffff90b0d06f6000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662635 s, 15.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642568 s, 16.3 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 10:57:30 (1787842650) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:57:33 (1787842653) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:57:44 (1787842664) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.404391 s, 25.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:57:58 (1787842678) llite.lustre-ffff90b0d0059800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296795 s, 35.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff90b0c4e5a000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:58:12 (1787842692) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:58:54 (1787842734) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:59:34 (1787842774) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:59:37 (1787842777) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.248616 s, 12.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.17777 s, 17.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdda414e (nosync,stale) Mirror 2: 0xdda414e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 22b06a34dbebf2b8cf3fa06f98c5ad58 - mirror 1 checksum: 6bd1c144be11eb419ec044bec4603e4a - mirror 2 checksum: 22b06a34dbebf2b8cf3fa06f98c5ad58 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:59:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:59:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 22b06a34dbebf2b8cf3fa06f98c5ad58 - mirror 1 checksum: 6bd1c144be11eb419ec044bec4603e4a - mirror 2 checksum: 6bd1c144be11eb419ec044bec4603e4a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:59:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:59:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:59:51 (1787842791) Failing mds1 on oleg130-server Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 10:59:57 (1787842797) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server Failover mds1 to oleg130-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:21 (1787842821) targets are mounted 11:00:21 (1787842821) facet_failover done oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341449 s, 30.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:00:39 (1787842839) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:00:42 (1787842842) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:00:45 (1787842845) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0207144 s, 791 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:00:54 (1787842854) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:00:57 (1787842857) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130547 s, 314 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0870383 s, 47.1 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:01:06 (1787842866) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:01:29 (1787842889) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:01:58 (1787842918) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:02:24 (1787842944) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:02:50 (1787842970) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:02:58 (1787842978) mirror create pid 55595 mirror split pid 55596 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55595 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55596 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 11:04:06 (1787843046) Seed set to 3049 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x2c5af 000002 1787843049.479658 trunc from 00000000 to 0x02c5ae (0x2c5af bytes) 000004 1787843049.512182 mapread 0x011fd4 thru 0x021881 (0xf8ae bytes) 000006 1787843049.576419 mapread 0x01f0a7 thru 0x026dc8 (0x7d22 bytes) truncating to largest ever: 0x39243 000007 1787843049.600743 trunc from 0x02c5af to 0x039242 (0xcc94 bytes) 000008 1787843049.635942 read 0x02a607 thru 0x031e0a (0x7804 bytes) 000009 1787843049.645995 read 0x006ed5 thru 0x00f47e (0x85aa bytes) 000013 1787843049.672401 mapwrite 0x00537f thru 0x00cf5d (0x7bdf bytes) 000014 1787843049.729504 trunc from 0x039243 to 0x02ea32 (0xa810 bytes) 000015 1787843049.752326 read 0x022364 thru 0x0295c7 (0x7264 bytes) 000018 1787843049.756124 mapwrite 0x0224c9 thru 0x02d30f (0xae47 bytes) 000021 1787843049.801052 trunc from 0x02ea33 to 0x011632 (0x1d400 bytes) 000022 1787843049.818071 mapread 0x0037b9 thru 0x0098e2 (0x612a bytes) 000028 1787843049.824795 mapread 0x00f1ee thru 0x011632 (0x2445 bytes) 000029 1787843049.825947 mapwrite 0x00cc4d thru 0x0128c8 (0x5c7c bytes) 000030 1787843049.902688 trunc from 0x0128c9 to 0x02ea5f (0x1c197 bytes) 000031 1787843049.920256 mapwrite 0x0381be thru 0x03ffff (0x7e42 bytes) 000032 1787843049.993399 write 0x015195 thru 0x01be23 (0x6c8f bytes) 000034 1787843049.995030 read 0x03f16d thru 0x03ffff (0xe93 bytes) 000035 1787843049.995578 write 0x038770 thru 0x03ffff (0x7890 bytes) truncating to largest ever: 0x3f0d8 000036 1787843049.997794 trunc from 0x040000 to 0x03f0d7 (0xf28 bytes) 000039 1787843050.018765 mapwrite 0x02fe40 thru 0x035213 (0x53d4 bytes) 000044 1787843050.068164 trunc from 0x03f0d8 to 0x01d390 (0x21d47 bytes) 000045 1787843050.088471 read 0x01d075 thru 0x01d390 (0x31c bytes) 000048 1787843050.091260 mapwrite 0x001516 thru 0x00a9f7 (0x94e2 bytes) 000049 1787843050.140706 mapread 0x00830e thru 0x017224 (0xef17 bytes) 000050 1787843050.141874 mapwrite 0x039aca thru 0x03ffff (0x6536 bytes) 000051 1787843050.238874 write 0x008e07 thru 0x010285 (0x747f bytes) 000054 1787843050.243095 mapread 0x01ea83 thru 0x0237e8 (0x4d66 bytes) 000056 1787843050.245308 trunc from 0x040000 to 0x011ab6 (0x2e549 bytes) 000057 1787843050.261064 write 0x02bdc4 thru 0x03b7ea (0xfa27 bytes) 000059 1787843050.263328 trunc from 0x03b7eb to 0x02c336 (0xf4b4 bytes) 000060 1787843050.283409 write 0x022572 thru 0x031b87 (0xf616 bytes) 000061 1787843050.289899 write 0x02876d thru 0x02a7a1 (0x2035 bytes) 000062 1787843050.291005 read 0x018a76 thru 0x01cc35 (0x41c0 bytes) 000063 1787843050.305455 trunc from 0x031b88 to 0x001731 (0x30456 bytes) 000064 1787843050.321205 read 0x000e50 thru 0x001731 (0x8e2 bytes) 000065 1787843050.322139 mapwrite 0x0303da thru 0x039ff4 (0x9c1b bytes) 000066 1787843050.422750 mapwrite 0x015320 thru 0x01d06c (0x7d4d bytes) 000070 1787843050.479343 mapwrite 0x03c3c3 thru 0x03d0c0 (0xcfe bytes) 000071 1787843050.536376 mapread 0x0053b0 thru 0x007fb2 (0x2c03 bytes) 000072 1787843050.538037 read 0x011a26 thru 0x0218be (0xfe99 bytes) 000074 1787843050.539468 write 0x009701 thru 0x00fd59 (0x6659 bytes) 000075 1787843050.544613 mapwrite 0x026d5d thru 0x033a39 (0xccdd bytes) 000077 1787843050.594923 trunc from 0x03d0c1 to 0x0122b3 (0x2ae0d bytes) 000078 1787843050.613211 mapread 0x004798 thru 0x009fd2 (0x583b bytes) 000079 1787843050.623029 mapread 0x00d931 thru 0x0122b3 (0x4983 bytes) 000080 1787843050.624894 trunc from 0x0122b4 to 0x01049f (0x1e14 bytes) 000082 1787843050.680412 write 0x02c4f1 thru 0x039432 (0xcf42 bytes) 000083 1787843050.682106 mapwrite 0x033ef1 thru 0x03ebdf (0xacef bytes) 000084 1787843050.770500 write 0x0343ae thru 0x03719e (0x2df1 bytes) 000085 1787843050.773659 mapwrite 0x03133d thru 0x03fbb0 (0xe874 bytes) 000086 1787843050.847291 trunc from 0x03fbb1 to 0x01d979 (0x22237 bytes) 000087 1787843050.874404 MIRROR_EXTEND to 2 mirrors 000088 1787843052.055083 mapwrite 0x037118 thru 0x03dfa8 (0x6e91 bytes) 000089 1787843052.279890 mapwrite 0x017112 thru 0x018ffd (0x1eec bytes) 000090 1787843052.357302 write 0x026443 thru 0x02e913 (0x84d1 bytes) 000091 1787843052.370237 mapwrite 0x03e668 thru 0x03ffff (0x1998 bytes) 000092 1787843052.468395 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000093 1787843052.719087 mapwrite 0x03cb9f thru 0x03ffff (0x3461 bytes) 000094 1787843052.831134 write 0x037538 thru 0x03beb1 (0x497a bytes) 000095 1787843052.832234 mapwrite 0x0201c2 thru 0x02b5aa (0xb3e9 bytes) 000096 1787843052.919821 write 0x010bc9 thru 0x01ae88 (0xa2c0 bytes) 000097 1787843052.928861 mapread 0x03299e thru 0x03ddad (0xb410 bytes) 000098 1787843052.939720 write 0x02489d thru 0x02d0c0 (0x8824 bytes) 000099 1787843052.941209 write 0x025065 thru 0x026550 (0x14ec bytes) 000102 1787843052.945052 read 0x034b67 thru 0x037942 (0x2ddc bytes) 000103 1787843052.945934 mapwrite 0x006bfc thru 0x00c9cb (0x5dd0 bytes) 000104 1787843053.052959 mapread 0x0200b5 thru 0x02e710 (0xe65c bytes) 000105 1787843053.056360 write 0x01796d thru 0x02503e (0xd6d2 bytes) 000106 1787843053.059992 trunc from 0x040000 to 0x01deec (0x22113 bytes) 000107 1787843053.088491 MIRROR_EXTEND to 2 mirrors 000108 1787843054.126711 mapread 0x00eec5 thru 0x01235a (0x3496 bytes) 000109 1787843054.150196 trunc from 0x01deed to 0x00b6dd (0x1280f bytes) 000111 1787843054.236280 mapwrite 0x025c06 thru 0x0333a9 (0xd7a4 bytes) 000112 1787843054.377048 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3f40d 000114 1787843055.298590 trunc from 0x0333aa to 0x03f40c (0xc063 bytes) 000116 1787843055.469675 trunc from 0x03f40d to 0x034eec (0xa520 bytes) 000117 1787843055.500896 mapread 0x00c137 thru 0x0133be (0x7288 bytes) 000121 1787843055.544691 write 0x009564 thru 0x01701b (0xdab8 bytes) 000125 1787843055.565057 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34eed): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34eed): Mirror 2: 0xcba3953f (stale) CRC-32 checksum value for chunk [0, 0x34eed): Mirror 1: 0x36bce346 Mirror 3: 0x36bce346 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34eed: skipped 000127 1787843057.162702 write 0x028563 thru 0x02bc5c (0x36fa bytes) 000128 1787843057.260587 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34eed): Mirror 3: 0x36bce346 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34eed): Mirror 3: 0x36bce346 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34eed: skipped 000130 1787843057.582310 write 0x01d07c thru 0x0270dd (0xa062 bytes) 000131 1787843057.594744 write 0x034cb9 thru 0x03e824 (0x9b6c bytes) 000134 1787843057.602357 mapread 0x01d2df thru 0x020d03 (0x3a25 bytes) 000135 1787843057.607728 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000136 1787843057.753041 mapwrite 0x01ed70 thru 0x027991 (0x8c22 bytes) 000138 1787843057.816539 mapwrite 0x00f8f5 thru 0x0108ab (0xfb7 bytes) 000139 1787843057.865767 write 0x031251 thru 0x03559f (0x434f bytes) 000140 1787843057.874764 write 0x033261 thru 0x03ffff (0xcd9f bytes) 000144 1787843057.883567 read 0x0327ff thru 0x03753d (0x4d3f bytes) 000147 1787843057.886016 read 0x0118e8 thru 0x01ca3c (0xb155 bytes) 000148 1787843057.901328 MIRROR_EXTEND to 2 mirrors 000149 1787843058.700362 read 0x01c8cc thru 0x020908 (0x403d bytes) 000150 1787843058.708568 mapread 0x02a59d thru 0x0361bd (0xbc21 bytes) 000151 1787843058.734738 mapwrite 0x02f933 thru 0x03806a (0x8738 bytes) 000153 1787843058.905622 read 0x021bb6 thru 0x0311a0 (0xf5eb bytes) 000155 1787843058.923888 trunc from 0x040000 to 0x035541 (0xaabe bytes) 000157 1787843058.986629 mapread 0x009551 thru 0x00bf37 (0x29e7 bytes) 000158 1787843058.987638 mapwrite 0x00b6c4 thru 0x014e53 (0x9790 bytes) 000159 1787843059.145172 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000160 1787843059.430796 write 0x016c34 thru 0x01c128 (0x54f5 bytes) 000164 1787843059.450309 MIRROR_EXTEND to 2 mirrors 000165 1787843060.211772 read 0x02b810 thru 0x02e143 (0x2934 bytes) 000166 1787843060.217940 mapwrite 0x035d85 thru 0x03a9de (0x4c5a bytes) 000167 1787843060.415250 mapwrite 0x0311e6 thru 0x039c6d (0x8a88 bytes) 000168 1787843060.461659 mapwrite 0x036759 thru 0x03cbfb (0x64a3 bytes) 000169 1787843060.538935 mapread 0x02373c thru 0x033437 (0xfcfc bytes) 000170 1787843060.556638 write 0x01fa67 thru 0x023fe7 (0x4581 bytes) 000172 1787843060.571618 trunc from 0x03cbfc to 0x01a687 (0x22574 bytes) 000173 1787843060.596160 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1a688): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a688): Mirror 2: 0xf924ecfa (stale) CRC-32 checksum value for chunk [0, 0x1a688): Mirror 1: 0xf924ecfa Mirror 3: 0xf924ecfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a688: skipped 000174 1787843061.404314 mapread 0x00e2b7 thru 0x012bed (0x4937 bytes) 000175 1787843061.427966 trunc from 0x01a688 to 0x035b77 (0x1b4f0 bytes) 000176 1787843061.530963 trunc from 0x035b78 to 0x0336bc (0x24bb bytes) 000177 1787843061.555078 write 0x00de8d thru 0x016a86 (0x8bfa bytes) 000179 1787843061.570080 read 0x0229b1 thru 0x02a8a5 (0x7ef5 bytes) 000180 1787843061.581894 read 0x007eeb thru 0x00c060 (0x4176 bytes) 000182 1787843061.596698 mapwrite 0x01bd32 thru 0x028fa3 (0xd272 bytes) 000183 1787843061.665871 trunc from 0x0336bd to 0x01d086 (0x16636 bytes) 000184 1787843061.691611 read 0x003e56 thru 0x007096 (0x3241 bytes) 000186 1787843061.692996 mapwrite 0x01795c thru 0x01afb1 (0x3656 bytes) 000187 1787843061.746867 mapwrite 0x02b0da thru 0x036be4 (0xbb0b bytes) 000189 1787843061.903472 read 0x033f44 thru 0x036be4 (0x2ca1 bytes) 000190 1787843061.904667 trunc from 0x036be5 to 0x020cb1 (0x15f33 bytes) 000192 1787843061.931405 mapwrite 0x011a92 thru 0x01cee3 (0xb452 bytes) 000194 1787843062.002913 mapwrite 0x0194a7 thru 0x01ef5a (0x5ab4 bytes) 000196 1787843062.073690 write 0x022826 thru 0x024f4c (0x2727 bytes) 000198 1787843062.077774 read 0x001590 thru 0x00ca80 (0xb4f1 bytes) 000202 1787843062.081381 mapwrite 0x018a2a thru 0x01b02e (0x2605 bytes) 000203 1787843062.138917 read 0x00902c thru 0x012c83 (0x9c58 bytes) 000205 1787843062.141199 read 0x003e9e thru 0x00e768 (0xa8cb bytes) 000206 1787843062.143398 read 0x001710 thru 0x00bb28 (0xa419 bytes) 000208 1787843062.145180 mapread 0x00d8d6 thru 0x014181 (0x68ac bytes) 000212 1787843062.148649 mapread 0x0191af thru 0x01db2e (0x4980 bytes) 000213 1787843062.150009 mapwrite 0x002da6 thru 0x00e309 (0xb564 bytes) 000215 1787843062.201618 mapwrite 0x0236fe thru 0x02e04d (0xa950 bytes) 000216 1787843062.287824 write 0x01fdab thru 0x026469 (0x66bf bytes) 000219 1787843062.297263 MIRROR_RESYNC 3 mirrors 000220 1787843063.152476 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000221 1787843063.466740 trunc from 0x02e04e to 0x01e6ef (0xf95e bytes) 000222 1787843063.608502 mapread 0x01b387 thru 0x01e6ef (0x3369 bytes) 000223 1787843063.627377 read 0x001603 thru 0x011548 (0xff46 bytes) 000233 1787843063.651394 MIRROR_RESYNC 2 mirrors 000234 1787843064.404419 read 0x002a21 thru 0x010655 (0xdc35 bytes) 000235 1787843064.411944 mapread 0x005148 thru 0x009d46 (0x4bff bytes) 000236 1787843064.414152 mapwrite 0x004958 thru 0x013254 (0xe8fd bytes) 000237 1787843064.655021 mapwrite 0x0285b2 thru 0x030171 (0x7bc0 bytes) 000239 1787843064.854910 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30172): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30172): Mirror 3: 0xcfe0be09 (stale) CRC-32 checksum value for chunk [0, 0x30172): Mirror 1: 0xd7985d15 Mirror 4: 0xd7985d15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30172: skipped 000240 1787843065.994151 read 0x017f92 thru 0x018b35 (0xba4 bytes) 000241 1787843066.002630 write 0x02039a thru 0x020504 (0x16b bytes) 000243 1787843066.076109 mapwrite 0x01112b thru 0x018aa0 (0x7976 bytes) 000244 1787843066.152702 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30172): Mirror 4: 0xd7985d15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30172): Mirror 4: 0xd7985d15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30172: skipped 000245 1787843066.522104 write 0x0064af thru 0x009b8d (0x36df bytes) 000246 1787843066.535769 mapread 0x00f525 thru 0x018250 (0x8d2c bytes) 000247 1787843066.585340 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30172): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30172): Mirror 4: 0xd7985d15 (stale) CRC-32 checksum value for chunk [0, 0x30172): Mirror 1: 0x8f2507cd Mirror 5: 0x8f2507cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30172: skipped 000251 1787843067.955770 mapwrite 0x01a50d thru 0x026cb4 (0xc7a8 bytes) 000252 1787843068.261695 read 0x008302 thru 0x012f5b (0xac5a bytes) 000253 1787843068.272465 write 0x018d5c thru 0x02359c (0xa841 bytes) 000255 1787843068.282043 mapwrite 0x03d5a2 thru 0x03ff15 (0x2974 bytes) 000260 1787843068.399555 mapread 0x026dba thru 0x034815 (0xda5c bytes) 000261 1787843068.412598 write 0x02ff3b thru 0x03a00a (0xa0d0 bytes) 000262 1787843068.415585 trunc from 0x03ff16 to 0x02e83f (0x116d6 bytes) 000266 1787843068.454691 MIRROR_RESYNC 3 mirrors 000267 1787843069.347654 write 0x01a744 thru 0x02954d (0xee0a bytes) 000271 1787843069.434297 mapread 0x015305 thru 0x019f7f (0x4c7b bytes) 000272 1787843069.456469 MIRROR_RESYNC 3 mirrors 000273 1787843070.595026 MIRROR_RESYNC 3 mirrors 000274 1787843071.090185 mapwrite 0x01c3ea thru 0x0294fe (0xd115 bytes) 000275 1787843071.273271 trunc from 0x02e840 to 0x0048ee (0x29f51 bytes) 000277 1787843071.338928 mapwrite 0x01c36f thru 0x027a23 (0xb6b5 bytes) 000278 1787843071.475570 mapread 0x021ffe thru 0x027a23 (0x5a26 bytes) 000279 1787843071.477648 mapread 0x00c644 thru 0x00df31 (0x18ee bytes) 000281 1787843071.540433 write 0x038f1d thru 0x03b1b7 (0x229b bytes) 000283 1787843071.546207 mapwrite 0x0370b3 thru 0x037ff0 (0xf3e bytes) 000284 1787843071.643057 trunc from 0x03b1b8 to 0x008ed6 (0x322e1 bytes) 000285 1787843071.669143 write 0x029cea thru 0x0389b8 (0xeccf bytes) 000287 1787843071.675462 read 0x00fdbb thru 0x01a6df (0xa925 bytes) 000289 1787843071.686563 write 0x02123e thru 0x02fcd5 (0xea98 bytes) 000290 1787843071.699151 mapwrite 0x02fc8c thru 0x038815 (0x8b8a bytes) 000291 1787843071.761373 write 0x000ea4 thru 0x0076e1 (0x683e bytes) 000293 1787843071.766596 read 0x00460a thru 0x00db4d (0x9544 bytes) 000294 1787843071.787816 write 0x003a20 thru 0x010151 (0xc732 bytes) 000295 1787843071.788902 read 0x025b4e thru 0x02ac9d (0x5150 bytes) 000299 1787843071.791597 write 0x01aadb thru 0x01f9fb (0x4f21 bytes) 000300 1787843071.794186 write 0x013186 thru 0x015714 (0x258f bytes) 000301 1787843071.795648 write 0x011e7b thru 0x015961 (0x3ae7 bytes) 000303 1787843071.797079 mapread 0x00fc6c thru 0x018ec8 (0x925d bytes) 000305 1787843071.800739 mapwrite 0x0276ac thru 0x035f68 (0xe8bd bytes) 000307 1787843071.852989 read 0x0119cd thru 0x01b801 (0x9e35 bytes) 000309 1787843071.854154 mapwrite 0x0150ca thru 0x01e76c (0x96a3 bytes) 000310 1787843071.912500 MIRROR_RESYNC 3 mirrors 000311 1787843072.730594 trunc from 0x0389b9 to 0x015ede (0x22ada bytes) 000315 1787843072.816235 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15edf): Mirror 5: 0xd297e73e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15edf): Mirror 5: 0xd297e73e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15edf: skipped 000317 1787843073.079313 mapread 0x003993 thru 0x006936 (0x2fa4 bytes) 000318 1787843073.098547 mapwrite 0x01cf22 thru 0x0273ee (0xa4cd bytes) 000322 1787843073.185763 mapwrite 0x00f2b5 thru 0x01efae (0xfcfa bytes) 000324 1787843073.239261 read 0x015e54 thru 0x022bbd (0xcd6a bytes) 000325 1787843073.239973 mapread 0x00ad64 thru 0x01081d (0x5aba bytes) 000326 1787843073.240907 read 0x01bad9 thru 0x02306b (0x7593 bytes) 000330 1787843073.244508 write 0x013efb thru 0x014ae3 (0xbe9 bytes) 000333 1787843073.247354 trunc from 0x0273ef to 0x0355ef (0xe201 bytes) 000335 1787843073.269056 mapwrite 0x0262a1 thru 0x02716e (0xece bytes) 000337 1787843073.310053 mapwrite 0x032abf thru 0x03ffff (0xd541 bytes) 000339 1787843073.392974 mapwrite 0x03ca0f thru 0x03ffff (0x35f1 bytes) 000340 1787843073.435265 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x70998485 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa472f1f8 Mirror 6: 0xa472f1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000341 1787843074.917710 mapread 0x01e271 thru 0x02c43a (0xe1ca bytes) 000343 1787843074.948605 mapwrite 0x016037 thru 0x023fce (0xdf98 bytes) 000344 1787843075.175343 trunc from 0x040000 to 0x0291c5 (0x16e3a bytes) 000345 1787843075.215275 trunc from 0x0291c6 to 0x02bf4b (0x2d86 bytes) 000352 1787843075.244185 mapread 0x01bd30 thru 0x01f522 (0x37f3 bytes) 000353 1787843075.245674 write 0x001f75 thru 0x0026e2 (0x76e bytes) 000356 1787843075.263786 read 0x0188d9 thru 0x023d91 (0xb4b9 bytes) 000357 1787843075.266510 mapwrite 0x0376b0 thru 0x03ffff (0x8950 bytes) 000360 1787843075.411314 mapread 0x0018f4 thru 0x009ea2 (0x85af bytes) 000361 1787843075.432554 write 0x00e668 thru 0x019e4a (0xb7e3 bytes) 000362 1787843075.435329 mapwrite 0x03c6be thru 0x03f895 (0x31d8 bytes) 000365 1787843075.524354 read 0x013db6 thru 0x023147 (0xf392 bytes) 000366 1787843075.525405 write 0x01d4ff thru 0x0260fa (0x8bfc bytes) 000367 1787843075.527871 read 0x038161 thru 0x03ffff (0x7e9f bytes) 000368 1787843075.529302 mapwrite 0x030d4f thru 0x03ffff (0xf2b1 bytes) 000369 1787843075.623393 read 0x037443 thru 0x038e71 (0x1a2f bytes) 000370 1787843075.624120 trunc from 0x040000 to 0x03ae96 (0x5169 bytes) 000373 1787843075.655653 mapread 0x00b65b thru 0x00d794 (0x213a bytes) 000375 1787843075.657112 write 0x018379 thru 0x01f14a (0x6dd2 bytes) 000376 1787843075.663358 read 0x01fca7 thru 0x023e1d (0x4177 bytes) 000378 1787843075.665152 trunc from 0x03ae97 to 0x0031d3 (0x37cc3 bytes) 000379 1787843075.704748 trunc from 0x0031d4 to 0x015d55 (0x12b82 bytes) 000380 1787843075.731699 mapread 0x01519e thru 0x015d55 (0xbb8 bytes) 000381 1787843075.745474 mapread 0x0120d2 thru 0x015d55 (0x3c84 bytes) 000382 1787843075.746575 mapread 0x0057ac thru 0x00c0ef (0x6944 bytes) 000384 1787843075.749333 mapread 0x00bf8f thru 0x00cbc9 (0xc3b bytes) 000385 1787843075.750468 trunc from 0x015d56 to 0x0257a7 (0xfa52 bytes) 000388 1787843075.797435 trunc from 0x0257a8 to 0x017985 (0xde22 bytes) 000390 1787843075.820511 read 0x006bdc thru 0x0125c9 (0xb9ee bytes) 000392 1787843075.827995 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17986): Mirror 6: 0x540f9a10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17986): Mirror 6: 0x540f9a10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17986: skipped 000395 1787843076.150758 MIRROR_RESYNC 2 mirrors 000397 1787843077.132521 MIRROR_EXTEND to 3 mirrors 000398 1787843078.133825 mapwrite 0x037da4 thru 0x03ffff (0x825c bytes) 000399 1787843078.323652 trunc from 0x040000 to 0x00bbb8 (0x34447 bytes) 000401 1787843078.350098 trunc from 0x00bbb9 to 0x027711 (0x1bb59 bytes) 000402 1787843078.373673 read 0x00e8e1 thru 0x012999 (0x40b9 bytes) 000404 1787843078.387351 read 0x025dc6 thru 0x027711 (0x194c bytes) 000405 1787843078.401548 mapread 0x0181fa thru 0x018ce8 (0xaef bytes) 000406 1787843078.402538 write 0x01a8b7 thru 0x024485 (0x9bcf bytes) 000408 1787843078.405116 read 0x000fb5 thru 0x0068c0 (0x590c bytes) 000409 1787843078.405798 write 0x03a93b thru 0x03f570 (0x4c36 bytes) 000411 1787843078.408531 write 0x022a7d thru 0x02a3c4 (0x7948 bytes) 000415 1787843078.417353 mapwrite 0x00f85b thru 0x00ff1e (0x6c4 bytes) 000418 1787843078.483558 mapwrite 0x0073f8 thru 0x0108fc (0x9505 bytes) 000419 1787843078.545872 mapwrite 0x001337 thru 0x00e2b7 (0xcf81 bytes) 000420 1787843078.619288 mapwrite 0x02706e thru 0x0319e6 (0xa979 bytes) 000421 1787843078.690498 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f571): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f571): Mirror 6: 0x47f98b0d (stale) Mirror 7: 0x47f98b0d (stale) CRC-32 checksum value for chunk [0, 0x3f571): Mirror 1: 0x74867ee2 Mirror 8: 0x74867ee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f571: skipped 000422 1787843079.577804 mapwrite 0x02467f thru 0x02cd4e (0x86d0 bytes) 000425 1787843079.822034 read 0x03d060 thru 0x03f570 (0x2511 bytes) 000426 1787843079.829927 mapwrite 0x037431 thru 0x03be68 (0x4a38 bytes) 000427 1787843079.941416 mapread 0x039d12 thru 0x03af74 (0x1263 bytes) 000428 1787843079.943447 trunc from 0x03f571 to 0x01bb97 (0x239d9 bytes) 000430 1787843079.967721 mapwrite 0x037896 thru 0x03ffff (0x876a bytes) 000433 1787843080.049556 write 0x036b54 thru 0x03ec3b (0x80e8 bytes) 000434 1787843080.058403 MIRROR_RESYNC 4 mirrors 000436 1787843081.340244 read 0x008c32 thru 0x008c61 (0x30 bytes) 000439 1787843081.346817 write 0x0095a3 thru 0x00d369 (0x3dc7 bytes) 000440 1787843081.457621 mapread 0x029938 thru 0x02cc31 (0x32fa bytes) 000444 1787843081.495329 MIRROR_RESYNC 4 mirrors 000445 1787843082.698334 mapwrite 0x01fb5d thru 0x02d337 (0xd7db bytes) 000447 1787843082.891919 read 0x024bcb thru 0x025c17 (0x104d bytes) 000449 1787843082.892990 read 0x0241dd thru 0x033ddb (0xfbff bytes) 000452 1787843082.911566 mapread 0x034ccf thru 0x0363e1 (0x1713 bytes) 000453 1787843082.914704 mapread 0x02524a thru 0x02ac56 (0x5a0d bytes) 000454 1787843082.916517 write 0x00a4e7 thru 0x016808 (0xc322 bytes) 000455 1787843082.920583 trunc from 0x040000 to 0x013df1 (0x2c20e bytes) 000456 1787843082.946053 write 0x02bc0b thru 0x02bf9d (0x393 bytes) 000457 1787843082.952594 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bf9e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf9e): Mirror 6: 0x2ad07d88 (stale) Mirror 7: 0x2ad07d88 (stale) Mirror 8: 0x2ad07d88 (stale) CRC-32 checksum value for chunk [0, 0x2bf9e): Mirror 1: 0x80d3d1a7 Mirror 9: 0x80d3d1a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf9e: skipped 000458 1787843084.267392 trunc from 0x02bf9e to 0x00a2cb (0x21cd2 bytes) 000462 1787843084.476908 read 0x002cec thru 0x00a2cb (0x75e0 bytes) 000463 1787843084.500264 mapwrite 0x027537 thru 0x02ff02 (0x89cc bytes) 000464 1787843084.726154 mapread 0x008fc1 thru 0x00958f (0x5cf bytes) 000465 1787843084.727377 trunc from 0x02ff03 to 0x031eed (0x1feb bytes) 000466 1787843084.776161 mapwrite 0x013e9f thru 0x013fc5 (0x127 bytes) 000469 1787843084.943498 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31eee): Mirror 6: 0xdbbbbbeb (stale) Mirror 8: 0xdbbbbbeb (stale) Mirror 9: 0xc57595f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31eee): Mirror 6: 0xdbbbbbeb (stale) Mirror 8: 0xdbbbbbeb (stale) Mirror 9: 0xc57595f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31eee: skipped 000470 1787843085.543361 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31eee): Mirror 8: 0xdbbbbbeb (stale) Mirror 9: 0xc57595f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31eee): Mirror 8: 0xdbbbbbeb (stale) Mirror 9: 0xc57595f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31eee: skipped 000471 1787843086.063406 mapwrite 0x03ddba thru 0x03f45f (0x16a6 bytes) 000472 1787843086.152434 mapread 0x000cd9 thru 0x00d9d1 (0xccf9 bytes) 000473 1787843086.162770 read 0x012120 thru 0x013f70 (0x1e51 bytes) 000474 1787843086.169386 write 0x010b4b thru 0x011b46 (0xffc bytes) 000475 1787843086.179040 mapread 0x021228 thru 0x02cc62 (0xba3b bytes) 000478 1787843086.199366 mapwrite 0x039cbb thru 0x03da2a (0x3d70 bytes) 000479 1787843086.262879 read 0x013254 thru 0x01b444 (0x81f1 bytes) 000480 1787843086.273349 mapwrite 0x004dc8 thru 0x010e33 (0xc06c bytes) 000481 1787843086.321624 trunc from 0x03f460 to 0x00af77 (0x344e8 bytes) 000483 1787843086.347675 mapread 0x009ff4 thru 0x00af77 (0xf84 bytes) 000484 1787843086.351647 mapwrite 0x02cb8a thru 0x034a92 (0x7f09 bytes) 000486 1787843086.529652 MIRROR_RESYNC 3 mirrors 000487 1787843088.160934 write 0x0391ee thru 0x03ffff (0x6e12 bytes) 000488 1787843088.286288 trunc from 0x040000 to 0x03979b (0x6864 bytes) 000489 1787843088.324353 trunc from 0x03979c to 0x0173dc (0x223bf bytes) 000490 1787843088.353275 trunc from 0x0173dd to 0x023953 (0xc577 bytes) 000491 1787843088.391394 read 0x015bee thru 0x017cf6 (0x2109 bytes) 000492 1787843088.402206 mapread 0x0062fb thru 0x00755b (0x1261 bytes) 000496 1787843088.417816 write 0x03b634 thru 0x03ffff (0x49cc bytes) 000498 1787843088.427162 mapwrite 0x0381a5 thru 0x03ffff (0x7e5b bytes) 000499 1787843088.489180 trunc from 0x040000 to 0x00ef27 (0x310d8 bytes) 000501 1787843088.512822 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xef28): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xef28): Mirror 8: 0xf1736c01 (stale) Mirror 9: 0xf1736c01 (stale) CRC-32 checksum value for chunk [0, 0xef28): Mirror 1: 0xf1736c01 Mirror 10: 0xf1736c01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xef28: skipped 000502 1787843089.660781 mapwrite 0x018935 thru 0x01c3b6 (0x3a82 bytes) 000504 1787843089.844374 trunc from 0x01c3b7 to 0x012d6a (0x964c bytes) 000505 1787843089.889589 MIRROR_RESYNC 4 mirrors 000506 1787843090.948997 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000512 1787843091.291255 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000518 1787843091.644422 MIRROR_RESYNC 2 mirrors 000519 1787843092.059430 read 0x01187c thru 0x012d6a (0x14ef bytes) 000522 1787843092.072126 read 0x01287c thru 0x012d6a (0x4ef bytes) 000523 1787843092.073180 mapwrite 0x037255 thru 0x03be94 (0x4c40 bytes) 000524 1787843092.267927 mapwrite 0x03902a thru 0x03ffff (0x6fd6 bytes) 000527 1787843092.350016 trunc from 0x040000 to 0x01fef5 (0x2010a bytes) 000528 1787843092.376729 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 10: 0xbafa1d78 (stale) CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 1: 0xbafa1d78 Mirror 11: 0xbafa1d78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fef6: skipped 000530 1787843093.335697 write 0x02f5c3 thru 0x03e7c9 (0xf207 bytes) 000532 1787843093.413238 read 0x0195e8 thru 0x019f66 (0x97f bytes) 000533 1787843093.422587 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e7ca): Mirror 11: 0xdc13a0bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7ca): Mirror 11: 0xdc13a0bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7ca: skipped 000534 1787843093.704435 mapwrite 0x02d3d7 thru 0x0397fc (0xc426 bytes) 000536 1787843093.785459 mapread 0x02a840 thru 0x03914f (0xe910 bytes) 000537 1787843093.791269 MIRROR_RESYNC 2 mirrors 000538 1787843094.629548 mapread 0x03a3bc thru 0x03cc5d (0x28a2 bytes) 000541 1787843094.654054 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000543 1787843094.803032 write 0x017d89 thru 0x01bf71 (0x41e9 bytes) 000544 1787843094.833624 MIRROR_EXTEND to 2 mirrors 000546 1787843095.824131 trunc from 0x03e7ca to 0x01c22e (0x2259b bytes) 000547 1787843095.951622 mapwrite 0x000fc8 thru 0x0013e7 (0x420 bytes) 000548 1787843096.042190 trunc from 0x01c22f to 0x039a45 (0x1d817 bytes) 000549 1787843096.070250 write 0x02576b thru 0x02b2f7 (0x5b8d bytes) 000550 1787843096.082285 write 0x011f23 thru 0x01327c (0x135a bytes) 000551 1787843096.093157 trunc from 0x039a46 to 0x025588 (0x144bd bytes) 000552 1787843096.115161 mapwrite 0x00d83a thru 0x014d59 (0x7520 bytes) 000553 1787843096.176430 trunc from 0x025589 to 0x0202b1 (0x52d7 bytes) 000554 1787843096.197144 trunc from 0x0202b2 to 0x003931 (0x1c980 bytes) 000556 1787843096.220568 mapread 0x001c94 thru 0x003931 (0x1c9e bytes) 000557 1787843096.226891 write 0x015d38 thru 0x01f633 (0x98fc bytes) 000559 1787843096.230338 mapread 0x00290e thru 0x0124d5 (0xfbc8 bytes) 000561 1787843096.246099 mapwrite 0x000482 thru 0x00b5fb (0xb17a bytes) 000562 1787843096.307927 read 0x01e52b thru 0x01f633 (0x1109 bytes) 000563 1787843096.308525 mapread 0x009c8e thru 0x01750f (0xd882 bytes) 000565 1787843096.312074 write 0x00e781 thru 0x016836 (0x80b6 bytes) 000567 1787843096.315348 read 0x009419 thru 0x00c2e4 (0x2ecc bytes) 000568 1787843096.326947 MIRROR_RESYNC 2 mirrors 000569 1787843096.982360 trunc from 0x01f634 to 0x017988 (0x7cab bytes) 000570 1787843097.082938 mapread 0x011719 thru 0x01391c (0x2204 bytes) 000573 1787843097.108249 MIRROR_RESYNC 2 mirrors 000574 1787843098.229991 mapread 0x0076eb thru 0x00bad3 (0x43e9 bytes) 000575 1787843098.255262 mapwrite 0x01d5a7 thru 0x01fc6a (0x26c4 bytes) 000579 1787843098.436397 trunc from 0x01fc6b to 0x01a6b4 (0x55b6 bytes) 000580 1787843098.467955 mapwrite 0x014e76 thru 0x01dab8 (0x8c43 bytes) 000581 1787843098.564816 mapread 0x01180d thru 0x01983e (0x8032 bytes) 000582 1787843098.566105 read 0x000010 thru 0x00194a (0x193b bytes) 000583 1787843098.572447 read 0x001b73 thru 0x00b223 (0x96b1 bytes) 000584 1787843098.586649 read 0x012ae5 thru 0x01dab8 (0xafd4 bytes) 000585 1787843098.587371 trunc from 0x01dab9 to 0x026978 (0x8ec0 bytes) 000586 1787843098.621874 write 0x031fe1 thru 0x036dd2 (0x4df2 bytes) 000587 1787843098.623518 mapread 0x021b3c thru 0x0281d5 (0x669a bytes) 000589 1787843098.642956 write 0x0149b1 thru 0x0196ac (0x4cfc bytes) 000590 1787843098.645010 mapread 0x0094ed thru 0x00c2af (0x2dc3 bytes) 000591 1787843098.650130 trunc from 0x036dd3 to 0x038384 (0x15b2 bytes) 000592 1787843098.693327 read 0x01ab2a thru 0x01d8e6 (0x2dbd bytes) 000593 1787843098.698244 mapwrite 0x00f69b thru 0x01bd2f (0xc695 bytes) 000595 1787843098.786689 mapread 0x01817d thru 0x01ae58 (0x2cdc bytes) 000596 1787843098.789917 trunc from 0x038385 to 0x016818 (0x21b6c bytes) 000597 1787843098.833982 MIRROR_RESYNC 2 mirrors 000598 1787843099.859472 write 0x0029e5 thru 0x004421 (0x1a3d bytes) 000599 1787843099.986047 mapread 0x0158ec thru 0x016818 (0xf2d bytes) 000600 1787843100.005102 trunc from 0x016819 to 0x00df6c (0x88ac bytes) 000601 1787843100.034142 read 0x0090d2 thru 0x00df6c (0x4e9b bytes) 000602 1787843100.046638 mapwrite 0x02a437 thru 0x02e04d (0x3c17 bytes) 000603 1787843100.120672 write 0x03c7c5 thru 0x03eabf (0x22fb bytes) truncating to largest ever: 0x3fc76 000604 1787843100.121942 trunc from 0x03eac0 to 0x03fc75 (0x11b6 bytes) 000609 1787843100.157479 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000611 1787843100.333329 read 0x03f89e thru 0x03fc75 (0x3d8 bytes) 000612 1787843100.339867 trunc from 0x03fc76 to 0x01cef4 (0x22d81 bytes) 000614 1787843100.358053 mapwrite 0x014450 thru 0x022125 (0xdcd6 bytes) 000615 1787843100.469202 write 0x022d6a thru 0x02825d (0x54f4 bytes) 000619 1787843100.474527 trunc from 0x02825e to 0x03296a (0xa70d bytes) 000620 1787843100.500422 MIRROR_EXTEND to 2 mirrors 000621 1787843101.584983 write 0x037bf9 thru 0x03ffff (0x8407 bytes) 000622 1787843101.699440 mapread 0x00781d thru 0x012a5d (0xb241 bytes) 000624 1787843101.759038 read 0x03f4cf thru 0x03ffff (0xb31 bytes) 000627 1787843101.763986 trunc from 0x040000 to 0x00f6ff (0x30900 bytes) 000628 1787843101.796357 trunc from 0x00f700 to 0x03a33b (0x2ac3c bytes) 000629 1787843101.829991 mapread 0x016920 thru 0x01834d (0x1a2e bytes) 000632 1787843101.851142 mapread 0x013b92 thru 0x01ef77 (0xb3e6 bytes) 000633 1787843101.853262 mapread 0x01a206 thru 0x01d718 (0x3513 bytes) 000634 1787843101.864098 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000638 1787843102.130994 MIRROR_EXTEND to 2 mirrors 000639 1787843103.126212 mapread 0x02621a thru 0x02da6a (0x7851 bytes) 000640 1787843103.160016 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000641 1787843103.300274 mapread 0x023fec thru 0x02e447 (0xa45c bytes) 000642 1787843103.345131 read 0x035ad2 thru 0x03a33b (0x486a bytes) 000643 1787843103.356247 mapread 0x033a4a thru 0x034370 (0x927 bytes) 000644 1787843103.359519 mapwrite 0x02d3a7 thru 0x030ce2 (0x393c bytes) 000646 1787843103.498895 read 0x0340b6 thru 0x03710a (0x3055 bytes) 000650 1787843103.514232 write 0x0034e2 thru 0x00eca7 (0xb7c6 bytes) 000654 1787843103.522935 trunc from 0x03a33c to 0x016e0b (0x23530 bytes) 000655 1787843103.547801 read 0x001b53 thru 0x0047c5 (0x2c73 bytes) 000657 1787843103.556226 mapread 0x0099ed thru 0x00b611 (0x1c25 bytes) 000659 1787843103.561121 mapwrite 0x0171a9 thru 0x019d0f (0x2b67 bytes) 000661 1787843103.655004 mapwrite 0x02c94b thru 0x036d1e (0xa3d4 bytes) 000662 1787843103.830067 trunc from 0x036d1f to 0x020db7 (0x15f67 bytes) 000663 1787843103.852541 read 0x0194a1 thru 0x020db7 (0x7917 bytes) 000664 1787843103.853768 write 0x00cd1a thru 0x00ce67 (0x14e bytes) 000665 1787843103.856017 write 0x0185cd thru 0x01c456 (0x3e8a bytes) 000666 1787843103.857525 trunc from 0x020db8 to 0x01f217 (0x1ba0 bytes) 000667 1787843103.881671 mapread 0x00c45c thru 0x015163 (0x8d08 bytes) 000668 1787843103.883165 mapwrite 0x01a04a thru 0x024b75 (0xab2c bytes) 000669 1787843103.970254 mapwrite 0x00d6e8 thru 0x01bb0d (0xe426 bytes) 000671 1787843104.041434 mapread 0x010398 thru 0x01a99b (0xa604 bytes) 000672 1787843104.044964 mapwrite 0x039c18 thru 0x03ffff (0x63e8 bytes) 000674 1787843104.216524 write 0x038f7b thru 0x03ffff (0x7085 bytes) 000675 1787843104.218956 write 0x0173ff thru 0x01f78f (0x8391 bytes) 000677 1787843104.222215 write 0x0107b7 thru 0x012d5f (0x25a9 bytes) 000678 1787843104.224094 read 0x039aa2 thru 0x03c991 (0x2ef0 bytes) 000679 1787843104.224914 trunc from 0x040000 to 0x0123b7 (0x2dc48 bytes) 000680 1787843104.252085 trunc from 0x0123b8 to 0x037792 (0x253db bytes) 000681 1787843104.271067 read 0x011dd1 thru 0x01f193 (0xd3c3 bytes) 000682 1787843104.283123 read 0x02a10d thru 0x030257 (0x614b bytes) 000684 1787843104.284368 write 0x0106fc thru 0x01e0e5 (0xd9ea bytes) 000685 1787843104.287040 mapwrite 0x0344a5 thru 0x03a930 (0x648c bytes) 000686 1787843104.371507 write 0x0176d9 thru 0x0202cd (0x8bf5 bytes) 000688 1787843104.373365 write 0x004df0 thru 0x010f31 (0xc142 bytes) 000689 1787843104.374696 mapwrite 0x025f0d thru 0x033f87 (0xe07b bytes) 000690 1787843104.426778 MIRROR_EXTEND to 2 mirrors 000691 1787843104.966710 mapwrite 0x033175 thru 0x03a42a (0x72b6 bytes) 000692 1787843105.201748 trunc from 0x03a931 to 0x03191b (0x9015 bytes) 000693 1787843105.273882 write 0x02ba33 thru 0x037b84 (0xc152 bytes) 000694 1787843105.285095 mapwrite 0x0173ac thru 0x01d1f1 (0x5e46 bytes) 000695 1787843105.357812 write 0x0318fe thru 0x039929 (0x802c bytes) 000697 1787843105.359615 trunc from 0x03992a to 0x0092e0 (0x30649 bytes) 000701 1787843105.378017 read 0x000eb8 thru 0x001c30 (0xd79 bytes) 000704 1787843105.383550 write 0x010b71 thru 0x0145f5 (0x3a85 bytes) 000706 1787843105.385325 trunc from 0x0145f6 to 0x004054 (0x105a1 bytes) 000709 1787843105.401507 mapwrite 0x00e25d thru 0x0198b5 (0xb659 bytes) 000710 1787843105.497732 read 0x00226b thru 0x00d5a9 (0xb33f bytes) 000713 1787843105.518925 write 0x038532 thru 0x03ffff (0x7ace bytes) 000714 1787843105.526431 mapwrite 0x039c34 thru 0x03ffff (0x63cc bytes) 000715 1787843105.595142 mapwrite 0x03e480 thru 0x03ffff (0x1b80 bytes) 000716 1787843105.663071 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000719 1787843105.857109 trunc from 0x040000 to 0x01aac5 (0x2553a bytes) 000720 1787843105.876873 trunc from 0x01aac6 to 0x026d92 (0xc2cd bytes) 000723 1787843105.894773 trunc from 0x026d93 to 0x0076eb (0x1f6a7 bytes) 000725 1787843105.914340 trunc from 0x0076ec to 0x0043da (0x3311 bytes) 000726 1787843105.936658 mapwrite 0x008d1c thru 0x00f8f4 (0x6bd9 bytes) 000729 1787843106.027910 mapread 0x00cb02 thru 0x00f8f4 (0x2df3 bytes) 000730 1787843106.029269 trunc from 0x00f8f5 to 0x00ad58 (0x4b9c bytes) 000732 1787843106.046028 mapwrite 0x037eaa thru 0x03b581 (0x36d8 bytes) 000736 1787843106.121801 mapwrite 0x02bb29 thru 0x035d22 (0xa1fa bytes) 000739 1787843106.188155 mapwrite 0x00feb9 thru 0x016bf6 (0x6d3e bytes) 000743 1787843106.243661 trunc from 0x03b582 to 0x01c7cc (0x1edb5 bytes) 000744 1787843106.268229 write 0x006193 thru 0x00b40e (0x527c bytes) 000745 1787843106.269387 read 0x005d9d thru 0x009c09 (0x3e6d bytes) 000746 1787843106.270011 read 0x00eddf thru 0x01ad41 (0xbf63 bytes) 000748 1787843106.284504 MIRROR_EXTEND to 2 mirrors 000749 1787843107.137083 trunc from 0x01c7cd to 0x016172 (0x665a bytes) 000750 1787843107.319731 MIRROR_RESYNC 2 mirrors 000751 1787843108.261123 read 0x003eac thru 0x00e595 (0xa6ea bytes) 000753 1787843108.276364 read 0x0008c1 thru 0x00c5a6 (0xbce6 bytes) 000754 1787843108.296247 trunc from 0x016173 to 0x01b71f (0x55ad bytes) 000755 1787843108.404693 write 0x01341a thru 0x01cced (0x98d4 bytes) 000756 1787843108.416797 write 0x039f6a thru 0x03eabc (0x4b53 bytes) 000757 1787843108.418171 read 0x022d2c thru 0x032209 (0xf4de bytes) 000759 1787843108.447059 mapwrite 0x01cd5b thru 0x024c64 (0x7f0a bytes) 000760 1787843108.514266 mapwrite 0x0195c9 thru 0x01dc71 (0x46a9 bytes) 000761 1787843108.567487 mapread 0x01dc36 thru 0x02bd31 (0xe0fc bytes) 000764 1787843108.569560 mapwrite 0x000b9f thru 0x00cf46 (0xc3a8 bytes) 000765 1787843108.630166 mapwrite 0x0079a2 thru 0x014595 (0xcbf4 bytes) 000766 1787843108.684968 mapwrite 0x00367b thru 0x00f9ba (0xc340 bytes) 000767 1787843108.735209 read 0x0027ce thru 0x011a48 (0xf27b bytes) 000769 1787843108.741254 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3eabd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabd): Mirror 2: 0xe99ae52 (stale) CRC-32 checksum value for chunk [0, 0x3eabd): Mirror 1: 0x7e18d07a Mirror 3: 0x7e18d07a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabd: skipped 000770 1787843110.204610 trunc from 0x03eabd to 0x0050d6 (0x399e6 bytes) 000771 1787843110.310734 mapwrite 0x018fdd thru 0x02831e (0xf342 bytes) 000772 1787843110.440805 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2831f): Mirror 3: 0x5eab01ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2831f): Mirror 3: 0x5eab01ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2831f: skipped 000774 1787843110.757524 trunc from 0x02831f to 0x036931 (0xe613 bytes) 000775 1787843110.796937 mapread 0x013537 thru 0x01ea5d (0xb527 bytes) 000777 1787843110.879402 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36932): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36932): Mirror 3: 0xa3c6424c (stale) CRC-32 checksum value for chunk [0, 0x36932): Mirror 1: 0x3500068e Mirror 4: 0x3500068e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36932: skipped 000779 1787843112.382088 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) 000780 1787843112.751374 mapwrite 0x03327e thru 0x0376c8 (0x444b bytes) 000781 1787843113.131409 read 0x0325fb thru 0x0376c8 (0x50ce bytes) 000782 1787843113.138651 write 0x0234a8 thru 0x031d45 (0xe89e bytes) 000783 1787843113.159195 read 0x02f20a thru 0x0376c8 (0x84bf bytes) 000785 1787843113.160308 read 0x013f92 thru 0x01ffdd (0xc04c bytes) 000786 1787843113.187387 write 0x00928b thru 0x0169b6 (0xd72c bytes) 000787 1787843113.215717 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: 0xe73aa1b7 (stale) CRC-32 checksum value for chunk [0, 0x376c9): Mirror 1: 0x976a0ddf Mirror 5: 0x976a0ddf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376c9: skipped 000789 1787843114.603743 mapread 0x01efe5 thru 0x022f0c (0x3f28 bytes) 000791 1787843114.631745 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: 0xe73aa1b7 (stale) CRC-32 checksum value for chunk [0, 0x376c9): Mirror 1: 0x976a0ddf Mirror 5: 0x976a0ddf Mirror 6: 0x976a0ddf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376c9: skipped 000794 1787843115.896724 MIRROR_RESYNC 4 mirrors 000795 1787843116.705727 MIRROR_RESYNC 4 mirrors 000796 1787843117.399154 write 0x02d048 thru 0x032e9f (0x5e58 bytes) 000797 1787843117.543110 read 0x033073 thru 0x0376c8 (0x4656 bytes) 000801 1787843117.570381 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376c9): Mirror 4: 0x976a0ddf (stale) Mirror 5: 0x976a0ddf (stale) Mirror 6: 0x976a0ddf (stale) CRC-32 checksum value for chunk [0, 0x376c9): Mirror 1: 0xdbed224 Mirror 7: 0xdbed224 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376c9: skipped 000804 1787843119.148824 mapwrite 0x03ae9b thru 0x03ffff (0x5165 bytes) 000808 1787843119.506208 mapread 0x01df8b thru 0x026aec (0x8b62 bytes) 000809 1787843119.563535 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2850c72a (stale) Mirror 5: 0x2850c72a (stale) Mirror 6: 0x2850c72a (stale) Mirror 7: 0xd3f562f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd328d60 Mirror 8: 0xfd328d60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1787843120.816421 write 0x037011 thru 0x03ffff (0x8fef bytes) 000811 1787843120.936858 write 0x035f31 thru 0x03ffff (0xa0cf bytes) 000813 1787843120.946195 trunc from 0x040000 to 0x035832 (0xa7cd bytes) 000816 1787843121.008608 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35833): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35833): Mirror 4: 0x8e96b6eb (stale) Mirror 5: 0x8e96b6eb (stale) Mirror 6: 0x8e96b6eb (stale) Mirror 7: 0xf7ac80c3 (stale) Mirror 8: 0xf7ac80c3 (stale) CRC-32 checksum value for chunk [0, 0x35833): Mirror 1: 0xf7ac80c3 Mirror 9: 0xf7ac80c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35833: skipped 000819 1787843122.600634 mapwrite 0x026348 thru 0x02ac4c (0x4905 bytes) 000820 1787843122.829781 mapwrite 0x00c40e thru 0x00f61c (0x320f bytes) 000821 1787843122.900773 trunc from 0x035833 to 0x037fb1 (0x277f bytes) 000822 1787843122.932071 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb2): Mirror 4: 0xfa417034 (stale) Mirror 5: 0xfa417034 (stale) Mirror 6: 0xfa417034 (stale) Mirror 8: 0x3f12d11 (stale) Mirror 9: 0x9341fe96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb2): Mirror 4: 0xfa417034 (stale) Mirror 5: 0xfa417034 (stale) Mirror 6: 0xfa417034 (stale) Mirror 8: 0x3f12d11 (stale) Mirror 9: 0x9341fe96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb2: skipped 000823 1787843123.376393 trunc from 0x037fb2 to 0x03aeb0 (0x2eff bytes) 000825 1787843123.396702 read 0x01dea7 thru 0x023184 (0x52de bytes) 000826 1787843123.413366 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3aeb1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aeb1): Mirror 4: 0xbef68299 (stale) Mirror 5: 0xbef68299 (stale) Mirror 6: 0xbef68299 (stale) Mirror 8: 0x1db26dfb (stale) Mirror 9: 0x9f47f4dc (stale) CRC-32 checksum value for chunk [0, 0x3aeb1): Mirror 1: 0xe69eda8 Mirror 10: 0xe69eda8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aeb1: skipped 000827 1787843124.967357 mapread 0x00734c thru 0x01209f (0xad54 bytes) 000828 1787843125.008970 read 0x016246 thru 0x02411b (0xded6 bytes) 000830 1787843125.024174 read 0x02d333 thru 0x02f313 (0x1fe1 bytes) 000831 1787843125.036174 write 0x03a363 thru 0x03ffff (0x5c9d bytes) 000833 1787843125.160083 read 0x00cf6a thru 0x0123f5 (0x548c bytes) 000834 1787843125.161427 trunc from 0x040000 to 0x023eca (0x1c135 bytes) 000835 1787843125.207986 read 0x01d7a3 thru 0x023eca (0x6728 bytes) 000839 1787843125.220417 trunc from 0x023ecb to 0x0337d6 (0xf90c bytes) 000840 1787843125.250618 read 0x020225 thru 0x0292d1 (0x90ad bytes) 000842 1787843125.262615 read 0x004d9f thru 0x009659 (0x48bb bytes) 000844 1787843125.277496 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x337d7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337d7): Mirror 4: 0x8059d98f (stale) Mirror 5: 0x8059d98f (stale) Mirror 6: 0x8059d98f (stale) Mirror 8: 0x96c11f1f (stale) Mirror 9: 0x96c11f1f (stale) Mirror 10: 0x448c6c04 (stale) CRC-32 checksum value for chunk [0, 0x337d7): Mirror 1: 0xfee478d Mirror 11: 0xfee478d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337d7: skipped 000846 1787843126.892774 mapwrite 0x02c995 thru 0x02ccea (0x356 bytes) 000849 1787843127.158519 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x337d7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337d7): Mirror 4: 0x8059d98f (stale) Mirror 5: 0x8059d98f (stale) Mirror 6: 0x8059d98f (stale) Mirror 8: 0x96c11f1f (stale) Mirror 9: 0x96c11f1f (stale) Mirror 10: 0x448c6c04 (stale) Mirror 11: 0xfee478d (stale) CRC-32 checksum value for chunk [0, 0x337d7): Mirror 1: 0x5e74f03d Mirror 12: 0x5e74f03d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337d7: skipped 000850 1787843129.350720 trunc from 0x0337d7 to 0x01ec18 (0x14bbe bytes) 000851 1787843129.450693 mapwrite 0x023461 thru 0x02ca79 (0x9619 bytes) 000852 1787843129.559381 read 0x02ac04 thru 0x02ca79 (0x1e76 bytes) 000853 1787843129.566376 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ca7a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca7a): Mirror 4: 0x5761fbb4 (stale) Mirror 5: 0x5761fbb4 (stale) Mirror 6: 0x5761fbb4 (stale) Mirror 8: 0x5761fbb4 (stale) Mirror 9: 0x5761fbb4 (stale) Mirror 10: 0x855505c6 (stale) Mirror 11: 0xa5cb9e39 (stale) Mirror 12: 0x12a6bd86 (stale) CRC-32 checksum value for chunk [0, 0x2ca7a): Mirror 1: 0x82696208 Mirror 13: 0x82696208 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca7a: skipped 000855 1787843130.443737 mapwrite 0x0217f2 thru 0x030317 (0xeb26 bytes) 000856 1787843130.673994 trunc from 0x030318 to 0x015a8d (0x1a88a bytes) 000857 1787843130.698291 write 0x009d7d thru 0x01834c (0xe5d0 bytes) 000859 1787843130.711081 MIRROR_RESYNC 10 mirrors 000860 1787843132.665508 trunc from 0x01834d to 0x011d2b (0x6621 bytes) 000865 1787843132.798800 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 6: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 10: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 6: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 10: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d2c: skipped 000866 1787843133.347112 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 6: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 6: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d2c: skipped 000868 1787843133.825956 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d2c: skipped 000869 1787843134.396098 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 4: 0x46495361 (stale) Mirror 5: 0x46495361 (stale) Mirror 9: 0x46495361 (stale) Mirror 11: 0x46495361 (stale) Mirror 12: 0x46495361 (stale) Mirror 13: 0x46495361 (stale) CRC-32 checksum value for chunk [0, 0x11d2c): Mirror 1: 0x46495361 Mirror 14: 0x46495361 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d2c: skipped 000872 1787843135.829886 trunc from 0x011d2c to 0x017a2d (0x5d02 bytes) 000873 1787843135.969740 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 11: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 13: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 11: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 13: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17a2e: skipped 000875 1787843136.505446 mapread 0x00bbe0 thru 0x01392b (0x7d4c bytes) 000877 1787843136.538181 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 11: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 11: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17a2e: skipped 000878 1787843137.151102 read 0x0113d0 thru 0x017a2d (0x665e bytes) 000879 1787843137.165430 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17a2e): Mirror 5: 0xe64644de (stale) Mirror 9: 0xe64644de (stale) Mirror 12: 0xe64644de (stale) Mirror 14: 0xe9be2948 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17a2e: skipped 000880 1787843137.679012 read 0x000a26 thru 0x00baab (0xb086 bytes) 000882 1787843137.699452 write 0x02fbc1 thru 0x033d36 (0x4176 bytes) 000883 1787843137.707063 mapwrite 0x019d10 thru 0x026e07 (0xd0f8 bytes) 000885 1787843137.862620 trunc from 0x033d37 to 0x02fd45 (0x3ff1 bytes) 000886 1787843137.890928 write 0x015f1a thru 0x0164a6 (0x58d bytes) 000887 1787843137.897397 read 0x0178f1 thru 0x024294 (0xc9a4 bytes) 000888 1787843137.899564 read 0x015412 thru 0x01749e (0x208d bytes) 000889 1787843137.907957 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 5: 0xe2871cb9 (stale) Mirror 9: 0xe2871cb9 (stale) Mirror 12: 0xe2871cb9 (stale) Mirror 14: 0x2c6223f0 (stale) CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 1: 0x5871d6e7 Mirror 15: 0x5871d6e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd46: skipped 000891 1787843139.419261 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 5: 0xe2871cb9 (stale) Mirror 9: 0xe2871cb9 (stale) Mirror 14: 0x2c6223f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 5: 0xe2871cb9 (stale) Mirror 9: 0xe2871cb9 (stale) Mirror 14: 0x2c6223f0 (stale) CRC-32 checksum value for chunk [0, 0x2fd46): Mirror 1: 0x5871d6e7 Mirror 15: 0x5871d6e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd46: skipped 000893 1787843140.155430 read 0x017f4a thru 0x025a36 (0xdaed bytes) 000894 1787843140.169926 write 0x023e85 thru 0x027ba0 (0x3d1c bytes) 000895 1787843140.333295 write 0x000c8e thru 0x00db74 (0xcee7 bytes) 000896 1787843140.348448 mapread 0x01130b thru 0x01a836 (0x952c bytes) 000897 1787843140.374937 mapread 0x00195a thru 0x00b5b3 (0x9c5a bytes) 000898 1787843140.375999 trunc from 0x02fd46 to 0x02b41e (0x4927 bytes) 000900 1787843140.402266 mapwrite 0x00f22f thru 0x01c1da (0xcfac bytes) 000901 1787843140.489191 mapread 0x000a9d thru 0x00c9bc (0xbf20 bytes) 000903 1787843140.490710 write 0x00addb thru 0x00e7cc (0x39f2 bytes) 000904 1787843140.493870 read 0x000620 thru 0x00b419 (0xadfa bytes) 000905 1787843140.494872 trunc from 0x02b41f to 0x01b85d (0xfbc1 bytes) 000906 1787843140.515097 mapwrite 0x031996 thru 0x03e96a (0xcfd5 bytes) 000911 1787843140.600750 write 0x00a624 thru 0x00d949 (0x3326 bytes) 000912 1787843140.603741 mapread 0x016755 thru 0x01a95b (0x4207 bytes) 000916 1787843140.607134 write 0x004490 thru 0x00b2bf (0x6e30 bytes) 000917 1787843140.609126 read 0x00d826 thru 0x01a052 (0xc82d bytes) 000918 1787843140.610244 read 0x03bc2b thru 0x03e96a (0x2d40 bytes) 000921 1787843140.612268 write 0x03bc05 thru 0x03ffff (0x43fb bytes) 000922 1787843140.613880 write 0x018d93 thru 0x01f769 (0x69d7 bytes) 000923 1787843140.616424 trunc from 0x040000 to 0x011d01 (0x2e2fe bytes) 000924 1787843140.632273 mapwrite 0x017f93 thru 0x019f43 (0x1fb1 bytes) 000926 1787843140.688875 mapwrite 0x01b15f thru 0x0236a9 (0x854b bytes) 000928 1787843140.780084 read 0x01c0ea thru 0x0236a9 (0x75c0 bytes) 000929 1787843140.786047 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x236aa): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x236aa): Mirror 5: 0x627c60cf (stale) Mirror 9: 0x627c60cf (stale) Mirror 14: 0x57337048 (stale) Mirror 15: 0x747b43a8 (stale) CRC-32 checksum value for chunk [0, 0x236aa): Mirror 1: 0x396ec2a6 Mirror 16: 0x396ec2a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x236aa: skipped 000931 1787843142.065283 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x170:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x236aa): Mirror 5: 0x627c60cf (stale) Mirror 9: 0x627c60cf (stale) Mirror 14: 0x57337048 (stale) Mirror 15: 0x747b43a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x236aa): Mirror 5: 0x627c60cf (stale) Mirror 9: 0x627c60cf (stale) Mirror 14: 0x57337048 (stale) Mirror 15: 0x747b43a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x236aa: skipped 000932 1787843142.484089 write 0x034ab7 thru 0x03cb7d (0x80c7 bytes) 000935 1787843142.495573 mapread 0x005704 thru 0x011c11 (0xc50e bytes) 000936 1787843142.523257 write 0x03f839 thru 0x03ffff (0x7c7 bytes) 000938 1787843142.525435 mapwrite 0x015228 thru 0x016795 (0x156e bytes) 000940 1787843142.579796 mapwrite 0x0364e9 thru 0x038434 (0x1f4c bytes) 000941 1787843142.621491 MIRROR_RESYNC 5 mirrors 000943 1787843144.336126 mapread 0x01fa10 thru 0x022e79 (0x346a bytes) 000945 1787843144.370160 trunc from 0x040000 to 0x00428e (0x3bd71 bytes) 000947 1787843144.464322 read 0x000fc1 thru 0x00428e (0x32ce bytes) 000949 1787843144.477648 MIRROR_RESYNC 5 mirrors 000950 1787843145.407612 mapwrite 0x01a728 thru 0x022af0 (0x83c9 bytes) 000951 1787843145.565739 read 0x01967e thru 0x022af0 (0x9473 bytes) 000952 1787843145.567350 mapread 0x00731d thru 0x011787 (0xa46b bytes) 000953 1787843145.585513 write 0x023268 thru 0x02a7d6 (0x756f bytes) 000954 1787843145.592354 MIRROR_RESYNC 5 mirrors 000955 1787843146.906346 mapread 0x019b70 thru 0x02995d (0xfdee bytes) 000956 1787843146.980578 trunc from 0x02a7d7 to 0x01dbc8 (0xcc0e bytes) 000957 1787843147.118870 mapread 0x01b2f5 thru 0x01dbc8 (0x28d4 bytes) 000958 1787843147.137376 trunc from 0x01dbc9 to 0x02748b (0x98c3 bytes) 000959 1787843147.179086 write 0x036ee8 thru 0x03d710 (0x6829 bytes) 000960 1787843147.180585 trunc from 0x03d711 to 0x017ff7 (0x25719 bytes) 000964 1787843147.207188 mapwrite 0x002a16 thru 0x00c6b2 (0x9c9d bytes) 000966 1787843147.340161 write 0x01ab56 thru 0x01f7bf (0x4c6a bytes) 000967 1787843147.344301 write 0x003e1c thru 0x0065b0 (0x2795 bytes) 000970 1787843147.351699 write 0x01428c thru 0x019784 (0x54f9 bytes) 000974 1787843147.367462 read 0x006894 thru 0x00b555 (0x4cc2 bytes) 000975 1787843147.371269 mapwrite 0x03f5c3 thru 0x03ffff (0xa3d bytes) 000976 1787843147.529276 write 0x0314d1 thru 0x038718 (0x7248 bytes) 000977 1787843147.535625 write 0x01ee9b thru 0x02684b (0x79b1 bytes) 000979 1787843147.552893 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfbbdf25a (stale) Mirror 9: 0xfbbdf25a (stale) Mirror 14: 0xfbbdf25a (stale) Mirror 15: 0xfbbdf25a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8af2cd0 Mirror 16: 0xd8af2cd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1787843148.802782 mapread 0x010fb8 thru 0x016f88 (0x5fd1 bytes) 000983 1787843148.827539 write 0x03a473 thru 0x03ffff (0x5b8d bytes) 000984 1787843148.923304 mapwrite 0x005e11 thru 0x012c8f (0xce7f bytes) 000985 1787843148.999555 mapwrite 0x00d556 thru 0x017c5e (0xa709 bytes) 000986 1787843149.063980 MIRROR_RESYNC 6 mirrors 000987 1787843150.920531 read 0x01de07 thru 0x02d6d4 (0xf8ce bytes) 000991 1787843150.939213 mapwrite 0x010a9a thru 0x013cc2 (0x3229 bytes) 000992 1787843151.270984 mapwrite 0x03440f thru 0x03a4be (0x60b0 bytes) 000993 1787843151.364056 mapread 0x0131e0 thru 0x0220e7 (0xef08 bytes) 000995 1787843151.400160 read 0x0127f8 thru 0x014055 (0x185e bytes) 000997 1787843151.401270 mapwrite 0x000346 thru 0x00830a (0x7fc5 bytes) 000998 1787843151.535175 trunc from 0x040000 to 0x01a34f (0x25cb0 bytes) 000999 1787843151.572416 mapread 0x0182a3 thru 0x01a34f (0x20ad bytes) 001000 1787843151.574427 read 0x015818 thru 0x015f80 (0x769 bytes) All operations completed A-OK! PASS 70a (112s) == sanity-flr test 71: check mirror extend parent fid ==== 11:05:58 (1787843158) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 11:06:02 (1787843162) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283958 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142719 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:06:13 (1787843173) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4861440 .. Extending file size to 7024384 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8188224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8035712 .. Extending file size to 8323264 .. Extending file size to 8806880 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807680 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9159232 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58592 58593 58594 58596 58597 done 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:07:38 (1787843258) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2943136 .. Extending file size to 7115168 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8455392 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7626400 .. Extending file size to 8763712 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9240064 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8812832 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60103 60104 60105 60108 60109 failed 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:09:00 (1787843340) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198438 s, 10.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.470706 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xbc36c3d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:09:26 (1787843366) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0979684 s, 21.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 11:09:35 (1787843375) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:09:43 (1787843383) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167054 s, 12.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:09:52 (1787843392) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.231123 s, 9.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:10:02 (1787843402) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233279 s, 9.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:10:38 (1787843438) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190901 s, 5.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:11:42 (1787843502) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.323813 s, 6.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:12:48 (1787843568) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. Creating new pool test_204f oleg130-server: Pool lustre.test_204f created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275773 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg130-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:14:12 (1787843652) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:14:20 (1787843660) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:14:28 (1787843668) oleg130-server: Pool lustre.test_206 created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg130-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:14:54 (1787843694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10666 s, 9.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070269 s, 14.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x65fc2c81 Mirror 3: 0x65fc2c81 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:15:06 (1787843706) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:15:11 (1787843711) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:15:15 (1787843715) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.205446 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2270000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x14d0000 Read this (4 bytes): BBBB 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:15:56 (1787843756) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0383428 s, 214 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:16:03 (1787843763) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056313 s, 18.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e5 lmm_fid: [0x200000404:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e5 lmm_fid: [0x200000404:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834405 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 11:16:05.000000000 -0400 Modify: 2026-08-27 11:16:05.000000000 -0400 Change: 2026-08-27 11:16:07.000000000 -0400 Birth: 2026-08-27 11:16:05.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:16:12 (1787843772) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:16:15 (1787843775) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.441703 s, 92.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0459906 s, 89.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 11:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 11:16 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:16:24 (1787843784) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:16:32 (1787843792) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1907 sec =========== 11:16:41 (1787843801) === sanity-flr: start cleanup 11:16:43 (1787843803) === === sanity-flr: finish cleanup 11:16:50 (1787843810) ===