-----============= acceptance-small: sanity-flr ============----- Thu Jul 30 02:28:43 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg320-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:29:14 (1785392954) === oleg320-client.virtnet: executing check_config_client /mnt/lustre oleg320-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg320-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c2ed2aba000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c2ed2aba000.idle_timeout=debug disable quota as required oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:29:59 (1785392999) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7635] == sanity-flr test 0a: lfs mirror create with -N option == 02:30:02 (1785393002) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:30:25 (1785393025) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:30:30 (1785393030) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:30:35 (1785393035) PASS 0d (34s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:31:08 (1785393068) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:31:13 (1785393073) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 02:31:18 (1785393078) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c2ed2aba000.stats=clear osc.lustre-OST0001-osc-ffff8c2ed2aba000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:31:28 (1785393088) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:31:43 (1785393103) PASS 0j (14s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:31:57 (1785393117) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:32:01 (1785393121) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:32:12 (1785393132) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 02:32:24 (1785393144) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:32:33 (1785393153) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:32:38 (1785393158) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:32:46 (1785393166) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:33:01 (1785393181) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:33:10 (1785393190) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 1.09111 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244727 s, 4.3 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:33:22 (1785393202) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.561919 s, 35.5 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:33:34 (1785393214) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:33:46 (1785393226) Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (94s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:35:21 (1785393321) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.4318 s, 14.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:36:15 (1785393375) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (124s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:38:19 (1785393499) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 9.01624 s, 5.8 MB/s -rw-r--r-- 1 root root 52428800 Jul 30 02:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.358s user 0m0.001s sys 0m0.130s Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.547s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m2.157s user 0m0.002s sys 0m0.169s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg320-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2ed2aba000.ost_server_uuid in FULL state after 0 sec PASS 33b (130s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:40:29 (1785393629) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:40:35 (1785393635) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (6s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:40:41 (1785393641) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (7s) == sanity-flr test 35: allow to write to mirrored files == 02:40:47 (1785393647) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 02:40:58 (1785393658) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.77671 s, 16.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:41:14 (1785393674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548059 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056317 s, 18.6 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:41:23 (1785393683) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:41:34 (1785393694) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103472 s, 990 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00322185 s, 318 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00087223 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00292263 s, 350 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 02:41:48 (1785393708) create test files with size 15729158 .. e4f49a0bfba28767c6a092e154632a16 - 7bed657d11fa6c0fcb577d310ccad4a8 - 54af8c6f188faa7f9641ec4a200f056f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre PASS 37 (34s) == sanity-flr test 38: resync ============================ 02:42:22 (1785393742) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:42:41 (1785393761) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:42:49 (1785393769) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158122 s, 6.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120064 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 02:43:01 (1785393781) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0923541 s, 22.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.267228 s, 15.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.411077 s, 5.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (28s) == sanity-flr test 42: lfs mirror verify ================= 02:43:29 (1785393809) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (6s) == sanity-flr test 43a: mirror pick on write ============= 02:43:34 (1785393814) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:43:39 (1785393819) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317954 s, 33.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (13s) == sanity-flr test 44a: lfs mirror split check =========== 02:43:52 (1785393832) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:44:00 (1785393840) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (16s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:44:19 (1785393859) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.489995 s, 21.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:44:35 (1785393875) llite.lustre-ffff8c2ed2b09000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.505562 s, 20.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 llite.lustre-ffff8c2ed86e4800.xattr_cache=1 PASS 44e (22s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:44:57 (1785393897) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (63s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:46:00 (1785393960) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (61s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:47:01 (1785394021) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 02:47:07 (1785394027) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.399127 s, 7.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.347473 s, 9.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x951d495f (nosync,stale) Mirror 2: 0x951d495f (nosync,stale) ** verify mirror 2 doesn't change original checksum: afb3cfd2704f4fbdd7928fe5f194f871 - mirror 1 checksum: aa9d27176a332e23d6cadf93b628a235 - mirror 2 checksum: afb3cfd2704f4fbdd7928fe5f194f871 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T02:47:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T02:47:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: afb3cfd2704f4fbdd7928fe5f194f871 - mirror 1 checksum: aa9d27176a332e23d6cadf93b628a235 - mirror 2 checksum: aa9d27176a332e23d6cadf93b628a235 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T02:47:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T02:47:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (17s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:47:27 (1785394047) Failing mds1 on oleg320-server Stopping /mnt/lustre-mds1 (opts:) on oleg320-server 02:47:36 (1785394056) shut down facet: mds1 facet_host: oleg320-server facet_failover_host: oleg320-server Failover mds1 to oleg320-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 02:48:13 (1785394093) targets are mounted 02:48:13 (1785394093) facet_failover done oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594109 s, 17.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (71s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:48:37 (1785394117) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:48:42 (1785394122) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:48:47 (1785394127) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0228025 s, 719 kB/s PASS 50c (14s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:49:01 (1785394141) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:49:07 (1785394147) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00344637 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.100168 s, 40.9 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:49:18 (1785394158) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (33s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:49:52 (1785394192) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:50:25 (1785394225) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:50:51 (1785394251) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:51:20 (1785394280) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 02:51:29 (1785394289) mirror create pid 54319 mirror split pid 54320 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54319 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54320 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 02:52:40 (1785394360) Seed set to 4363 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000004 1785394364.604503 MIRROR_EXTEND to 2 mirrors 000005 1785394365.463166 write 0x018807 thru 0x01c689 (0x3e83 bytes) 000006 1785394365.608157 write 0x01fa27 thru 0x02893b (0x8f15 bytes) 000007 1785394365.609627 mapwrite 0x02cc84 thru 0x02e678 (0x19f5 bytes) 000009 1785394365.894876 read 0x000f44 thru 0x00bab6 (0xab73 bytes) 000010 1785394365.919329 mapwrite 0x024b1a thru 0x033e2a (0xf311 bytes) 000011 1785394366.153794 write 0x01ab49 thru 0x01cb92 (0x204a bytes) 000012 1785394366.172592 mapread 0x03022b thru 0x033e2a (0x3c00 bytes) 000013 1785394366.187037 MIRROR_RESYNC 2 mirrors 000015 1785394368.576278 read 0x022db4 thru 0x02b127 (0x8374 bytes) 000017 1785394368.620333 write 0x02885e thru 0x03305b (0xa7fe bytes) 000018 1785394368.937415 mapread 0x00e910 thru 0x01a6aa (0xbd9b bytes) 000019 1785394369.031476 read 0x00cfd5 thru 0x0141f5 (0x7221 bytes) truncating to largest ever: 0x18412 000021 1785394369.032757 trunc from 0x033e2b to 0x018411 (0x1ba19 bytes) 000022 1785394369.094260 write 0x039a27 thru 0x03ffff (0x65d9 bytes) 000025 1785394369.107218 mapread 0x02b343 thru 0x02ea81 (0x373f bytes) 000028 1785394369.138673 write 0x03d7d9 thru 0x03ffff (0x2827 bytes) truncating to largest ever: 0x3c9a7 000031 1785394369.154451 trunc from 0x040000 to 0x03c9a6 (0x3659 bytes) 000034 1785394369.194052 write 0x0357a1 thru 0x036375 (0xbd5 bytes) 000035 1785394369.217081 MIRROR_RESYNC 2 mirrors 000037 1785394371.275521 MIRROR_EXTEND to 3 mirrors 000038 1785394373.383784 write 0x028173 thru 0x030638 (0x84c6 bytes) 000039 1785394373.642316 read 0x00ccb1 thru 0x012929 (0x5c79 bytes) 000040 1785394373.683037 read 0x026a03 thru 0x0350c8 (0xe6c6 bytes) 000043 1785394373.724443 read 0x0180bf thru 0x026d82 (0xecc4 bytes) 000044 1785394373.725325 mapwrite 0x029ff8 thru 0x035a36 (0xba3f bytes) 000046 1785394374.015483 trunc from 0x03c9a7 to 0x038d87 (0x3c1f bytes) 000048 1785394374.147188 mapread 0x02e74f thru 0x038d87 (0xa639 bytes) 000049 1785394374.155728 trunc from 0x038d88 to 0x01959c (0x1f7eb bytes) 000050 1785394374.231598 write 0x0157aa thru 0x022f45 (0xd79c bytes) 000055 1785394374.282656 mapread 0x009d52 thru 0x00c525 (0x27d4 bytes) 000057 1785394374.284276 write 0x01300d thru 0x0138b1 (0x8a5 bytes) 000058 1785394374.285271 read 0x00f3b7 thru 0x01b4f3 (0xc13d bytes) 000059 1785394374.304700 mapwrite 0x015e2d thru 0x01fe13 (0x9fe7 bytes) 000060 1785394374.486468 trunc from 0x022f46 to 0x017059 (0xbeec bytes) 000061 1785394374.570729 mapread 0x014de3 thru 0x017059 (0x2277 bytes) 000062 1785394374.580485 write 0x003847 thru 0x00b05a (0x7814 bytes) 000064 1785394374.593444 trunc from 0x01705a to 0x039436 (0x223dd bytes) 000065 1785394374.657296 read 0x026377 thru 0x02aeee (0x4b78 bytes) 000067 1785394374.684738 mapread 0x02d8ed thru 0x033a3c (0x6150 bytes) 000070 1785394374.702750 read 0x024b80 thru 0x02ed01 (0xa182 bytes) 000072 1785394374.729187 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39437): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39437): Mirror 2: 0x982fcf77 (stale) Mirror 3: 0x982fcf77 (stale) CRC-32 checksum value for chunk [0, 0x39437): Mirror 1: 0xd0907747 Mirror 4: 0xd0907747 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39437: skipped 000073 1785394377.433326 mapwrite 0x00f27f thru 0x00f84f (0x5d1 bytes) truncating to largest ever: 0x3d40c 000074 1785394377.742542 trunc from 0x039437 to 0x03d40b (0x3fd5 bytes) 000077 1785394377.824571 write 0x032512 thru 0x03cbc1 (0xa6b0 bytes) 000078 1785394377.848380 trunc from 0x03d40c to 0x0100f8 (0x2d313 bytes) 000079 1785394377.878048 trunc from 0x0100f9 to 0x027792 (0x1769a bytes) 000080 1785394377.921037 MIRROR_RESYNC 4 mirrors 000081 1785394380.206343 mapread 0x00e34a thru 0x01364b (0x5302 bytes) 000082 1785394380.292254 mapread 0x02525b thru 0x027792 (0x2538 bytes) 000083 1785394380.304712 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000084 1785394381.244512 read 0x022b91 thru 0x027792 (0x4c02 bytes) 000085 1785394381.301706 mapwrite 0x006233 thru 0x011ac1 (0xb88f bytes) 000088 1785394381.815992 trunc from 0x027793 to 0x0334b3 (0xbd21 bytes) 000089 1785394381.948436 trunc from 0x0334b4 to 0x011550 (0x21f63 bytes) 000090 1785394381.982605 mapread 0x00c1b0 thru 0x011550 (0x53a1 bytes) 000091 1785394381.985957 mapread 0x00c45d thru 0x011550 (0x50f4 bytes) 000092 1785394381.987382 trunc from 0x011551 to 0x0268ad (0x1535d bytes) 000094 1785394382.013615 mapread 0x00a256 thru 0x011b8a (0x7935 bytes) 000095 1785394382.014923 trunc from 0x0268ae to 0x033901 (0xd054 bytes) 000096 1785394382.057835 trunc from 0x033902 to 0x017465 (0x1c49c bytes) 000098 1785394382.082924 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17466): Mirror 4: 0x6ce11460 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17466): Mirror 4: 0x6ce11460 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17466: skipped 000099 1785394382.754230 read 0x005017 thru 0x0052fd (0x2e7 bytes) 000100 1785394382.781557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000101 1785394383.136001 write 0x0263b0 thru 0x02960c (0x325d bytes) 000103 1785394383.149973 write 0x0129d5 thru 0x01e377 (0xb9a3 bytes) 000106 1785394383.171567 read 0x004f52 thru 0x00c245 (0x72f4 bytes) 000107 1785394383.206675 trunc from 0x02960d to 0x01320a (0x16402 bytes) 000109 1785394383.256178 trunc from 0x01320b to 0x00d242 (0x5fc8 bytes) 000112 1785394383.286534 mapread 0x0086a8 thru 0x00d242 (0x4b9b bytes) 000113 1785394383.307635 write 0x030215 thru 0x037d40 (0x7b2c bytes) 000114 1785394383.312621 mapread 0x010019 thru 0x01eb96 (0xeb7e bytes) 000115 1785394383.356639 write 0x02b697 thru 0x0336d5 (0x803f bytes) 000116 1785394383.369903 mapwrite 0x025f1e thru 0x026b22 (0xc05 bytes) 000118 1785394383.530144 MIRROR_EXTEND to 2 mirrors 000119 1785394384.619371 read 0x02cd1b thru 0x0340e5 (0x73cb bytes) 000121 1785394384.640843 MIRROR_RESYNC 2 mirrors 000122 1785394385.170069 read 0x01d779 thru 0x02876b (0xaff3 bytes) 000123 1785394385.211253 mapwrite 0x03f530 thru 0x03ffff (0xad0 bytes) 000125 1785394385.557904 read 0x00e70c thru 0x01c9b8 (0xe2ad bytes) 000126 1785394385.600516 write 0x01a335 thru 0x024b7b (0xa847 bytes) 000127 1785394385.628445 trunc from 0x040000 to 0x01451f (0x2bae0 bytes) 000129 1785394385.728911 trunc from 0x014520 to 0x027e11 (0x138f2 bytes) 000130 1785394385.818189 mapread 0x0217c7 thru 0x027e11 (0x664b bytes) 000131 1785394385.961720 mapwrite 0x02e284 thru 0x02ea79 (0x7f6 bytes) 000133 1785394386.246259 mapwrite 0x02738c thru 0x02da45 (0x66ba bytes) 000134 1785394386.566490 MIRROR_RESYNC 2 mirrors 000135 1785394388.234714 mapwrite 0x013cb5 thru 0x01c674 (0x89c0 bytes) 000136 1785394388.462334 write 0x022268 thru 0x027677 (0x5410 bytes) 000138 1785394388.487402 read 0x020657 thru 0x02ea79 (0xe423 bytes) 000140 1785394388.513899 mapwrite 0x02f3fe thru 0x03b4e1 (0xc0e4 bytes) 000142 1785394388.693744 trunc from 0x03b4e2 to 0x014864 (0x26c7d bytes) 000143 1785394388.729369 mapwrite 0x02cc0b thru 0x03c4bf (0xf8b5 bytes) 000144 1785394388.876163 read 0x03aa47 thru 0x03c4bf (0x1a79 bytes) 000148 1785394388.885928 mapread 0x033e18 thru 0x037527 (0x3710 bytes) 000149 1785394388.910448 MIRROR_RESYNC 2 mirrors 000150 1785394390.743476 mapread 0x000a7c thru 0x00f451 (0xe9d6 bytes) 000151 1785394390.839618 mapwrite 0x038f5f thru 0x03ffff (0x70a1 bytes) 000153 1785394391.207835 trunc from 0x040000 to 0x01e976 (0x21689 bytes) 000154 1785394391.302339 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e977): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e977): Mirror 2: 0xef7dd5a4 (stale) CRC-32 checksum value for chunk [0, 0x1e977): Mirror 1: 0xef7dd5a4 Mirror 3: 0xef7dd5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e977: skipped 000156 1785394393.396707 mapwrite 0x03ef3a thru 0x03ffff (0x10c6 bytes) 000158 1785394393.854666 read 0x03966e thru 0x03ffff (0x6992 bytes) 000159 1785394393.878148 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe45caa61 (stale) Mirror 3: 0x5f2f9aac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf617ced2 Mirror 4: 0xf617ced2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000161 1785394395.685280 mapread 0x005fca thru 0x012c13 (0xcc4a bytes) 000162 1785394395.803136 read 0x01a07c thru 0x02716e (0xd0f3 bytes) 000164 1785394395.843257 mapwrite 0x003f36 thru 0x004ab1 (0xb7c bytes) 000165 1785394396.075201 mapread 0x0072d5 thru 0x0138d3 (0xc5ff bytes) 000166 1785394396.122217 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe45caa61 (stale) Mirror 3: 0x5f2f9aac (stale) Mirror 4: 0xf617ced2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1fbcb277 Mirror 5: 0x1fbcb277 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000168 1785394397.805226 trunc from 0x040000 to 0x00ef37 (0x310c8 bytes) 000169 1785394398.032091 write 0x03570b thru 0x039384 (0x3c7a bytes) 000170 1785394398.045312 MIRROR_RESYNC 5 mirrors 000171 1785394400.732170 mapwrite 0x03f4ad thru 0x03ffff (0xb53 bytes) 000172 1785394400.989458 mapwrite 0x02372a thru 0x0251a4 (0x1a7b bytes) 000173 1785394401.093109 trunc from 0x040000 to 0x02496b (0x1b694 bytes) truncating to largest ever: 0x3e1e2 000174 1785394401.135645 trunc from 0x02496c to 0x03e1e1 (0x19876 bytes) 000175 1785394401.164177 trunc from 0x03e1e2 to 0x0288fd (0x158e4 bytes) 000176 1785394401.190769 mapread 0x00fd76 thru 0x01c9c0 (0xcc4b bytes) truncating to largest ever: 0x3ffe1 000178 1785394401.229621 trunc from 0x0288fe to 0x03ffe0 (0x176e3 bytes) 000179 1785394401.266331 trunc from 0x03ffe1 to 0x03e505 (0x1adb bytes) 000180 1785394401.292580 mapread 0x038130 thru 0x03e505 (0x63d6 bytes) 000182 1785394401.307756 mapwrite 0x02fa36 thru 0x032ffe (0x35c9 bytes) 000183 1785394401.406495 write 0x02fcf1 thru 0x0346d3 (0x49e3 bytes) 000184 1785394401.434390 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e506): Mirror 2: 0x4e36656 (stale) Mirror 3: 0x4e36656 (stale) Mirror 4: 0x4e36656 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e506): Mirror 2: 0x4e36656 (stale) Mirror 3: 0x4e36656 (stale) Mirror 4: 0x4e36656 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e506: skipped 000186 1785394402.176056 write 0x03e15b thru 0x03ef1b (0xdc1 bytes) 000187 1785394402.189283 trunc from 0x03ef1c to 0x022529 (0x1c9f2 bytes) 000188 1785394402.214701 mapwrite 0x027d8f thru 0x02b694 (0x3906 bytes) 000191 1785394402.477189 mapwrite 0x01c76c thru 0x028912 (0xc1a7 bytes) 000193 1785394402.604682 mapread 0x022abb thru 0x022d6f (0x2b5 bytes) 000194 1785394402.615015 mapread 0x000da2 thru 0x00ac7c (0x9edb bytes) 000196 1785394402.641966 mapwrite 0x0235d9 thru 0x02f49b (0xbec3 bytes) 000198 1785394402.767903 read 0x00cdf5 thru 0x01297f (0x5b8b bytes) 000199 1785394402.779028 write 0x0276ca thru 0x0375d0 (0xff07 bytes) 000200 1785394402.793214 mapread 0x01ab05 thru 0x027abb (0xcfb7 bytes) 000202 1785394402.795183 read 0x0291ff thru 0x0375d0 (0xe3d2 bytes) 000204 1785394402.814023 read 0x013809 thru 0x015d8a (0x2582 bytes) 000205 1785394402.824928 mapread 0x029a6e thru 0x02cb90 (0x3123 bytes) 000206 1785394402.828461 mapwrite 0x0001cd thru 0x000365 (0x199 bytes) 000207 1785394403.040353 MIRROR_RESYNC 4 mirrors 000211 1785394404.881385 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000212 1785394405.421316 trunc from 0x0375d1 to 0x0347b8 (0x2e18 bytes) 000214 1785394405.606038 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x347b9): Mirror 2: 0x72295f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347b9): Mirror 2: 0x72295f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347b9: skipped 000215 1785394406.259148 write 0x01880a thru 0x01cb8d (0x4384 bytes) 000216 1785394406.307479 mapwrite 0x01fe12 thru 0x02ee3a (0xf029 bytes) 000217 1785394406.492057 mapwrite 0x01d60a thru 0x029c85 (0xc67c bytes) 000219 1785394406.647788 mapwrite 0x006d52 thru 0x00ac01 (0x3eb0 bytes) 000221 1785394406.911003 mapwrite 0x00ada2 thru 0x00f5b6 (0x4815 bytes) 000222 1785394407.070444 trunc from 0x0347b9 to 0x031882 (0x2f36 bytes) 000223 1785394407.128510 write 0x00f275 thru 0x015ceb (0x6a77 bytes) 000224 1785394407.140819 mapwrite 0x00355c thru 0x00a69e (0x7143 bytes) 000226 1785394407.268739 mapwrite 0x00388a thru 0x008710 (0x4e87 bytes) 000227 1785394407.462072 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000230 1785394407.873098 trunc from 0x031883 to 0x02a467 (0x741b bytes) 000231 1785394407.903389 write 0x01b162 thru 0x0269ab (0xb84a bytes) 000232 1785394407.948521 mapread 0x00bbdb thru 0x016199 (0xa5bf bytes) 000233 1785394408.012836 mapread 0x023612 thru 0x02a467 (0x6e56 bytes) 000237 1785394408.039587 write 0x010beb thru 0x01a5b2 (0x99c8 bytes) 000238 1785394408.044191 mapread 0x002fe8 thru 0x011afc (0xeb15 bytes) 000239 1785394408.059786 write 0x0045a8 thru 0x00791e (0x3377 bytes) 000241 1785394408.061400 mapwrite 0x0089c0 thru 0x014cc9 (0xc30a bytes) 000242 1785394408.176355 read 0x01065b thru 0x01af8f (0xa935 bytes) 000244 1785394408.177486 mapread 0x008fb8 thru 0x01106a (0x80b3 bytes) 000245 1785394408.178474 read 0x012471 thru 0x020ebf (0xea4f bytes) 000247 1785394408.191484 read 0x008d39 thru 0x018a64 (0xfd2c bytes) 000248 1785394408.193464 read 0x0185ac thru 0x01d732 (0x5187 bytes) 000250 1785394408.202114 write 0x03e06d thru 0x03ffff (0x1f93 bytes) 000254 1785394408.211521 write 0x020d04 thru 0x029ac2 (0x8dbf bytes) 000255 1785394408.214046 read 0x017afb thru 0x01d6e6 (0x5bec bytes) 000256 1785394408.229061 trunc from 0x040000 to 0x02c722 (0x138dd bytes) 000257 1785394408.266147 mapread 0x02b3a4 thru 0x02c722 (0x137f bytes) 000261 1785394408.309249 trunc from 0x02c723 to 0x036236 (0x9b14 bytes) 000262 1785394408.340409 mapwrite 0x00a0e1 thru 0x00cac1 (0x29e1 bytes) 000264 1785394408.525755 mapread 0x018eb4 thru 0x01f4eb (0x6638 bytes) 000266 1785394408.535959 mapwrite 0x02998c thru 0x02c9dd (0x3052 bytes) 000267 1785394408.661468 trunc from 0x036237 to 0x01abb4 (0x1b682 bytes) 000270 1785394408.719428 mapwrite 0x00c20d thru 0x00e3a8 (0x219c bytes) 000273 1785394408.900257 MIRROR_EXTEND to 2 mirrors 000275 1785394410.415361 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000277 1785394410.663194 trunc from 0x01abb5 to 0x02ecfe (0x1414a bytes) 000278 1785394410.722943 read 0x02e5f3 thru 0x02ecfe (0x70c bytes) 000280 1785394410.744516 mapwrite 0x027299 thru 0x03379c (0xc504 bytes) 000281 1785394410.974645 mapread 0x0214ef thru 0x029ebf (0x89d1 bytes) 000283 1785394411.002569 write 0x02e293 thru 0x033045 (0x4db3 bytes) 000284 1785394411.014342 read 0x024037 thru 0x024acd (0xa97 bytes) 000285 1785394411.015523 read 0x01545d thru 0x01ca36 (0x75da bytes) 000286 1785394411.045431 read 0x008383 thru 0x011d15 (0x9993 bytes) 000287 1785394411.055079 trunc from 0x03379d to 0x026709 (0xd093 bytes) 000288 1785394411.104492 write 0x0213a9 thru 0x0248a9 (0x3501 bytes) 000289 1785394411.112476 mapwrite 0x02c634 thru 0x036ca9 (0xa676 bytes) 000290 1785394411.334928 mapread 0x022155 thru 0x03170f (0xf5bb bytes) 000291 1785394411.336586 mapwrite 0x038d54 thru 0x03ffff (0x72ac bytes) 000293 1785394411.494851 write 0x025382 thru 0x02f991 (0xa610 bytes) 000294 1785394411.507527 trunc from 0x040000 to 0x03e06a (0x1f95 bytes) 000296 1785394411.568726 read 0x015102 thru 0x021f88 (0xce87 bytes) 000298 1785394411.573725 mapwrite 0x00e107 thru 0x011806 (0x3700 bytes) 000299 1785394411.638360 read 0x000699 thru 0x00af0a (0xa872 bytes) 000300 1785394411.639013 mapwrite 0x00040c thru 0x0036be (0x32b3 bytes) 000303 1785394411.762824 mapwrite 0x00bb0c thru 0x012797 (0x6c8c bytes) 000304 1785394411.859214 write 0x00631d thru 0x015eb6 (0xfb9a bytes) 000307 1785394411.877239 mapwrite 0x01b812 thru 0x025555 (0x9d44 bytes) 000308 1785394412.025336 trunc from 0x03e06b to 0x02e7a2 (0xf8c8 bytes) 000312 1785394412.065197 mapread 0x002f07 thru 0x0053c4 (0x24be bytes) 000317 1785394412.082426 write 0x0213c1 thru 0x0287a0 (0x73e0 bytes) 000318 1785394412.086468 trunc from 0x02e7a3 to 0x030e23 (0x2681 bytes) 000320 1785394412.121227 mapread 0x001e32 thru 0x010787 (0xe956 bytes) 000322 1785394412.125798 mapread 0x03086f thru 0x030e23 (0x5b5 bytes) 000323 1785394412.138726 mapread 0x01c719 thru 0x022e24 (0x670c bytes) 000324 1785394412.139818 trunc from 0x030e24 to 0x01a5a1 (0x16882 bytes) 000325 1785394412.173174 mapread 0x00c984 thru 0x011075 (0x46f2 bytes) 000326 1785394412.176895 trunc from 0x01a5a2 to 0x00aca2 (0xf8ff bytes) 000327 1785394412.213323 read 0x0033b8 thru 0x00aca2 (0x78eb bytes) 000328 1785394412.214207 mapwrite 0x02c431 thru 0x03105e (0x4c2e bytes) 000329 1785394412.479380 trunc from 0x03105f to 0x0370ca (0x606c bytes) 000332 1785394412.535225 mapread 0x01df0e thru 0x01ed55 (0xe48 bytes) 000335 1785394412.541838 mapwrite 0x01ecb4 thru 0x022d1c (0x4069 bytes) 000336 1785394412.660278 trunc from 0x0370cb to 0x00d8a5 (0x29825 bytes) 000338 1785394412.710423 mapwrite 0x00d0af thru 0x012412 (0x5364 bytes) 000339 1785394412.936678 trunc from 0x012413 to 0x00a00d (0x8405 bytes) 000343 1785394412.999835 mapread 0x007326 thru 0x007b15 (0x7f0 bytes) 000344 1785394413.001729 trunc from 0x00a00e to 0x00a7bb (0x7ae bytes) 000348 1785394413.060853 mapread 0x009c96 thru 0x00a7bb (0xb26 bytes) 000353 1785394413.078031 mapwrite 0x03e35a thru 0x03ffff (0x1ca6 bytes) 000357 1785394413.308686 mapwrite 0x00e9dd thru 0x0118c0 (0x2ee4 bytes) 000358 1785394413.496011 read 0x000de5 thru 0x00981e (0x8a3a bytes) 000359 1785394413.497028 read 0x000da4 thru 0x005433 (0x4690 bytes) 000360 1785394413.501808 mapread 0x011c1a thru 0x0185d3 (0x69ba bytes) 000361 1785394413.518700 trunc from 0x040000 to 0x0141eb (0x2be14 bytes) 000363 1785394413.578076 MIRROR_EXTEND to 2 mirrors 000366 1785394414.993028 write 0x03533f thru 0x03877a (0x343c bytes) 000367 1785394415.159137 mapread 0x01e78c thru 0x01ebc7 (0x43c bytes) 000369 1785394415.177932 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3877b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3877b): Mirror 2: 0xbdd69be (stale) CRC-32 checksum value for chunk [0, 0x3877b): Mirror 1: 0x9f59cae Mirror 3: 0x9f59cae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3877b: skipped 000370 1785394417.333090 read 0x021ae7 thru 0x024fda (0x34f4 bytes) 000375 1785394417.352192 mapwrite 0x03bf70 thru 0x03de29 (0x1eba bytes) 000379 1785394417.651571 write 0x008cf1 thru 0x011f35 (0x9245 bytes) 000381 1785394417.710221 mapread 0x02229d thru 0x02805f (0x5dc3 bytes) 000382 1785394417.732096 read 0x000936 thru 0x003539 (0x2c04 bytes) 000383 1785394417.749682 mapwrite 0x03e244 thru 0x03ffff (0x1dbc bytes) 000385 1785394417.938782 mapwrite 0x01349f thru 0x01cc3a (0x979c bytes) 000386 1785394418.127618 write 0x037a3e thru 0x03d101 (0x56c4 bytes) 000388 1785394418.129283 mapread 0x015b65 thru 0x0248d3 (0xed6f bytes) 000389 1785394418.135393 write 0x026765 thru 0x029c8c (0x3528 bytes) 000390 1785394418.140215 mapread 0x0081d6 thru 0x008905 (0x730 bytes) 000392 1785394418.144297 trunc from 0x040000 to 0x010f3f (0x2f0c0 bytes) 000394 1785394418.197842 write 0x035420 thru 0x03dea9 (0x8a8a bytes) 000395 1785394418.203452 trunc from 0x03deaa to 0x00ff78 (0x2df31 bytes) 000396 1785394418.237280 write 0x034b06 thru 0x0362c6 (0x17c1 bytes) 000398 1785394418.257174 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x362c7): Mirror 3: 0xcb6dd7a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x362c7): Mirror 3: 0xcb6dd7a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x362c7: skipped 000399 1785394418.879496 mapread 0x016a65 thru 0x021884 (0xae20 bytes) 000400 1785394418.958812 mapwrite 0x03cfa9 thru 0x03ffff (0x3057 bytes) 000402 1785394419.122129 read 0x0095f5 thru 0x00b9c0 (0x23cc bytes) 000403 1785394419.136631 mapwrite 0x0028ff thru 0x00e0ed (0xb7ef bytes) 000404 1785394419.254404 write 0x00d57c thru 0x011185 (0x3c0a bytes) 000405 1785394419.255574 mapwrite 0x02db3d thru 0x031781 (0x3c45 bytes) 000406 1785394419.368448 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000408 1785394419.597302 mapwrite 0x008a0e thru 0x0131bb (0xa7ae bytes) 000410 1785394419.741045 read 0x00c29e thru 0x01a690 (0xe3f3 bytes) 000413 1785394419.750508 trunc from 0x040000 to 0x03edce (0x1231 bytes) 000416 1785394419.773713 write 0x03af2f thru 0x03ffff (0x50d1 bytes) 000420 1785394419.787949 write 0x01742c thru 0x023b2a (0xc6ff bytes) 000421 1785394419.823437 mapwrite 0x00d6f7 thru 0x0147d6 (0x70e0 bytes) 000422 1785394419.937602 read 0x00a69c thru 0x01015c (0x5ac1 bytes) 000424 1785394419.939628 mapwrite 0x032fe5 thru 0x03ffff (0xd01b bytes) 000425 1785394420.104146 trunc from 0x040000 to 0x003935 (0x3c6ca bytes) 000426 1785394420.142285 MIRROR_EXTEND to 2 mirrors 000427 1785394421.662776 read 0x001a82 thru 0x003935 (0x1eb4 bytes) 000428 1785394421.684472 trunc from 0x003936 to 0x02472e (0x20df9 bytes) 000432 1785394421.841583 trunc from 0x02472f to 0x038ebf (0x14791 bytes) 000435 1785394421.887942 mapread 0x002ddc thru 0x00a196 (0x73bb bytes) 000436 1785394421.910155 mapwrite 0x00f2cb thru 0x00f77b (0x4b1 bytes) 000437 1785394422.040625 read 0x00b82d thru 0x013d37 (0x850b bytes) 000438 1785394422.055673 mapwrite 0x00dd34 thru 0x01858e (0xa85b bytes) 000439 1785394422.152031 mapwrite 0x000e87 thru 0x003cc5 (0x2e3f bytes) 000440 1785394422.258283 mapread 0x0154e3 thru 0x01b5e0 (0x60fe bytes) 000442 1785394422.259951 mapwrite 0x016dac thru 0x019c0a (0x2e5f bytes) 000443 1785394422.357152 trunc from 0x038ec0 to 0x016fa5 (0x21f1a bytes) 000444 1785394422.394234 write 0x028709 thru 0x032de8 (0xa6e0 bytes) 000445 1785394422.400177 trunc from 0x032de9 to 0x03198e (0x145a bytes) 000446 1785394422.436215 read 0x00ff60 thru 0x01667d (0x671e bytes) 000447 1785394422.437222 read 0x00f008 thru 0x014113 (0x510c bytes) 000448 1785394422.446600 read 0x000090 thru 0x00925a (0x91cb bytes) 000449 1785394422.452911 mapwrite 0x020e81 thru 0x024959 (0x3ad9 bytes) 000450 1785394422.583413 mapwrite 0x00cfa1 thru 0x00f559 (0x25b9 bytes) 000451 1785394422.723795 MIRROR_RESYNC 2 mirrors 000452 1785394423.968251 mapread 0x014a83 thru 0x01a27e (0x57fc bytes) 000453 1785394424.046102 trunc from 0x03198f to 0x028dba (0x8bd4 bytes) 000454 1785394424.172139 write 0x03c158 thru 0x03ffff (0x3ea8 bytes) 000455 1785394424.175802 trunc from 0x040000 to 0x029a70 (0x1658f bytes) 000456 1785394424.224186 mapwrite 0x00cd65 thru 0x0105d4 (0x3870 bytes) 000457 1785394424.364010 MIRROR_RESYNC 2 mirrors 000458 1785394425.127162 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000459 1785394425.242250 mapwrite 0x03ce0a thru 0x03ffff (0x31f6 bytes) 000460 1785394425.321128 write 0x0040e6 thru 0x009deb (0x5d06 bytes) 000462 1785394425.333179 MIRROR_EXTEND to 2 mirrors 000463 1785394425.839470 write 0x017175 thru 0x01b4b9 (0x4345 bytes) 000464 1785394426.025881 mapread 0x01180f thru 0x01f38a (0xdb7c bytes) 000465 1785394426.080149 trunc from 0x040000 to 0x03dcd2 (0x232d bytes) 000466 1785394426.156660 write 0x005bad thru 0x00cd1f (0x7173 bytes) 000467 1785394426.169414 trunc from 0x03dcd3 to 0x00f2ca (0x2ea08 bytes) 000469 1785394426.219083 MIRROR_RESYNC 2 mirrors 000470 1785394427.606240 mapwrite 0x035e87 thru 0x03ffff (0xa179 bytes) 000472 1785394428.078811 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4a9b77ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8953fd0b Mirror 3: 0x8953fd0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1785394429.678913 write 0x03dd91 thru 0x03ffff (0x226f bytes) 000474 1785394429.818535 read 0x001d77 thru 0x00aeda (0x9164 bytes) 000475 1785394429.856217 mapwrite 0x008f02 thru 0x010c06 (0x7d05 bytes) 000477 1785394430.091226 write 0x03ea70 thru 0x03ffff (0x1590 bytes) 000479 1785394430.101100 mapwrite 0x027024 thru 0x02b9a0 (0x497d bytes) 000483 1785394430.236355 trunc from 0x040000 to 0x007464 (0x38b9b bytes) 000484 1785394430.301065 mapread 0x001124 thru 0x00214c (0x1029 bytes) 000485 1785394430.302115 trunc from 0x007465 to 0x026426 (0x1efc2 bytes) 000487 1785394430.356236 MIRROR_RESYNC 3 mirrors 000488 1785394432.285385 mapwrite 0x00d04e thru 0x019789 (0xc73c bytes) 000492 1785394432.751581 mapread 0x008d88 thru 0x009c96 (0xf0f bytes) 000493 1785394432.758114 read 0x014244 thru 0x01519d (0xf5a bytes) 000496 1785394432.771433 write 0x00b45d thru 0x014a85 (0x9629 bytes) 000499 1785394432.795273 MIRROR_RESYNC 3 mirrors 000500 1785394434.715400 MIRROR_RESYNC 3 mirrors 000501 1785394435.411439 mapread 0x008556 thru 0x00c856 (0x4301 bytes) 000502 1785394435.457830 write 0x00e8b9 thru 0x011381 (0x2ac9 bytes) 000503 1785394435.606965 write 0x02174a thru 0x028d82 (0x7639 bytes) 000504 1785394435.622191 write 0x019adc thru 0x01d40c (0x3931 bytes) 000505 1785394435.655808 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28d83): Mirror 3: 0xc9dd99ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d83): Mirror 3: 0xc9dd99ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d83: skipped 000508 1785394436.080380 mapread 0x00d782 thru 0x01576a (0x7fe9 bytes) 000509 1785394436.120375 mapwrite 0x0210fd thru 0x02eeb1 (0xddb5 bytes) 000510 1785394436.300472 write 0x005357 thru 0x00b6fc (0x63a6 bytes) 000511 1785394436.316172 trunc from 0x02eeb2 to 0x01e7c4 (0x106ed bytes) 000512 1785394436.347705 write 0x024943 thru 0x02e0d4 (0x9792 bytes) 000516 1785394436.356407 mapwrite 0x01df55 thru 0x01f62e (0x16da bytes) 000518 1785394436.539128 read 0x00f656 thru 0x00fbbc (0x567 bytes) 000519 1785394436.541237 mapwrite 0x008a54 thru 0x00f930 (0x6edd bytes) 000520 1785394436.663519 read 0x005b69 thru 0x0073ab (0x1843 bytes) 000521 1785394436.668810 mapread 0x01ae3e thru 0x023e46 (0x9009 bytes) 000522 1785394436.676154 write 0x00ada6 thru 0x018c45 (0xdea0 bytes) 000524 1785394436.680945 read 0x00de63 thru 0x0155ec (0x778a bytes) 000525 1785394436.697248 mapwrite 0x019d2c thru 0x023d96 (0xa06b bytes) 000526 1785394436.787326 write 0x00fd57 thru 0x0132ff (0x35a9 bytes) 000527 1785394436.799603 read 0x00155d thru 0x007e1b (0x68bf bytes) 000529 1785394436.820259 mapwrite 0x02c8bc thru 0x03a15f (0xd8a4 bytes) 000530 1785394437.086066 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a160): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a160): Mirror 3: 0xedc40f72 (stale) CRC-32 checksum value for chunk [0, 0x3a160): Mirror 1: 0x9f7d4e0c Mirror 4: 0x9f7d4e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a160: skipped 000531 1785394438.988580 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000532 1785394439.668636 mapread 0x015e19 thru 0x021f31 (0xc119 bytes) 000533 1785394439.786717 trunc from 0x03a160 to 0x01cf81 (0x1d1de bytes) 000534 1785394440.056249 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1cf82): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf82): Mirror 4: 0x8bdb798d (stale) CRC-32 checksum value for chunk [0, 0x1cf82): Mirror 1: 0x8bdb798d Mirror 5: 0x8bdb798d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf82: skipped 000535 1785394441.875018 MIRROR_RESYNC 3 mirrors 000536 1785394443.271974 read 0x019d90 thru 0x01cf81 (0x31f2 bytes) 000537 1785394443.294257 read 0x018cb2 thru 0x01cf81 (0x42d0 bytes) 000539 1785394443.313144 MIRROR_RESYNC 3 mirrors 000540 1785394444.146780 read 0x00fa52 thru 0x01bae1 (0xc090 bytes) 000541 1785394444.147358 write 0x01b861 thru 0x02b7a8 (0xff48 bytes) 000542 1785394444.401913 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b7a9): Mirror 5: 0x647d102a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b7a9): Mirror 5: 0x647d102a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b7a9: skipped 000544 1785394445.030073 write 0x00e2f8 thru 0x00f9e3 (0x16ec bytes) 000545 1785394445.061481 trunc from 0x02b7a9 to 0x02a055 (0x1753 bytes) 000546 1785394445.080930 read 0x01032c thru 0x014b54 (0x4829 bytes) 000547 1785394445.112796 mapread 0x024268 thru 0x02a055 (0x5dee bytes) 000548 1785394445.158747 write 0x00a685 thru 0x013658 (0x8fd4 bytes) 000550 1785394445.170665 mapwrite 0x0069f0 thru 0x0124fd (0xbb0e bytes) 000552 1785394445.324707 read 0x01a20f thru 0x01d74a (0x353c bytes) 000553 1785394445.333625 mapwrite 0x0258db thru 0x034438 (0xeb5e bytes) 000556 1785394445.525169 read 0x0215db thru 0x02a976 (0x939c bytes) 000558 1785394445.528322 read 0x0239d2 thru 0x02c82b (0x8e5a bytes) 000560 1785394445.529485 mapwrite 0x01cbc7 thru 0x024097 (0x74d1 bytes) 000562 1785394445.716855 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000567 1785394446.048755 write 0x018113 thru 0x02459a (0xc488 bytes) 000569 1785394446.066784 read 0x01b5a4 thru 0x029a3c (0xe499 bytes) 000573 1785394446.094320 write 0x0222b4 thru 0x0275e6 (0x5333 bytes) 000574 1785394446.104678 write 0x024797 thru 0x02e41e (0x9c88 bytes) 000575 1785394446.121053 trunc from 0x034439 to 0x0082d0 (0x2c168 bytes) 000579 1785394446.152951 write 0x0182e0 thru 0x025e16 (0xdb37 bytes) 000580 1785394446.156783 write 0x02ab23 thru 0x0398a1 (0xed7f bytes) 000581 1785394446.166362 mapwrite 0x005ecb thru 0x011f3c (0xc072 bytes) 000582 1785394446.466395 trunc from 0x0398a2 to 0x00b33b (0x2e566 bytes) 000583 1785394446.491470 write 0x0269aa thru 0x035366 (0xe9bd bytes) 000584 1785394446.495387 mapwrite 0x03afba thru 0x03ffff (0x5046 bytes) 000586 1785394446.745475 mapread 0x012ec2 thru 0x01d409 (0xa548 bytes) 000587 1785394446.845517 MIRROR_EXTEND to 2 mirrors 000588 1785394448.403575 write 0x021ac2 thru 0x02cd13 (0xb252 bytes) 000590 1785394448.588129 MIRROR_RESYNC 2 mirrors 000592 1785394450.290207 write 0x001236 thru 0x001635 (0x400 bytes) 000593 1785394450.489224 mapread 0x022c59 thru 0x027ea1 (0x5249 bytes) 000595 1785394450.546046 read 0x03da41 thru 0x03ffff (0x25bf bytes) 000596 1785394450.570324 MIRROR_RESYNC 2 mirrors 000597 1785394452.095729 trunc from 0x040000 to 0x0113af (0x2ec50 bytes) 000598 1785394452.282637 read 0x002e1d thru 0x005b33 (0x2d17 bytes) 000600 1785394452.305197 mapwrite 0x03f868 thru 0x03ffff (0x798 bytes) 000601 1785394452.560977 trunc from 0x040000 to 0x008b0a (0x374f5 bytes) 000603 1785394452.612099 mapread 0x004e04 thru 0x008b0a (0x3d07 bytes) 000604 1785394452.645332 trunc from 0x008b0b to 0x0192ed (0x107e3 bytes) 000605 1785394452.712127 trunc from 0x0192ee to 0x02f5a3 (0x162b6 bytes) 000607 1785394452.767560 mapwrite 0x02bf3e thru 0x0328fa (0x69bd bytes) 000608 1785394452.939200 trunc from 0x0328fb to 0x031358 (0x15a2 bytes) 000609 1785394452.960213 mapread 0x02e735 thru 0x031358 (0x2c24 bytes) 000610 1785394452.961935 read 0x025164 thru 0x026a4f (0x18ec bytes) 000611 1785394452.970778 trunc from 0x031359 to 0x0307d0 (0xb88 bytes) 000612 1785394453.021062 trunc from 0x0307d1 to 0x0052cf (0x2b501 bytes) 000613 1785394453.079321 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x52d0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x52d0): Mirror 2: 0x67f813b9 (stale) CRC-32 checksum value for chunk [0, 0x52d0): Mirror 1: 0x67f813b9 Mirror 3: 0x67f813b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x52d0: skipped 000614 1785394454.420735 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x52d0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x52d0): Mirror 2: 0x67f813b9 (stale) CRC-32 checksum value for chunk [0, 0x52d0): Mirror 1: 0x67f813b9 Mirror 3: 0x67f813b9 Mirror 4: 0x67f813b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x52d0: skipped 000616 1785394455.685277 trunc from 0x0052d0 to 0x032aca (0x2d7fb bytes) 000619 1785394455.922009 MIRROR_RESYNC 4 mirrors 000620 1785394457.772851 mapwrite 0x0373fa thru 0x03ffff (0x8c06 bytes) 000621 1785394458.073920 mapwrite 0x02a65e thru 0x02ce2e (0x27d1 bytes) 000623 1785394458.213260 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x444db0a5 (stale) Mirror 3: 0x444db0a5 (stale) Mirror 4: 0x444db0a5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdec371ea Mirror 5: 0xdec371ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000624 1785394460.554117 write 0x0000ae thru 0x001894 (0x17e7 bytes) 000625 1785394460.740607 mapread 0x03e5e9 thru 0x03ffff (0x1a17 bytes) 000626 1785394460.783155 trunc from 0x040000 to 0x01b6a5 (0x2495a bytes) 000627 1785394460.825418 read 0x012f4e thru 0x01a231 (0x72e4 bytes) 000629 1785394460.851616 read 0x000758 thru 0x010486 (0xfd2f bytes) 000630 1785394460.870691 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b6a6): Mirror 2: 0xa4e1b482 (stale) Mirror 3: 0xa4e1b482 (stale) Mirror 4: 0xa4e1b482 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b6a6): Mirror 2: 0xa4e1b482 (stale) Mirror 3: 0xa4e1b482 (stale) Mirror 4: 0xa4e1b482 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b6a6: skipped 000631 1785394461.490748 trunc from 0x01b6a6 to 0x029b12 (0xe46d bytes) 000632 1785394461.553678 mapread 0x0109b6 thru 0x01d0d4 (0xc71f bytes) 000633 1785394461.651431 mapread 0x002c6d thru 0x003151 (0x4e5 bytes) 000635 1785394461.671028 trunc from 0x029b13 to 0x01d667 (0xc4ab bytes) 000636 1785394461.720383 read 0x014578 thru 0x01d667 (0x90f0 bytes) 000637 1785394461.721823 trunc from 0x01d668 to 0x01d5ad (0xba bytes) 000638 1785394461.761809 mapwrite 0x037794 thru 0x03fe6d (0x86da bytes) 000640 1785394462.071249 read 0x00b014 thru 0x00de49 (0x2e36 bytes) 000641 1785394462.105224 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe6e): Mirror 2: 0x488e73cf (stale) Mirror 4: 0x488e73cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe6e): Mirror 2: 0x488e73cf (stale) Mirror 4: 0x488e73cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe6e: skipped 000642 1785394462.643934 mapwrite 0x01fffc thru 0x02c9f4 (0xc9f9 bytes) 000643 1785394462.870170 trunc from 0x03fe6e to 0x004938 (0x3b535 bytes) 000645 1785394462.906109 mapwrite 0x02f3d5 thru 0x031f18 (0x2b44 bytes) 000646 1785394463.025421 trunc from 0x031f19 to 0x0304ae (0x1a6a bytes) 000649 1785394463.106155 write 0x0208d6 thru 0x02a9e6 (0xa111 bytes) 000650 1785394463.123755 trunc from 0x0304af to 0x008681 (0x27e2d bytes) 000651 1785394463.185102 read 0x001987 thru 0x005577 (0x3bf1 bytes) 000657 1785394463.208747 mapwrite 0x018097 thru 0x0273cd (0xf337 bytes) 000658 1785394463.469975 write 0x0004d0 thru 0x00e3bd (0xdeee bytes) 000659 1785394463.502355 mapread 0x005596 thru 0x0113e8 (0xbe53 bytes) 000663 1785394463.522495 mapwrite 0x02562f thru 0x02f4a7 (0x9e79 bytes) 000665 1785394463.682754 mapread 0x01094a thru 0x014a55 (0x410c bytes) 000666 1785394463.683744 mapwrite 0x0304c5 thru 0x035335 (0x4e71 bytes) 000669 1785394463.965064 trunc from 0x035336 to 0x022c5b (0x126da bytes) 000670 1785394464.024091 MIRROR_RESYNC 3 mirrors 000671 1785394465.751500 mapwrite 0x01c2b3 thru 0x026a3b (0xa789 bytes) 000672 1785394466.030744 trunc from 0x026a3c to 0x00666a (0x203d1 bytes) 000674 1785394466.076086 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x666b): Mirror 4: 0x610d65c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x666b): Mirror 4: 0x610d65c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x666b: skipped 000676 1785394466.519109 read 0x0042b9 thru 0x00666a (0x23b2 bytes) 000677 1785394466.530029 trunc from 0x00666b to 0x037f52 (0x318e8 bytes) 000678 1785394466.557616 mapwrite 0x02c744 thru 0x032717 (0x5fd4 bytes) 000679 1785394466.850836 mapwrite 0x02c30e thru 0x039a78 (0xd76b bytes) 000682 1785394467.116807 mapwrite 0x02524a thru 0x03380b (0xe5c2 bytes) 000684 1785394467.262167 mapwrite 0x02b72b thru 0x039d10 (0xe5e6 bytes) 000686 1785394467.392660 MIRROR_RESYNC 2 mirrors 000687 1785394469.380440 trunc from 0x039d11 to 0x00cc75 (0x2d09b bytes) 000691 1785394469.551318 trunc from 0x00cc76 to 0x02f85f (0x22bea bytes) 000692 1785394469.619580 read 0x025167 thru 0x02b01b (0x5eb5 bytes) 000694 1785394469.633207 mapread 0x005ee7 thru 0x011c9c (0xbdb6 bytes) 000698 1785394469.828797 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f860): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f860): Mirror 4: 0x4113a55d (stale) CRC-32 checksum value for chunk [0, 0x2f860): Mirror 1: 0x7d97da43 Mirror 5: 0x7d97da43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f860: skipped 000699 1785394471.864599 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f860): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f860): Mirror 4: 0x4113a55d (stale) CRC-32 checksum value for chunk [0, 0x2f860): Mirror 1: 0x7d97da43 Mirror 5: 0x7d97da43 Mirror 6: 0x7d97da43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f860: skipped 000702 1785394473.872505 trunc from 0x02f860 to 0x016fd2 (0x1888d bytes) 000703 1785394474.089531 mapread 0x012d57 thru 0x014bb1 (0x1e5b bytes) 000704 1785394474.156996 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16fd3): Mirror 4: 0x8b61f91a (stale) Mirror 5: 0x8b61f91a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16fd3): Mirror 4: 0x8b61f91a (stale) Mirror 5: 0x8b61f91a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16fd3: skipped 000706 1785394474.789908 MIRROR_RESYNC 3 mirrors 000707 1785394475.667536 mapwrite 0x02c2fb thru 0x0344f9 (0x81ff bytes) 000708 1785394475.972274 write 0x01f1ea thru 0x0213d5 (0x21ec bytes) 000709 1785394476.003431 mapread 0x00b240 thru 0x01a4af (0xf270 bytes) 000713 1785394476.083904 mapwrite 0x027141 thru 0x02e49c (0x735c bytes) 000714 1785394476.211101 mapwrite 0x02f274 thru 0x031cac (0x2a39 bytes) 000716 1785394476.296660 write 0x03f9b2 thru 0x03ffff (0x64e bytes) 000717 1785394476.298492 mapread 0x030acb thru 0x0343d8 (0x390e bytes) 000718 1785394476.303157 trunc from 0x040000 to 0x002d1d (0x3d2e2 bytes) 000719 1785394476.349202 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d1e): Mirror 4: 0x7171f5fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1e): Mirror 4: 0x7171f5fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1e: skipped 000720 1785394476.796495 trunc from 0x002d1e to 0x03fe0d (0x3d0f0 bytes) 000721 1785394476.842713 read 0x01c215 thru 0x0232d1 (0x70bd bytes) 000723 1785394476.853962 write 0x00876d thru 0x00b0e3 (0x2977 bytes) 000725 1785394476.866282 trunc from 0x03fe0e to 0x018b95 (0x27278 bytes) 000726 1785394476.888789 mapread 0x000895 thru 0x00a06e (0x97da bytes) 000727 1785394476.909708 trunc from 0x018b96 to 0x0290d4 (0x1053f bytes) 000730 1785394476.928645 read 0x021941 thru 0x0290d4 (0x7794 bytes) 000733 1785394476.939057 mapread 0x00989c thru 0x009f62 (0x6c7 bytes) 000735 1785394476.941061 write 0x008a41 thru 0x00e682 (0x5c42 bytes) 000738 1785394476.944678 write 0x01d76a thru 0x02d2ee (0xfb85 bytes) 000739 1785394476.946539 mapread 0x00efc1 thru 0x010990 (0x19d0 bytes) 000740 1785394476.947758 trunc from 0x02d2ef to 0x0285ba (0x4d34 bytes) 000743 1785394476.967015 MIRROR_RESYNC 2 mirrors 000744 1785394477.912853 mapread 0x00704f thru 0x012379 (0xb32b bytes) 000747 1785394477.977968 write 0x001ed1 thru 0x007660 (0x5790 bytes) 000751 1785394478.155219 mapread 0x011dc6 thru 0x01b1c3 (0x93fe bytes) 000754 1785394478.229882 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x285bb): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285bb): Mirror 4: 0xb476b390 (stale) CRC-32 checksum value for chunk [0, 0x285bb): Mirror 1: 0x549d3083 Mirror 5: 0x549d3083 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285bb: skipped 000756 1785394480.006541 write 0x01f914 thru 0x02586c (0x5f59 bytes) 000757 1785394480.216074 read 0x011619 thru 0x01e8a4 (0xd28c bytes) 000758 1785394480.230253 read 0x01b5a0 thru 0x01eca3 (0x3704 bytes) 000760 1785394480.231750 read 0x00a2b8 thru 0x016380 (0xc0c9 bytes) 000761 1785394480.289788 trunc from 0x0285bb to 0x00b3a7 (0x1d213 bytes) 000762 1785394480.356241 mapwrite 0x021b99 thru 0x022927 (0xd8f bytes) 000763 1785394480.572015 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22928): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22928): Mirror 4: 0x698a0420 (stale) Mirror 5: 0xa59929cf (stale) CRC-32 checksum value for chunk [0, 0x22928): Mirror 1: 0x8445d10e Mirror 6: 0x8445d10e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22928: skipped 000764 1785394482.158079 mapwrite 0x017f2f thru 0x01f6c2 (0x7794 bytes) 000768 1785394482.610817 mapread 0x01045d thru 0x010c58 (0x7fc bytes) 000772 1785394482.664117 write 0x015cf3 thru 0x01e22b (0x8539 bytes) 000773 1785394482.675926 trunc from 0x022928 to 0x032511 (0xfbea bytes) 000775 1785394482.723828 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32512): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32512): Mirror 4: 0xeb7644b3 (stale) Mirror 5: 0x54347179 (stale) Mirror 6: 0x7b472902 (stale) CRC-32 checksum value for chunk [0, 0x32512): Mirror 1: 0xd004966f Mirror 7: 0xd004966f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32512: skipped 000776 1785394485.523637 write 0x00792f thru 0x009fff (0x26d1 bytes) 000777 1785394485.773851 mapwrite 0x033854 thru 0x039304 (0x5ab1 bytes) 000779 1785394486.000644 write 0x014010 thru 0x01787f (0x3870 bytes) 000780 1785394486.043882 trunc from 0x039305 to 0x02eedf (0xa425 bytes) 000783 1785394486.099109 trunc from 0x02eee0 to 0x029db4 (0x512b bytes) 000785 1785394486.157677 trunc from 0x029db5 to 0x00a9af (0x1f405 bytes) 000787 1785394486.178260 read 0x00a25a thru 0x00a9af (0x756 bytes) 000789 1785394486.189452 trunc from 0x00a9b0 to 0x00d89b (0x2eec bytes) 000790 1785394486.249204 write 0x012a65 thru 0x01ed7e (0xc31a bytes) 000792 1785394486.268035 MIRROR_RESYNC 5 mirrors 000793 1785394489.207982 write 0x037634 thru 0x03ffff (0x89cc bytes) 000795 1785394489.492143 write 0x00c3e0 thru 0x00f906 (0x3527 bytes) 000800 1785394489.568656 write 0x029fc8 thru 0x02d7fd (0x3836 bytes) 000801 1785394489.599685 mapwrite 0x02cb05 thru 0x03ba81 (0xef7d bytes) 000802 1785394489.953671 read 0x02030b thru 0x02eef8 (0xebee bytes) 000804 1785394489.965878 mapread 0x01fd57 thru 0x02c375 (0xc61f bytes) 000807 1785394489.989614 write 0x030dd1 thru 0x03cdfc (0xc02c bytes) 000808 1785394490.000028 trunc from 0x040000 to 0x00732f (0x38cd0 bytes) 000809 1785394490.071994 MIRROR_RESYNC 5 mirrors 000811 1785394492.730375 mapwrite 0x023d4e thru 0x025c57 (0x1f0a bytes) 000812 1785394493.039153 mapwrite 0x02aaaa thru 0x0301c9 (0x5720 bytes) 000813 1785394493.205507 mapwrite 0x01dbb7 thru 0x022a9c (0x4ee6 bytes) 000815 1785394493.355037 read 0x00234b thru 0x00a993 (0x8649 bytes) 000816 1785394493.366626 mapread 0x01ed1e thru 0x02c22e (0xd511 bytes) 000818 1785394493.371600 mapwrite 0x011d5d thru 0x014228 (0x24cc bytes) 000819 1785394493.551033 read 0x0102ea thru 0x01185b (0x1572 bytes) 000820 1785394493.552089 read 0x0019f6 thru 0x00c78b (0xad96 bytes) 000821 1785394493.607255 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: 0x8a2af76 (stale) Mirror 5: 0x8a2af76 (stale) Mirror 6: 0x8a2af76 (stale) Mirror 7: 0x8a2af76 (stale) CRC-32 checksum value for chunk [0, 0x301ca): Mirror 1: 0x535bda02 Mirror 8: 0x535bda02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ca: skipped 000822 1785394496.254538 read 0x01cc01 thru 0x021449 (0x4849 bytes) 000823 1785394496.314421 mapread 0x0288b5 thru 0x0301c9 (0x7915 bytes) 000824 1785394496.433600 read 0x01345f thru 0x020835 (0xd3d7 bytes) 000825 1785394496.463960 read 0x01f3e0 thru 0x022fed (0x3c0e bytes) 000828 1785394496.512020 mapread 0x00291b thru 0x00854d (0x5c33 bytes) 000830 1785394496.583004 write 0x0083b7 thru 0x0168da (0xe524 bytes) 000831 1785394496.908666 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: 0x8a2af76 (stale) Mirror 6: 0x8a2af76 (stale) Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: 0x8a2af76 (stale) Mirror 6: 0x8a2af76 (stale) Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ca: skipped 000832 1785394497.829052 read 0x003dcc thru 0x00ad60 (0x6f95 bytes) 000833 1785394497.857532 mapread 0x02c945 thru 0x0301c9 (0x3885 bytes) 000834 1785394497.896715 mapwrite 0x00189b thru 0x002c68 (0x13ce bytes) 000836 1785394498.252844 mapread 0x009fd4 thru 0x016b20 (0xcb4d bytes) 000838 1785394498.293553 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: 0x8a2af76 (stale) Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ca): Mirror 4: 0x8a2af76 (stale) Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ca: skipped 000839 1785394499.233907 mapwrite 0x0239bf thru 0x0298a3 (0x5ee5 bytes) 000840 1785394499.419704 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301ca): Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ca): Mirror 7: 0x8a2af76 (stale) Mirror 8: 0x535bda02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ca: skipped 000841 1785394500.189414 mapwrite 0x00c580 thru 0x0194d8 (0xcf59 bytes) 000842 1785394500.494494 write 0x02aa54 thru 0x02beae (0x145b bytes) 000843 1785394500.544425 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301ca): Mirror 7: 0x8a2af76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ca): Mirror 7: 0x8a2af76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ca: skipped 000844 1785394501.279977 trunc from 0x0301ca to 0x03bcc9 (0xbb00 bytes) 000845 1785394501.324615 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000846 1785394501.734979 trunc from 0x03bcca to 0x036723 (0x55a6 bytes) 000847 1785394501.810836 MIRROR_EXTEND to 2 mirrors 000848 1785394504.104970 MIRROR_EXTEND to 3 mirrors 000849 1785394506.465823 write 0x029426 thru 0x02e6fd (0x52d8 bytes) 000850 1785394506.711741 read 0x01f4a7 thru 0x02468a (0x51e4 bytes) 000851 1785394506.759347 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36724): Mirror 2: 0x56e03990 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36724): Mirror 2: 0x56e03990 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36724: skipped 000852 1785394507.709085 trunc from 0x036724 to 0x032046 (0x46dd bytes) 000853 1785394507.769606 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000855 1785394508.267685 read 0x020121 thru 0x029944 (0x9824 bytes) 000856 1785394508.295848 mapread 0x008ccd thru 0x015cf2 (0xd026 bytes) 000860 1785394508.410086 mapwrite 0x02bd99 thru 0x0346a4 (0x890c bytes) 000862 1785394508.632565 read 0x0294b0 thru 0x0346a4 (0xb1f5 bytes) 000867 1785394508.642023 read 0x00cd4e thru 0x00f651 (0x2904 bytes) 000868 1785394508.647540 read 0x006e75 thru 0x009b56 (0x2ce2 bytes) 000871 1785394508.655178 mapread 0x007255 thru 0x008bfa (0x19a6 bytes) 000873 1785394508.659911 trunc from 0x0346a5 to 0x002db7 (0x318ed bytes) 000874 1785394508.688395 read 0x000eee thru 0x002db7 (0x1eca bytes) 000876 1785394508.712895 mapwrite 0x03c60a thru 0x03ffff (0x39f6 bytes) 000878 1785394508.858183 read 0x038886 thru 0x03dcaf (0x542a bytes) 000879 1785394508.867484 write 0x03c454 thru 0x03ffff (0x3bac bytes) 000880 1785394508.892016 MIRROR_EXTEND to 2 mirrors 000882 1785394511.384310 read 0x02e0a5 thru 0x036826 (0x8782 bytes) 000884 1785394511.434967 mapread 0x00fee8 thru 0x0117ed (0x1906 bytes) 000886 1785394511.530912 mapwrite 0x03e2be thru 0x03ffff (0x1d42 bytes) 000887 1785394512.012939 read 0x01a718 thru 0x02206e (0x7957 bytes) 000889 1785394512.041832 write 0x023d87 thru 0x0329cb (0xec45 bytes) 000891 1785394512.050427 write 0x01c48b thru 0x027579 (0xb0ef bytes) 000892 1785394512.059342 read 0x02df19 thru 0x031c87 (0x3d6f bytes) 000893 1785394512.061165 mapwrite 0x03cc88 thru 0x03ffff (0x3378 bytes) 000895 1785394512.184500 mapwrite 0x0314d2 thru 0x036043 (0x4b72 bytes) 000896 1785394512.321391 write 0x038a52 thru 0x03ffff (0x75ae bytes) 000897 1785394512.324519 mapwrite 0x02ae3c thru 0x03a0b7 (0xf27c bytes) 000898 1785394512.512218 read 0x0359b8 thru 0x039139 (0x3782 bytes) 000902 1785394512.526664 write 0x02783e thru 0x02f052 (0x7815 bytes) 000904 1785394512.541798 write 0x00b755 thru 0x018960 (0xd20c bytes) 000905 1785394512.590487 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000907 1785394513.378539 write 0x005827 thru 0x010f08 (0xb6e2 bytes) 000911 1785394513.475572 trunc from 0x040000 to 0x00e91a (0x316e5 bytes) 000913 1785394513.536582 write 0x03e853 thru 0x03ffff (0x17ad bytes) 000915 1785394513.545153 write 0x0014a3 thru 0x0057c0 (0x431e bytes) 000917 1785394513.560566 write 0x01ae57 thru 0x024493 (0x963d bytes) 000919 1785394513.607046 mapread 0x03052f thru 0x036c19 (0x66eb bytes) 000922 1785394513.673743 mapwrite 0x00326d thru 0x013242 (0xffd6 bytes) 000923 1785394513.921731 read 0x03d02b thru 0x03ffff (0x2fd5 bytes) 000924 1785394513.936735 write 0x03f686 thru 0x03ffff (0x97a bytes) 000925 1785394513.952018 trunc from 0x040000 to 0x01795c (0x286a3 bytes) 000926 1785394514.018017 MIRROR_EXTEND to 2 mirrors 000927 1785394515.264230 mapwrite 0x01f6ee thru 0x02e1df (0xeaf2 bytes) 000928 1785394515.473139 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e1e0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1e0): Mirror 2: 0x20d5d49d (stale) CRC-32 checksum value for chunk [0, 0x2e1e0): Mirror 1: 0x9f61cbf2 Mirror 3: 0x9f61cbf2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1e0: skipped 000929 1785394516.554717 MIRROR_RESYNC 3 mirrors 000931 1785394518.440796 write 0x03fbaa thru 0x03ffff (0x456 bytes) 000932 1785394518.505409 mapwrite 0x034e09 thru 0x03ffff (0xb1f7 bytes) 000933 1785394518.557454 mapwrite 0x0325bb thru 0x033a46 (0x148c bytes) 000934 1785394518.582437 mapwrite 0x032231 thru 0x03ffff (0xddcf bytes) 000935 1785394518.626039 mapread 0x03aa23 thru 0x03c662 (0x1c40 bytes) 000936 1785394518.627072 write 0x0261c3 thru 0x02735f (0x119d bytes) 000937 1785394518.638433 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x805c2a11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x805c2a11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000938 1785394518.863783 write 0x0104bc thru 0x013535 (0x307a bytes) 000939 1785394518.872655 mapread 0x00b1a6 thru 0x00fb45 (0x49a0 bytes) 000941 1785394518.887511 trunc from 0x040000 to 0x024a1c (0x1b5e3 bytes) 000944 1785394518.904246 write 0x03a504 thru 0x03ffff (0x5afc bytes) 000946 1785394518.906678 mapread 0x0309b7 thru 0x03d4b7 (0xcb01 bytes) 000947 1785394518.918562 trunc from 0x040000 to 0x01e4af (0x21b50 bytes) 000948 1785394518.936801 MIRROR_RESYNC 2 mirrors 000950 1785394519.500267 mapread 0x011f1f thru 0x01e4af (0xc591 bytes) 000955 1785394519.532784 mapwrite 0x03b26a thru 0x03bb98 (0x92f bytes) 000956 1785394519.760133 mapwrite 0x028640 thru 0x032204 (0x9bc5 bytes) 000957 1785394519.962692 mapwrite 0x01745e thru 0x02381c (0xc3bf bytes) 000958 1785394520.064294 read 0x02be0a thru 0x0326cf (0x68c6 bytes) 000961 1785394520.069882 write 0x03d065 thru 0x03f511 (0x24ad bytes) 000963 1785394520.077612 read 0x01611a thru 0x01b425 (0x530c bytes) 000964 1785394520.078833 mapwrite 0x02aa8e thru 0x037adf (0xd052 bytes) 000967 1785394520.201825 read 0x0081f2 thru 0x008881 (0x690 bytes) 000968 1785394520.215460 read 0x02657a thru 0x026ea2 (0x929 bytes) 000969 1785394520.216169 read 0x00934e thru 0x00f3c6 (0x6079 bytes) 000972 1785394520.219522 mapwrite 0x0224c9 thru 0x02c236 (0x9d6e bytes) 000976 1785394520.260212 mapwrite 0x003756 thru 0x0064ba (0x2d65 bytes) 000979 1785394520.290711 mapwrite 0x01b733 thru 0x029c96 (0xe564 bytes) 000980 1785394520.331292 mapread 0x00fb8a thru 0x011859 (0x1cd0 bytes) 000983 1785394520.349212 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x12d:0x0]: Invalid argument (22) 000986 1785394520.654384 read 0x01e0de thru 0x0243b1 (0x62d4 bytes) 000990 1785394520.693024 read 0x01c50b thru 0x024eb6 (0x89ac bytes) 000993 1785394520.739387 write 0x03ac51 thru 0x03ffff (0x53af bytes) 000994 1785394520.745607 mapread 0x002634 thru 0x004a49 (0x2416 bytes) 000995 1785394520.756767 read 0x002c6e thru 0x00b747 (0x8ada bytes) 000996 1785394520.760417 read 0x033433 thru 0x03d8d8 (0xa4a6 bytes) 000997 1785394520.764842 mapread 0x022b0a thru 0x0241bb (0x16b2 bytes) 001000 1785394520.776972 read 0x02f847 thru 0x034727 (0x4ee1 bytes) All operations completed A-OK! PASS 70a (171s) == sanity-flr test 71: check mirror extend parent fid ==== 02:55:31 (1785394531) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (6s) == sanity-flr test 72: check append on FLR file ========== 02:55:37 (1785394537) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.498028 s, 4.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277048 s, 3.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x257:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x257:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (18s) == sanity-flr test 200a: stress test ===================== 02:55:55 (1785394555) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7561056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6071904 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8526016 .. Extending file size to 7177248 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7204320 .. Extending file size to 8420352 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9358272 .. Extending file size to 8804672 .. Waiting 57167 57169 57170 57173 57174 done 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) 192.168.203.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (95s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:57:31 (1785394651) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3153600 .. Extending file size to 6860768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8739008 .. Extending file size to 4507552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7520192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7612640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8151808 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8899072 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9184640 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8493024 .. Waiting 58310 58311 58312 58313 58316 done 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) 192.168.203.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (90s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:59:01 (1785394741) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271501 s, 7.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.428554 s, 4.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xda3c84f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:59:28 (1785394768) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121717 s, 17.2 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 02:59:39 (1785394779) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x251:0x0] } PASS 203 (12s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:59:51 (1785394791) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.458108 s, 4.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x252:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x253:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (14s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:00:05 (1785394805) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.437748 s, 4.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x255:0x0] } PASS 204b (13s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:00:18 (1785394818) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.50288 s, 4.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:00:54 (1785394854) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225171 s, 4.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } PASS 204d (70s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:02:04 (1785394924) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.644848 s, 3.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (71s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:03:15 (1785394995) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. Creating new pool test_204f oleg320-server: Pool lustre.test_204f created Adding targets to pool oleg320-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg320-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.305653 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg320-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg320-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg320-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:04:43 (1785395083) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:04:53 (1785395093) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:05:05 (1785395105) oleg320-server: Pool lustre.test_206 created Adding targets to pool oleg320-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg320-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg320-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg320-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg320-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:05:35 (1785395135) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744446 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605035 s, 17.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdc1a2d40 Mirror 3: 0xdc1a2d40 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:05:49 (1785395149) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:05:54 (1785395154) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:05:58 (1785395158) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.17538 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1db0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1db0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1db0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1800000 Read this (4 bytes): BBBB 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:06:43 (1785395203) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0316344 s, 259 kB/s the very first read read with warm cache PASS 209b (11s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:06:55 (1785395215) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906672 s, 11.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x19f lmm_fid: [0x200000404:0x19f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x19f lmm_fid: [0x200000404:0x19f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834335 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:06:57.000000000 -0400 Modify: 2026-07-30 03:06:57.000000000 -0400 Change: 2026-07-30 03:06:59.000000000 -0400 Birth: 2026-07-30 03:06:58.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:07:06 (1785395226) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:07:10 (1785395230) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.801591 s, 51.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.094641 s, 43.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 30 03:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 30 03:07 /mnt/lustre/f211.sanity-flr PASS 211 (13s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:07:24 (1785395244) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (12s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:07:35 (1785395255) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 2342 sec =========== 03:07:48 (1785395268) === sanity-flr: start cleanup 03:07:50 (1785395270) === === sanity-flr: finish cleanup 03:08:00 (1785395280) ===