-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 18:42:07 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg426-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:42:28 (1786747348) === oleg426-client.virtnet: executing check_config_client /mnt/lustre oleg426-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg426-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a4dd25c7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a4dd25c7000.idle_timeout=debug disable quota as required oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:43:02 (1786747382) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7671] == sanity-flr test 0a: lfs mirror create with -N option == 18:43:05 (1786747385) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:43:22 (1786747402) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:43:25 (1786747405) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:43:29 (1786747409) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:43:54 (1786747434) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:43:58 (1786747438) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:44:01 (1786747441) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a4dd25c7000.stats=clear osc.lustre-OST0001-osc-ffff8a4dd25c7000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:44:09 (1786747449) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:44:20 (1786747460) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:44:30 (1786747470) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:44:34 (1786747474) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:44:43 (1786747483) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:44:52 (1786747492) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:45:00 (1786747500) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:45:04 (1786747504) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:45:12 (1786747512) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:45:23 (1786747523) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:45:29 (1786747529) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.93763 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142366 s, 7.4 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:45:42 (1786747542) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.221732 s, 42.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:45:50 (1786747550) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:46:01 (1786747561) Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:47:16 (1786747636) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.846015 s, 13.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:47:59 (1786747679) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:49:14 (1786747754) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.53438 s, 9.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 18:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.885s user 0m0.001s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.150s user 0m0.001s sys 0m0.178s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.454s user 0m0.001s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a4dd25c7000.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:50:42 (1786747842) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:50:45 (1786747845) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:50:48 (1786747848) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 18:50:52 (1786747852) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 18:51:01 (1786747861) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.675206 s, 18.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:51:15 (1786747875) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230296 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025542 s, 41.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:51:22 (1786747882) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:51:29 (1786747889) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127842 s, 801 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00153463 s, 667 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00083174 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010004 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:51:39 (1786747899) create test files with size 15729608 .. 42550667d01ef524335b2e5d28e62b4c - a91f04cabb55115848bfb772633c92a6 - 7c7c996ee94ec8fe135f03e74a798658 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 18:52:01 (1786747921) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:52:15 (1786747935) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:52:22 (1786747942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127472 s, 8.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456552 s, 23.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 18:52:31 (1786747951) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113531 s, 18.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.303961 s, 13.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241857 s, 8.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 18:52:48 (1786747968) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:52:51 (1786747971) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:52:54 (1786747974) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041763 s, 25.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 18:53:02 (1786747982) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:53:06 (1786747986) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:53:15 (1786747995) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.276146 s, 38.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:53:26 (1786748006) llite.lustre-ffff8a4dd08d1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.26528 s, 39.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8a4dc3192000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:53:41 (1786748021) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:54:30 (1786748070) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:55:16 (1786748116) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:55:19 (1786748119) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.220252 s, 14.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14271 s, 22.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3197c386 (nosync,stale) Mirror 2: 0x3197c386 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3d7289ca5c489b554e03b15d8c8e3e10 - mirror 1 checksum: 0c4cae8db28d8f572518b00c89a2c2c4 - mirror 2 checksum: 3d7289ca5c489b554e03b15d8c8e3e10 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T18:55:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T18:55:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3d7289ca5c489b554e03b15d8c8e3e10 - mirror 1 checksum: 0c4cae8db28d8f572518b00c89a2c2c4 - mirror 2 checksum: 0c4cae8db28d8f572518b00c89a2c2c4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T18:55:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T18:55:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:55:32 (1786748132) Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 18:55:38 (1786748138) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server Failover mds1 to oleg426-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 18:56:01 (1786748161) targets are mounted 18:56:01 (1786748161) facet_failover done oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455013 s, 23.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:56:19 (1786748179) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:56:22 (1786748182) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:56:26 (1786748186) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00571543 s, 2.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:56:33 (1786748193) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:56:37 (1786748197) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00828256 s, 495 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0628542 s, 65.2 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:56:44 (1786748204) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:57:08 (1786748228) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:57:37 (1786748257) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:58:00 (1786748280) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:58:25 (1786748305) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 18:58:34 (1786748314) mirror create pid 54358 mirror split pid 54360 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54358 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54360 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 18:59:41 (1786748381) Seed set to 8383 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1786748384.113066 write 0x01a303 thru 0x021984 (0x7682 bytes) 000004 1786748384.115437 write 0x03af2c thru 0x03ffff (0x50d4 bytes) truncating to largest ever: 0x2d881 000005 1786748384.126038 trunc from 0x040000 to 0x02d880 (0x1277f bytes) 000007 1786748384.142410 write 0x01c742 thru 0x0264b1 (0x9d70 bytes) 000010 1786748384.152677 trunc from 0x02d881 to 0x00fd8d (0x1daf3 bytes) truncating to largest ever: 0x34d09 000013 1786748384.188026 trunc from 0x00fd8e to 0x034d08 (0x24f7b bytes) 000014 1786748384.211791 mapread 0x0341e9 thru 0x034d08 (0xb20 bytes) 000015 1786748384.223183 read 0x0149af thru 0x016e29 (0x247b bytes) 000018 1786748384.237256 mapwrite 0x003dbe thru 0x00baec (0x7d2f bytes) 000019 1786748384.407985 read 0x0077c6 thru 0x01304d (0xb888 bytes) 000023 1786748384.445469 write 0x02f06a thru 0x037b11 (0x8aa8 bytes) 000024 1786748384.447479 write 0x0197e0 thru 0x023602 (0x9e23 bytes) 000025 1786748384.449561 trunc from 0x037b12 to 0x00707c (0x30a95 bytes) 000030 1786748384.474192 mapwrite 0x02ab05 thru 0x03a31f (0xf81b bytes) 000031 1786748384.697626 MIRROR_EXTEND to 2 mirrors 000032 1786748385.772086 MIRROR_RESYNC 2 mirrors 000035 1786748386.283297 trunc from 0x03a320 to 0x020f0d (0x19412 bytes) 000036 1786748386.439501 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20f0e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20f0e): Mirror 2: 0x5a2683ad (stale) CRC-32 checksum value for chunk [0, 0x20f0e): Mirror 1: 0x5a2683ad Mirror 3: 0x5a2683ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20f0e: skipped 000038 1786748387.695920 mapwrite 0x030470 thru 0x03a41b (0x9fac bytes) 000039 1786748387.901575 trunc from 0x03a41c to 0x02a14d (0x102ce bytes) 000040 1786748387.935284 read 0x010f6b thru 0x020579 (0xf60f bytes) 000041 1786748387.964058 mapread 0x017b32 thru 0x01a343 (0x2812 bytes) 000042 1786748387.965884 trunc from 0x02a14e to 0x026bb1 (0x359c bytes) 000046 1786748387.997360 read 0x00f7a1 thru 0x019acb (0xa32b bytes) 000047 1786748388.014539 trunc from 0x026bb2 to 0x020f34 (0x5c7d bytes) 000048 1786748388.099262 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20f35): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20f35): Mirror 2: 0xd45816a3 (stale) Mirror 3: 0xd45816a3 (stale) CRC-32 checksum value for chunk [0, 0x20f35): Mirror 1: 0xd45816a3 Mirror 4: 0xd45816a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20f35: skipped 000050 1786748389.127148 mapwrite 0x01f9e8 thru 0x02f2f6 (0xf90f bytes) 000052 1786748389.363758 mapwrite 0x032f79 thru 0x039617 (0x669f bytes) 000053 1786748389.480945 trunc from 0x039618 to 0x020e65 (0x187b2 bytes) 000055 1786748389.504433 mapread 0x01cdbe thru 0x020e65 (0x40a8 bytes) 000057 1786748389.507575 read 0x01081a thru 0x017fe9 (0x77d0 bytes) 000058 1786748389.524696 trunc from 0x020e66 to 0x022d1d (0x1eb8 bytes) 000059 1786748389.545541 trunc from 0x022d1e to 0x02cc78 (0x9f5b bytes) 000061 1786748389.576315 read 0x0094f8 thru 0x0104b9 (0x6fc2 bytes) 000062 1786748389.594936 mapwrite 0x02423a thru 0x033f4d (0xfd14 bytes) 000063 1786748389.687529 mapwrite 0x018473 thru 0x01bc96 (0x3824 bytes) 000064 1786748389.759805 mapread 0x0049fe thru 0x0075fc (0x2bff bytes) 000065 1786748389.763361 trunc from 0x033f4e to 0x000141 (0x33e0c bytes) 000066 1786748389.800227 trunc from 0x000142 to 0x02ffd1 (0x2fe90 bytes) 000067 1786748389.848945 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 2: 0xb9575848 (stale) Mirror 3: 0x4d61ef03 (stale) Mirror 4: 0x4d61ef03 (stale) CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 1: 0x8e4bd635 Mirror 5: 0x8e4bd635 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffd2: skipped 000070 1786748391.215504 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 2: 0xb9575848 (stale) Mirror 3: 0x4d61ef03 (stale) Mirror 4: 0x4d61ef03 (stale) CRC-32 checksum value for chunk [0, 0x2ffd2): Mirror 1: 0x8e4bd635 Mirror 5: 0x8e4bd635 Mirror 6: 0x8e4bd635 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffd2: skipped truncating to largest ever: 0x3565e 000071 1786748392.411508 trunc from 0x02ffd2 to 0x03565d (0x568c bytes) 000072 1786748392.490343 read 0x0225d5 thru 0x024e6a (0x2896 bytes) 000074 1786748392.497420 write 0x033e77 thru 0x03ace9 (0x6e73 bytes) 000075 1786748392.502724 read 0x029bc3 thru 0x032380 (0x87be bytes) 000077 1786748392.512434 trunc from 0x03acea to 0x034bf6 (0x60f3 bytes) 000079 1786748392.534100 mapwrite 0x030119 thru 0x034969 (0x4851 bytes) 000080 1786748392.582719 read 0x017ef1 thru 0x024c3d (0xcd4d bytes) 000081 1786748392.584035 write 0x002641 thru 0x010699 (0xe059 bytes) 000082 1786748392.591577 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34bf7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34bf7): Mirror 2: 0xc3812a58 (stale) Mirror 3: 0x657259a7 (stale) Mirror 4: 0x657259a7 (stale) Mirror 5: 0x2a2568cb (stale) Mirror 6: 0x2a2568cb (stale) CRC-32 checksum value for chunk [0, 0x34bf7): Mirror 1: 0x3d78e088 Mirror 7: 0x3d78e088 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34bf7: skipped 000083 1786748394.451144 write 0x03e1ce thru 0x03ffff (0x1e32 bytes) 000084 1786748394.524027 trunc from 0x040000 to 0x01a817 (0x257e8 bytes) 000087 1786748394.548363 write 0x002ce3 thru 0x006fbb (0x42d9 bytes) 000091 1786748394.564338 mapwrite 0x03ee0b thru 0x03ffff (0x11f5 bytes) 000092 1786748394.653099 trunc from 0x040000 to 0x00cd03 (0x332fc bytes) 000093 1786748394.682898 mapwrite 0x03ee50 thru 0x03ffff (0x11b0 bytes) 000095 1786748394.807267 write 0x03b5e3 thru 0x03ffff (0x4a1d bytes) 000097 1786748394.814259 mapread 0x007db3 thru 0x00fe0d (0x805b bytes) 000099 1786748394.833672 write 0x00a05c thru 0x017901 (0xd8a6 bytes) 000103 1786748394.852495 mapread 0x002a77 thru 0x011510 (0xea9a bytes) 000104 1786748394.854415 write 0x0226c2 thru 0x02c4a8 (0x9de7 bytes) 000105 1786748394.878656 mapwrite 0x02e6c9 thru 0x03ab66 (0xc49e bytes) 000106 1786748395.015964 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa4317ab0 (stale) Mirror 3: 0x7517006f (stale) Mirror 4: 0x7517006f (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x62eee93a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x33bd7eaa Mirror 8: 0x33bd7eaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000107 1786748396.505115 MIRROR_RESYNC 8 mirrors 000109 1786748399.200434 mapread 0x0247af thru 0x026064 (0x18b6 bytes) 000110 1786748399.284154 mapread 0x0208be thru 0x02ba14 (0xb157 bytes) 000112 1786748399.299368 MIRROR_RESYNC 8 mirrors 000113 1786748400.193401 MIRROR_RESYNC 8 mirrors 000114 1786748400.679854 mapread 0x0203b0 thru 0x02fac8 (0xf719 bytes) 000115 1786748400.689376 read 0x02fa9d thru 0x030fd4 (0x1538 bytes) 000116 1786748400.689779 mapwrite 0x02cf1e thru 0x0336f4 (0x67d7 bytes) 000117 1786748400.868072 mapread 0x0116a5 thru 0x020531 (0xee8d bytes) 000118 1786748400.901486 read 0x00a42a thru 0x00c625 (0x21fc bytes) 000119 1786748400.902210 read 0x012209 thru 0x01a49b (0x8293 bytes) 000120 1786748400.902857 mapwrite 0x03f74f thru 0x03ffff (0x8b1 bytes) 000123 1786748400.970829 MIRROR_RESYNC 8 mirrors 000124 1786748403.421828 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000125 1786748403.796130 mapread 0x003684 thru 0x006686 (0x3003 bytes) 000126 1786748403.820446 mapread 0x030cbf thru 0x03508f (0x43d1 bytes) 000127 1786748403.833570 trunc from 0x040000 to 0x005e73 (0x3a18c bytes) 000128 1786748403.911815 read 0x0057e6 thru 0x005e73 (0x68e bytes) 000129 1786748403.917342 read 0x0034e6 thru 0x005e73 (0x298e bytes) 000130 1786748403.927332 trunc from 0x005e74 to 0x0009bd (0x54b6 bytes) 000131 1786748403.950620 write 0x03070b thru 0x03dac4 (0xd3ba bytes) 000132 1786748403.953562 mapwrite 0x033e26 thru 0x03ffff (0xc1da bytes) 000134 1786748404.043418 read 0x0048ee thru 0x013b5e (0xf271 bytes) 000135 1786748404.052470 mapwrite 0x00d50a thru 0x013a3b (0x6532 bytes) 000136 1786748404.092767 trunc from 0x040000 to 0x00285b (0x3d7a4 bytes) 000137 1786748404.113047 mapwrite 0x025e2d thru 0x02c257 (0x642b bytes) 000138 1786748404.301629 read 0x02904f thru 0x02b44f (0x2401 bytes) 000140 1786748404.302844 read 0x006a35 thru 0x009f79 (0x3545 bytes) 000142 1786748404.314154 mapread 0x017cab thru 0x01b86a (0x3bc0 bytes) 000143 1786748404.315474 read 0x009c17 thru 0x01030b (0x66f5 bytes) 000144 1786748404.317275 trunc from 0x02c258 to 0x00899e (0x238b9 bytes) 000147 1786748404.337484 read 0x005b4f thru 0x006fb8 (0x146a bytes) 000148 1786748404.338575 mapread 0x001ee2 thru 0x003e78 (0x1f97 bytes) 000152 1786748404.342535 read 0x0080ee thru 0x00899e (0x8b1 bytes) 000153 1786748404.343574 mapwrite 0x03a6ec thru 0x03ffff (0x5914 bytes) 000154 1786748404.531937 MIRROR_RESYNC 7 mirrors 000155 1786748406.857058 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000156 1786748407.201216 mapread 0x00c48b thru 0x010376 (0x3eec bytes) 000157 1786748407.224481 mapwrite 0x017e57 thru 0x027a17 (0xfbc1 bytes) 000163 1786748407.488324 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b8cf60c (stale) Mirror 3: 0x9b8cf60c (stale) Mirror 4: 0x9b8cf60c (stale) Mirror 5: 0x9b8cf60c (stale) Mirror 7: 0x9b8cf60c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xccf62027 Mirror 8: 0xccf62027 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000164 1786748408.600960 mapwrite 0x00b240 thru 0x00ed79 (0x3b3a bytes) 000165 1786748408.849905 MIRROR_RESYNC 7 mirrors 000167 1786748410.596988 mapread 0x024a8e thru 0x02823d (0x37b0 bytes) 000169 1786748410.631185 trunc from 0x040000 to 0x02c889 (0x13776 bytes) 000172 1786748410.753904 trunc from 0x02c88a to 0x0298bb (0x2fce bytes) 000173 1786748410.776498 trunc from 0x0298bc to 0x010803 (0x190b8 bytes) 000174 1786748410.795390 mapread 0x005484 thru 0x00ae24 (0x59a1 bytes) 000175 1786748410.806225 trunc from 0x010804 to 0x022f7e (0x1277b bytes) 000177 1786748410.823948 mapwrite 0x023fe6 thru 0x030db2 (0xcdcd bytes) 000180 1786748410.911997 mapwrite 0x00dd2d thru 0x015ac8 (0x7d9c bytes) 000182 1786748410.977104 trunc from 0x030db3 to 0x01fda6 (0x1100c bytes) 000183 1786748410.994355 read 0x00f41a thru 0x01f006 (0xfbed bytes) 000185 1786748411.002372 trunc from 0x01fda7 to 0x029d8b (0x9fe5 bytes) 000186 1786748411.021327 trunc from 0x029d8c to 0x00f72a (0x1a661 bytes) 000187 1786748411.036555 write 0x00f1cc thru 0x013061 (0x3e96 bytes) 000188 1786748411.037875 write 0x00ada7 thru 0x00de00 (0x305a bytes) 000190 1786748411.046060 mapwrite 0x024b68 thru 0x0307f9 (0xbc92 bytes) 000191 1786748411.138270 mapwrite 0x02c8c7 thru 0x0377ad (0xaee7 bytes) 000194 1786748411.214316 mapread 0x027261 thru 0x031b6c (0xa90c bytes) 000197 1786748411.218691 mapread 0x0163ae thru 0x018bd2 (0x2825 bytes) 000198 1786748411.231445 mapwrite 0x0152d1 thru 0x01a0f0 (0x4e20 bytes) 000199 1786748411.292171 trunc from 0x0377ae to 0x01ed1f (0x18a8e bytes) 000200 1786748411.309349 write 0x0172e8 thru 0x01e032 (0x6d4b bytes) 000201 1786748411.311121 mapread 0x007f7d thru 0x00fe35 (0x7eb9 bytes) 000202 1786748411.318815 MIRROR_RESYNC 7 mirrors 000203 1786748412.625339 trunc from 0x01ed20 to 0x00588d (0x19492 bytes) 000204 1786748412.707367 mapwrite 0x0174c5 thru 0x021aec (0xa628 bytes) 000205 1786748412.836855 write 0x033dad thru 0x039882 (0x5ad6 bytes) 000206 1786748412.844062 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39883): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39883): Mirror 2: 0x27157625 (stale) Mirror 3: 0x27157625 (stale) Mirror 4: 0x27157625 (stale) Mirror 5: 0x27157625 (stale) Mirror 7: 0x27157625 (stale) Mirror 8: 0x27157625 (stale) CRC-32 checksum value for chunk [0, 0x39883): Mirror 1: 0x41dc1e80 Mirror 9: 0x41dc1e80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39883: skipped 000208 1786748414.580078 read 0x014ca1 thru 0x016588 (0x18e8 bytes) 000209 1786748414.613272 write 0x03e04e thru 0x03ffff (0x1fb2 bytes) 000213 1786748414.751180 read 0x02a7c3 thru 0x02e634 (0x3e72 bytes) 000214 1786748414.766044 mapread 0x0332b1 thru 0x03ffff (0xcd4f bytes) 000219 1786748414.773319 trunc from 0x040000 to 0x001330 (0x3eccf bytes) 000220 1786748414.801206 mapwrite 0x024e6c thru 0x02c807 (0x799c bytes) 000221 1786748414.960427 mapwrite 0x022b9a thru 0x0286af (0x5b16 bytes) 000225 1786748415.032390 mapwrite 0x0314da thru 0x0359cf (0x44f6 bytes) 000226 1786748415.155045 mapread 0x02190b thru 0x024f7b (0x3671 bytes) 000228 1786748415.164820 mapread 0x00b4c7 thru 0x010586 (0x50c0 bytes) 000229 1786748415.199090 trunc from 0x0359d0 to 0x00d842 (0x2818d bytes) 000231 1786748415.233110 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xd843): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd843): Mirror 2: 0xb2469d81 (stale) Mirror 3: 0xb2469d81 (stale) Mirror 4: 0xb2469d81 (stale) Mirror 5: 0xb2469d81 (stale) Mirror 7: 0xb2469d81 (stale) Mirror 8: 0xb2469d81 (stale) Mirror 9: 0x5e59c2ea (stale) CRC-32 checksum value for chunk [0, 0xd843): Mirror 1: 0x5e59c2ea Mirror 10: 0x5e59c2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd843: skipped 000232 1786748416.467996 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd843): Mirror 2: 0xb2469d81 (stale) Mirror 3: 0xb2469d81 (stale) Mirror 4: 0xb2469d81 (stale) Mirror 5: 0xb2469d81 (stale) Mirror 8: 0xb2469d81 (stale) Mirror 9: 0x5e59c2ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd843): Mirror 2: 0xb2469d81 (stale) Mirror 3: 0xb2469d81 (stale) Mirror 4: 0xb2469d81 (stale) Mirror 5: 0xb2469d81 (stale) Mirror 8: 0xb2469d81 (stale) Mirror 9: 0x5e59c2ea (stale) CRC-32 checksum value for chunk [0, 0xd843): Mirror 1: 0x5e59c2ea Mirror 10: 0x5e59c2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd843: skipped 000233 1786748417.288534 read 0x00486e thru 0x00d842 (0x8fd5 bytes) 000235 1786748417.346542 write 0x03f0c1 thru 0x03ffff (0xf3f bytes) 000236 1786748417.423538 mapwrite 0x00d80b thru 0x016c5d (0x9453 bytes) 000237 1786748417.562349 write 0x01f0a8 thru 0x028d21 (0x9c7a bytes) 000239 1786748417.594733 mapwrite 0x018a86 thru 0x02452a (0xbaa5 bytes) 000241 1786748417.681986 trunc from 0x040000 to 0x0099c9 (0x36636 bytes) truncating to largest ever: 0x3ae1b 000242 1786748417.709336 trunc from 0x0099ca to 0x03ae1a (0x31451 bytes) 000243 1786748417.728883 read 0x006d06 thru 0x01582d (0xeb28 bytes) 000247 1786748417.744848 read 0x01fe3d thru 0x0224aa (0x266e bytes) 000249 1786748417.750180 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae1b): Mirror 3: 0xb471e149 (stale) Mirror 4: 0xb471e149 (stale) Mirror 5: 0xb471e149 (stale) Mirror 8: 0xb471e149 (stale) Mirror 9: 0x9e139c46 (stale) Mirror 10: 0xeb2ad3aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae1b): Mirror 3: 0xb471e149 (stale) Mirror 4: 0xb471e149 (stale) Mirror 5: 0xb471e149 (stale) Mirror 8: 0xb471e149 (stale) Mirror 9: 0x9e139c46 (stale) Mirror 10: 0xeb2ad3aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae1b: skipped 000251 1786748418.238401 read 0x03aac3 thru 0x03ae1a (0x358 bytes) 000253 1786748418.245020 read 0x0119c0 thru 0x012913 (0xf54 bytes) 000256 1786748418.261071 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae1b): Mirror 3: 0xb471e149 (stale) Mirror 4: 0xb471e149 (stale) Mirror 8: 0xb471e149 (stale) Mirror 9: 0x9e139c46 (stale) Mirror 10: 0xeb2ad3aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae1b): Mirror 3: 0xb471e149 (stale) Mirror 4: 0xb471e149 (stale) Mirror 8: 0xb471e149 (stale) Mirror 9: 0x9e139c46 (stale) Mirror 10: 0xeb2ad3aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae1b: skipped 000257 1786748418.663464 write 0x0287d6 thru 0x02e4ae (0x5cd9 bytes) 000258 1786748418.679083 read 0x019606 thru 0x02302e (0x9a29 bytes) 000260 1786748418.743449 MIRROR_RESYNC 6 mirrors 000261 1786748420.871494 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000263 1786748421.200942 trunc from 0x03ae1b to 0x024798 (0x16682 bytes) 000264 1786748421.284014 write 0x02a6b5 thru 0x02f370 (0x4cbc bytes) 000266 1786748421.286713 mapwrite 0x03bd5a thru 0x03dfa3 (0x224a bytes) 000267 1786748421.375757 mapread 0x006178 thru 0x00631c (0x1a5 bytes) 000268 1786748421.385310 read 0x01c1c3 thru 0x01ec88 (0x2ac6 bytes) 000269 1786748421.400672 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dfa4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfa4): Mirror 3: 0x4fa85a2f (stale) Mirror 4: 0x4fa85a2f (stale) Mirror 8: 0x4fa85a2f (stale) Mirror 10: 0x4fa85a2f (stale) CRC-32 checksum value for chunk [0, 0x3dfa4): Mirror 1: 0xbbe581a8 Mirror 11: 0xbbe581a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfa4: skipped 000270 1786748422.599421 MIRROR_RESYNC 6 mirrors 000272 1786748424.386926 trunc from 0x03dfa4 to 0x004fe1 (0x38fc2 bytes) 000273 1786748424.471144 mapread 0x002195 thru 0x004fe1 (0x2e4d bytes) 000275 1786748424.486238 mapread 0x0003d9 thru 0x001106 (0xd2e bytes) 000279 1786748424.494451 read 0x0044a4 thru 0x004fe1 (0xb3e bytes) 000281 1786748424.500633 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4fe2): Mirror 4: 0xd3716462 (stale) Mirror 8: 0xd3716462 (stale) Mirror 10: 0xd3716462 (stale) Mirror 11: 0xd3716462 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4fe2): Mirror 4: 0xd3716462 (stale) Mirror 8: 0xd3716462 (stale) Mirror 10: 0xd3716462 (stale) Mirror 11: 0xd3716462 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4fe2: skipped 000282 1786748424.839687 write 0x012e16 thru 0x016837 (0x3a22 bytes) 000283 1786748424.845536 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x16838): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16838): Mirror 4: 0xf4b0f77f (stale) Mirror 8: 0xf4b0f77f (stale) Mirror 10: 0xf4b0f77f (stale) Mirror 11: 0xf4b0f77f (stale) CRC-32 checksum value for chunk [0, 0x16838): Mirror 1: 0x60189f63 Mirror 12: 0x60189f63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16838: skipped 000284 1786748425.841858 write 0x03e763 thru 0x03ffff (0x189d bytes) 000286 1786748425.911355 mapwrite 0x0008f5 thru 0x00dbaf (0xd2bb bytes) 000287 1786748425.985887 mapwrite 0x0212d4 thru 0x029c1c (0x8949 bytes) 000289 1786748426.160371 write 0x030820 thru 0x03dd8a (0xd56b bytes) 000293 1786748426.179630 trunc from 0x040000 to 0x01b513 (0x24aec bytes) 000294 1786748426.206106 read 0x00ea2c thru 0x012e7b (0x4450 bytes) 000295 1786748426.212788 write 0x013fd3 thru 0x01f91f (0xb94d bytes) 000297 1786748426.228929 trunc from 0x01f920 to 0x0292f4 (0x99d5 bytes) 000299 1786748426.288240 MIRROR_RESYNC 6 mirrors 000302 1786748427.861218 read 0x014186 thru 0x01aa7b (0x68f6 bytes) 000303 1786748427.874171 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000305 1786748428.279741 mapwrite 0x0195e2 thru 0x01f6f0 (0x610f bytes) 000306 1786748428.478566 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x292f5): Mirror 10: 0x251df465 (stale) Mirror 11: 0x251df465 (stale) Mirror 12: 0x251df465 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x292f5): Mirror 10: 0x251df465 (stale) Mirror 11: 0x251df465 (stale) Mirror 12: 0x251df465 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x292f5: skipped 000307 1786748428.869465 trunc from 0x0292f5 to 0x01a737 (0xebbd bytes) 000308 1786748428.905905 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a738): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a738): Mirror 10: 0x7cef5cde (stale) Mirror 11: 0x7cef5cde (stale) Mirror 12: 0x7cef5cde (stale) CRC-32 checksum value for chunk [0, 0x1a738): Mirror 1: 0x12ce0574 Mirror 13: 0x12ce0574 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a738: skipped 000312 1786748429.917869 mapread 0x00aa56 thru 0x01a699 (0xfc44 bytes) 000313 1786748429.964912 mapread 0x010b77 thru 0x018b98 (0x8022 bytes) 000315 1786748429.992815 mapwrite 0x03510d thru 0x03c4da (0x73ce bytes) 000316 1786748430.174269 mapwrite 0x01c6ef thru 0x027ba9 (0xb4bb bytes) 000317 1786748430.301907 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c4db): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4db): Mirror 10: 0x322caf17 (stale) Mirror 11: 0x322caf17 (stale) Mirror 12: 0x322caf17 (stale) Mirror 13: 0xc3b5a14d (stale) CRC-32 checksum value for chunk [0, 0x3c4db): Mirror 1: 0xe40755cb Mirror 14: 0xe40755cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4db: skipped 000318 1786748431.234374 read 0x01f79e thru 0x0257ff (0x6062 bytes) 000322 1786748431.273821 mapread 0x02e589 thru 0x03ae5d (0xc8d5 bytes) 000323 1786748431.333079 read 0x00233d thru 0x0047ba (0x247e bytes) 000324 1786748431.353577 read 0x030195 thru 0x035f90 (0x5dfc bytes) 000325 1786748431.366930 read 0x02eb6f thru 0x038721 (0x9bb3 bytes) 000326 1786748431.373060 mapread 0x019690 thru 0x022cd7 (0x9648 bytes) 000328 1786748431.397552 mapwrite 0x0175cd thru 0x0178a3 (0x2d7 bytes) 000330 1786748431.547437 trunc from 0x03c4db to 0x00595d (0x36b7d bytes) 000332 1786748431.589932 trunc from 0x00595e to 0x000b49 (0x4e14 bytes) 000334 1786748431.616532 read 0x0005a5 thru 0x000b49 (0x5a5 bytes) 000336 1786748431.622021 mapread 0x000825 thru 0x000b49 (0x325 bytes) 000339 1786748431.631530 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb4a): Mirror 10: 0xbca2264 (stale) Mirror 11: 0xbca2264 (stale) Mirror 13: 0xbca2264 (stale) Mirror 14: 0xbca2264 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb4a): Mirror 10: 0xbca2264 (stale) Mirror 11: 0xbca2264 (stale) Mirror 13: 0xbca2264 (stale) Mirror 14: 0xbca2264 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb4a: skipped 000340 1786748431.968419 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb4a): Mirror 10: 0xbca2264 (stale) Mirror 13: 0xbca2264 (stale) Mirror 14: 0xbca2264 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb4a): Mirror 10: 0xbca2264 (stale) Mirror 13: 0xbca2264 (stale) Mirror 14: 0xbca2264 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb4a: skipped 000341 1786748432.270510 trunc from 0x000b4a to 0x03516f (0x34626 bytes) 000342 1786748432.295526 write 0x03bd1d thru 0x03f720 (0x3a04 bytes) 000344 1786748432.304048 MIRROR_RESYNC 4 mirrors 000345 1786748433.616816 MIRROR_RESYNC 4 mirrors 000346 1786748434.133736 MIRROR_RESYNC 4 mirrors 000347 1786748434.621131 write 0x03b4f5 thru 0x03f5a5 (0x40b1 bytes) 000348 1786748434.742393 read 0x033fa0 thru 0x038c11 (0x4c72 bytes) 000349 1786748434.749930 mapwrite 0x034026 thru 0x0363b0 (0x238b bytes) 000350 1786748434.798884 trunc from 0x03f721 to 0x033bb0 (0xbb70 bytes) 000351 1786748434.836956 MIRROR_RESYNC 4 mirrors 000352 1786748436.031972 mapwrite 0x032cf2 thru 0x03ffff (0xd30e bytes) 000353 1786748436.256703 mapread 0x0099d6 thru 0x00dd2e (0x4359 bytes) 000354 1786748436.285089 write 0x025c3e thru 0x025c72 (0x35 bytes) 000355 1786748436.299987 trunc from 0x040000 to 0x031fe9 (0xe016 bytes) 000357 1786748436.319722 write 0x0298de thru 0x02a97e (0x10a1 bytes) 000359 1786748436.336334 write 0x012624 thru 0x01d7c0 (0xb19d bytes) 000361 1786748436.361838 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31fea): Mirror 10: 0x1d2bd741 (stale) Mirror 14: 0x1d2bd741 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fea): Mirror 10: 0x1d2bd741 (stale) Mirror 14: 0x1d2bd741 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fea: skipped 000362 1786748436.654140 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31fea): Mirror 10: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fea): Mirror 10: 0x1d2bd741 (stale) Mirror 14: 0x1d2bd741 (stale) CRC-32 checksum value for chunk [0, 0x31fea): Mirror 1: 0x6a79f54b Mirror 15: 0x6a79f54b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fea: skipped 000363 1786748437.690390 read 0x02f0f8 thru 0x031fe9 (0x2ef2 bytes) 000364 1786748437.711754 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31fea): Mirror 14: 0x1d2bd741 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fea): Mirror 14: 0x1d2bd741 (stale) CRC-32 checksum value for chunk [0, 0x31fea): Mirror 1: 0x6a79f54b Mirror 15: 0x6a79f54b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fea: skipped 000365 1786748438.259427 mapread 0x00c889 thru 0x0195dc (0xcd54 bytes) 000366 1786748438.335810 mapwrite 0x00a9f7 thru 0x00e37d (0x3987 bytes) 000367 1786748438.620524 mapwrite 0x012cb8 thru 0x018351 (0x569a bytes) 000375 1786748438.696390 mapread 0x0267e9 thru 0x02e2d5 (0x7aed bytes) 000377 1786748438.712259 trunc from 0x031fea to 0x02dd5e (0x428b bytes) 000378 1786748438.743321 read 0x0042c7 thru 0x00a1f1 (0x5f2b bytes) 000380 1786748438.744983 read 0x014528 thru 0x016fc7 (0x2aa0 bytes) 000382 1786748438.747552 read 0x01c710 thru 0x02acd1 (0xe5c2 bytes) 000383 1786748438.757536 mapread 0x012326 thru 0x0201ca (0xdea5 bytes) 000384 1786748438.759462 read 0x001dea thru 0x00d55e (0xb775 bytes) 000386 1786748438.765908 MIRROR_RESYNC 3 mirrors 000387 1786748439.479557 mapwrite 0x007158 thru 0x00b70e (0x45b7 bytes) 000389 1786748439.720055 MIRROR_RESYNC 3 mirrors 000390 1786748440.591101 mapread 0x006bdd thru 0x00f68c (0x8ab0 bytes) 000395 1786748440.613671 mapread 0x0023be thru 0x00d89a (0xb4dd bytes) 000396 1786748440.614888 trunc from 0x02dd5f to 0x028559 (0x5805 bytes) 000397 1786748440.689361 write 0x01e5d2 thru 0x02918e (0xabbd bytes) 000399 1786748440.701889 mapread 0x01d410 thru 0x022d2b (0x591c bytes) 000400 1786748440.716756 mapwrite 0x007808 thru 0x00d14b (0x5944 bytes) 000401 1786748440.820358 read 0x01fc51 thru 0x02918e (0x953e bytes) 000402 1786748440.826095 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2918f): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2918f): Mirror 14: 0x9cb8e599 (stale) Mirror 15: 0x9cb8e599 (stale) CRC-32 checksum value for chunk [0, 0x2918f): Mirror 1: 0xd6764131 Mirror 16: 0xd6764131 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2918f: skipped 000403 1786748441.841234 mapwrite 0x03ea2c thru 0x03ffff (0x15d4 bytes) 000404 1786748442.036483 trunc from 0x040000 to 0x0203ab (0x1fc54 bytes) 000407 1786748442.092371 write 0x0011ac thru 0x010001 (0xee56 bytes) 000409 1786748442.115552 trunc from 0x0203ac to 0x01f5ca (0xde1 bytes) 000410 1786748442.135865 write 0x017062 thru 0x022f3d (0xbedc bytes) 000412 1786748442.146300 mapwrite 0x02775f thru 0x027dae (0x650 bytes) 000413 1786748442.277780 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27daf): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27daf): Mirror 14: 0x79260430 (stale) Mirror 15: 0x79260430 (stale) Mirror 16: 0xb8880bbd (stale) CRC-32 checksum value for chunk [0, 0x27daf): Mirror 1: 0x9485d82 Mirror 17: 0x9485d82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27daf: skipped truncating to largest ever: 0x3e8c3 000414 1786748442.973467 trunc from 0x027daf to 0x03e8c2 (0x16b14 bytes) 000415 1786748443.097859 MIRROR_RESYNC 5 mirrors 000417 1786748444.426447 read 0x013313 thru 0x01f0e3 (0xbdd1 bytes) 000420 1786748444.442772 write 0x020f64 thru 0x02f481 (0xe51e bytes) 000421 1786748444.516019 read 0x01b644 thru 0x01ba22 (0x3df bytes) 000422 1786748444.530684 mapwrite 0x02a914 thru 0x0314b9 (0x6ba6 bytes) 000423 1786748444.601852 mapread 0x01252c thru 0x018c97 (0x676c bytes) 000424 1786748444.603005 read 0x016e7d thru 0x01d8ef (0x6a73 bytes) 000425 1786748444.603673 trunc from 0x03e8c3 to 0x02923d (0x15685 bytes) 000426 1786748444.629828 write 0x03081d thru 0x03fc97 (0xf47b bytes) 000428 1786748444.642522 write 0x021556 thru 0x02dc52 (0xc6fd bytes) 000429 1786748444.658415 mapread 0x03b4ab thru 0x03fc97 (0x47ed bytes) 000431 1786748444.664428 trunc from 0x03fc98 to 0x025118 (0x1ab7f bytes) 000432 1786748444.682107 write 0x03c6f2 thru 0x03ffff (0x390e bytes) 000434 1786748444.690542 trunc from 0x040000 to 0x03b0a3 (0x4f5c bytes) 000435 1786748444.720328 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b0a4): Mirror 14: 0xdd6028b4 (stale) Mirror 15: 0xdd6028b4 (stale) Mirror 16: 0xdd6028b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b0a4): Mirror 14: 0xdd6028b4 (stale) Mirror 15: 0xdd6028b4 (stale) Mirror 16: 0xdd6028b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b0a4: skipped 000436 1786748445.219576 trunc from 0x03b0a4 to 0x0183dc (0x22cc7 bytes) 000437 1786748445.234372 mapwrite 0x035bf9 thru 0x0389ce (0x2dd6 bytes) 000438 1786748445.315421 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x389cf): Mirror 14: 0x7a3821e0 (stale) Mirror 15: 0x7a3821e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389cf): Mirror 14: 0x7a3821e0 (stale) Mirror 15: 0x7a3821e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389cf: skipped 000439 1786748445.651155 trunc from 0x0389cf to 0x000f1f (0x37aaf bytes) 000440 1786748445.679723 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf20): Mirror 14: 0x1a6fe051 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf20): Mirror 14: 0x1a6fe051 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf20: skipped 000441 1786748445.961356 MIRROR_RESYNC 2 mirrors 000443 1786748446.652441 trunc from 0x000f20 to 0x020de6 (0x1fec7 bytes) 000444 1786748446.735220 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20de7): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20de7): Mirror 14: 0xedcd4460 (stale) CRC-32 checksum value for chunk [0, 0x20de7): Mirror 1: 0xedcd4460 Mirror 15: 0xedcd4460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20de7: skipped 000445 1786748447.747457 write 0x01196c thru 0x01dcd5 (0xc36a bytes) 000447 1786748447.916359 mapwrite 0x014474 thru 0x023ff5 (0xfb82 bytes) 000448 1786748448.013819 mapwrite 0x02e55d thru 0x02e76a (0x20e bytes) 000451 1786748448.094176 write 0x004745 thru 0x00e853 (0xa10f bytes) 000452 1786748448.104094 trunc from 0x02e76b to 0x019227 (0x15543 bytes) 000454 1786748448.120823 mapread 0x00f3ca thru 0x015486 (0x60bd bytes) 000455 1786748448.127943 mapread 0x005b1d thru 0x01584f (0xfd33 bytes) 000456 1786748448.129453 read 0x012812 thru 0x019227 (0x6a16 bytes) 000457 1786748448.131237 read 0x013daf thru 0x019227 (0x5479 bytes) 000458 1786748448.132352 mapwrite 0x00958a thru 0x00dc77 (0x46ee bytes) 000459 1786748448.175037 write 0x03683c thru 0x03ffff (0x97c4 bytes) 000460 1786748448.181760 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe6d050fd (stale) Mirror 15: 0xe6d050fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x45756da9 Mirror 16: 0x45756da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000462 1786748449.425762 write 0x0204cb thru 0x023e1c (0x3952 bytes) 000464 1786748449.499975 trunc from 0x040000 to 0x000390 (0x3fc6f bytes) 000465 1786748449.542146 trunc from 0x000391 to 0x016ed5 (0x16b45 bytes) 000466 1786748449.560160 read 0x00e9a9 thru 0x014428 (0x5a80 bytes) 000467 1786748449.570080 write 0x01c5fe thru 0x02424a (0x7c4d bytes) 000469 1786748449.572514 mapwrite 0x024f68 thru 0x0285d1 (0x366a bytes) 000471 1786748449.676112 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285d2): Mirror 14: 0xb3fd9f6c (stale) Mirror 16: 0x3b59fe0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285d2): Mirror 14: 0xb3fd9f6c (stale) Mirror 16: 0x3b59fe0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285d2: skipped 000473 1786748450.091060 mapread 0x019737 thru 0x025778 (0xc042 bytes) 000474 1786748450.145156 trunc from 0x0285d2 to 0x01d763 (0xae6e bytes) 000476 1786748450.176071 trunc from 0x01d764 to 0x00a6f4 (0x1306f bytes) 000478 1786748450.213288 read 0x0013ae thru 0x00a6f4 (0x9347 bytes) 000482 1786748450.252127 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 14: 0x849271d2 (stale) Mirror 16: 0xbb3115ad (stale) CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 1: 0x33d1c752 Mirror 17: 0x33d1c752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa6f5: skipped 000483 1786748451.704027 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 14: 0x849271d2 (stale) Mirror 16: 0xbb3115ad (stale) CRC-32 checksum value for chunk [0, 0xa6f5): Mirror 1: 0x33d1c752 Mirror 17: 0x33d1c752 Mirror 18: 0x33d1c752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa6f5: skipped 000484 1786748452.652686 mapread 0x005a57 thru 0x00a6f4 (0x4c9e bytes) 000485 1786748452.711861 trunc from 0x00a6f5 to 0x0362b1 (0x2bbbd bytes) 000486 1786748452.867533 mapread 0x005f74 thru 0x010a43 (0xaad0 bytes) 000488 1786748452.882803 mapwrite 0x00056b thru 0x010431 (0xfec7 bytes) 000489 1786748453.013518 read 0x02e515 thru 0x0362b1 (0x7d9d bytes) 000490 1786748453.028077 mapread 0x023f8a thru 0x032247 (0xe2be bytes) 000491 1786748453.053337 read 0x00b113 thru 0x01918e (0xe07c bytes) 000494 1786748453.073162 trunc from 0x0362b2 to 0x02cbe5 (0x96cc bytes) 000497 1786748453.101112 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2cbe6): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cbe6): Mirror 14: 0xde62d7d9 (stale) Mirror 16: 0x73a9baec (stale) Mirror 17: 0x26bb6dc4 (stale) Mirror 18: 0x26bb6dc4 (stale) CRC-32 checksum value for chunk [0, 0x2cbe6): Mirror 1: 0xe5dbc966 Mirror 19: 0xe5dbc966 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cbe6: skipped 000499 1786748453.994050 trunc from 0x02cbe6 to 0x02ac8a (0x1f5b bytes) 000500 1786748454.089079 mapwrite 0x010562 thru 0x013e8b (0x392a bytes) 000502 1786748454.184329 mapwrite 0x01746a thru 0x025861 (0xe3f8 bytes) 000505 1786748454.343922 mapread 0x01b6d6 thru 0x02ac8a (0xf5b5 bytes) 000506 1786748454.350372 read 0x001753 thru 0x00ab58 (0x9406 bytes) 000507 1786748454.365497 trunc from 0x02ac8b to 0x00c770 (0x1e51a bytes) 000509 1786748454.399172 read 0x005c9a thru 0x00ac70 (0x4fd7 bytes) 000511 1786748454.402248 trunc from 0x00c771 to 0x01eeb7 (0x12747 bytes) 000512 1786748454.440493 MIRROR_RESYNC 6 mirrors 000513 1786748456.077321 mapwrite 0x0143e1 thru 0x01b4ba (0x70da bytes) 000514 1786748456.316328 mapwrite 0x0331cf thru 0x039336 (0x6168 bytes) 000515 1786748456.497180 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39337): Mirror 14: 0x2bc9de0a (stale) Mirror 16: 0x2bc9de0a (stale) Mirror 18: 0x2bc9de0a (stale) Mirror 19: 0x2bc9de0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39337): Mirror 14: 0x2bc9de0a (stale) Mirror 16: 0x2bc9de0a (stale) Mirror 18: 0x2bc9de0a (stale) Mirror 19: 0x2bc9de0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39337: skipped 000516 1786748456.886434 trunc from 0x039337 to 0x01cb6e (0x1c7c8 bytes) 000518 1786748456.915355 mapread 0x015451 thru 0x01a153 (0x4d03 bytes) 000522 1786748456.946259 mapwrite 0x01b547 thru 0x01f4ac (0x3f66 bytes) 000523 1786748457.047889 mapread 0x01616e thru 0x01f4ac (0x933f bytes) 000524 1786748457.049156 trunc from 0x01f4ad to 0x01f1ad (0x2ff bytes) 000525 1786748457.072066 trunc from 0x01f1ae to 0x003a7c (0x1b731 bytes) 000526 1786748457.092784 write 0x001d09 thru 0x00a936 (0x8c2e bytes) 000527 1786748457.103266 MIRROR_RESYNC 5 mirrors 000530 1786748458.262973 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000534 1786748458.672698 write 0x0208e3 thru 0x02672d (0x5e4b bytes) 000535 1786748458.784333 write 0x015fee thru 0x0183ff (0x2412 bytes) 000536 1786748458.803329 write 0x019375 thru 0x01c183 (0x2e0f bytes) 000538 1786748458.824210 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2672e): Mirror 14: 0x1e424f8c (stale) Mirror 19: 0x1e424f8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2672e): Mirror 14: 0x1e424f8c (stale) Mirror 19: 0x1e424f8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2672e: skipped 000539 1786748459.166240 mapwrite 0x0221d6 thru 0x0244be (0x22e9 bytes) 000543 1786748459.240421 MIRROR_RESYNC 3 mirrors 000544 1786748460.170551 trunc from 0x02672e to 0x01ea69 (0x7cc4 bytes) 000546 1786748460.272290 trunc from 0x01ea6a to 0x0197f2 (0x5277 bytes) 000549 1786748460.303217 read 0x004c03 thru 0x007577 (0x2975 bytes) 000551 1786748460.314283 read 0x00966f thru 0x010c46 (0x75d8 bytes) 000552 1786748460.329264 trunc from 0x0197f3 to 0x018d26 (0xacc bytes) 000560 1786748460.365134 trunc from 0x018d27 to 0x0243b0 (0xb68a bytes) 000561 1786748460.380053 mapwrite 0x00d89c thru 0x018cbb (0xb420 bytes) 000564 1786748460.478991 read 0x0221eb thru 0x0243b0 (0x21c6 bytes) 000565 1786748460.516385 mapwrite 0x01b50b thru 0x01c865 (0x135b bytes) 000566 1786748460.656930 trunc from 0x0243b1 to 0x023457 (0xf59 bytes) 000567 1786748460.688714 write 0x008166 thru 0x015c00 (0xda9b bytes) 000569 1786748460.691292 write 0x02b5f7 thru 0x02d55b (0x1f65 bytes) 000570 1786748460.692516 write 0x026ee2 thru 0x02c0c7 (0x51e6 bytes) 000571 1786748460.700007 trunc from 0x02d55c to 0x006690 (0x26ecb bytes) 000573 1786748460.721446 write 0x023d87 thru 0x02721a (0x3494 bytes) 000576 1786748460.726463 mapread 0x013a23 thru 0x01ead3 (0xb0b1 bytes) 000578 1786748460.781551 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2721b): Mirror 14: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2721b): Mirror 14: 0xdc89f6ff (stale) Mirror 19: 0xdc89f6ff (stale) CRC-32 checksum value for chunk [0, 0x2721b): Mirror 1: 0x3759ea63 Mirror 20: 0x3759ea63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2721b: skipped 000581 1786748462.158988 mapread 0x00059f thru 0x00f230 (0xec92 bytes) 000582 1786748462.230140 trunc from 0x02721b to 0x0198e9 (0xd931 bytes) 000583 1786748462.359361 read 0x00502e thru 0x009869 (0x483c bytes) 000586 1786748462.368692 write 0x0032e3 thru 0x0084d7 (0x51f5 bytes) 000587 1786748462.377372 trunc from 0x0198ea to 0x03b9b7 (0x220ce bytes) 000589 1786748462.404767 mapwrite 0x01355f thru 0x01aa94 (0x7536 bytes) 000591 1786748462.467893 write 0x003a7c thru 0x006eed (0x3472 bytes) 000592 1786748462.469245 trunc from 0x03b9b8 to 0x012471 (0x29546 bytes) 000593 1786748462.493680 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12472): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12472): Mirror 14: 0xacc7ea46 (stale) Mirror 19: 0xacc7ea46 (stale) Mirror 20: 0x51f051c0 (stale) CRC-32 checksum value for chunk [0, 0x12472): Mirror 1: 0x2c5896ba Mirror 21: 0x2c5896ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12472: skipped 000595 1786748463.193297 read 0x00d651 thru 0x011f0d (0x48bd bytes) 000596 1786748463.228010 write 0x017a35 thru 0x01d629 (0x5bf5 bytes) 000599 1786748463.378804 read 0x01367b thru 0x017a1b (0x43a1 bytes) 000601 1786748463.397359 trunc from 0x01d62a to 0x0076ab (0x15f7e bytes) 000605 1786748463.467272 write 0x031a0d thru 0x037259 (0x584d bytes) 000607 1786748463.472501 read 0x00aae4 thru 0x0182f5 (0xd812 bytes) 000609 1786748463.487376 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3725a): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3725a): Mirror 14: 0xeb00a81a (stale) Mirror 19: 0xeb00a81a (stale) Mirror 20: 0x4c2ba135 (stale) Mirror 21: 0xb87075f0 (stale) CRC-32 checksum value for chunk [0, 0x3725a): Mirror 1: 0xd978edfa Mirror 22: 0xd978edfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3725a: skipped 000610 1786748464.675116 write 0x03717a thru 0x03ffff (0x8e86 bytes) 000611 1786748464.767219 mapread 0x02ef96 thru 0x031bbb (0x2c26 bytes) 000613 1786748464.789259 trunc from 0x040000 to 0x0399d4 (0x662b bytes) 000615 1786748464.817426 read 0x01ad78 thru 0x01bec2 (0x114b bytes) 000616 1786748464.827119 trunc from 0x0399d5 to 0x03948c (0x548 bytes) 000618 1786748464.847452 read 0x0389f4 thru 0x03948c (0xa99 bytes) 000620 1786748464.849478 trunc from 0x03948d to 0x0019c3 (0x37ac9 bytes) 000621 1786748464.882344 read 0x0010d4 thru 0x0019c3 (0x8f0 bytes) 000622 1786748464.908278 trunc from 0x0019c4 to 0x0186b4 (0x16cf1 bytes) 000623 1786748464.949369 read 0x008110 thru 0x0151d4 (0xd0c5 bytes) 000624 1786748464.995068 mapwrite 0x015ec6 thru 0x01ad9d (0x4ed8 bytes) 000626 1786748465.178921 read 0x00afad thru 0x019a22 (0xea76 bytes) 000627 1786748465.189456 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ad9e): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad9e): Mirror 14: 0x2ddbb5cd (stale) Mirror 19: 0x2ddbb5cd (stale) Mirror 20: 0xa58e1ddb (stale) Mirror 21: 0xbedd2a1d (stale) Mirror 22: 0x1038104e (stale) CRC-32 checksum value for chunk [0, 0x1ad9e): Mirror 1: 0xab40e6b7 Mirror 23: 0xab40e6b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad9e: skipped 000630 1786748466.599852 read 0x00e9a8 thru 0x01ad9d (0xc3f6 bytes) 000631 1786748466.616967 read 0x018115 thru 0x01ad9d (0x2c89 bytes) 000633 1786748466.632383 mapwrite 0x0196f4 thru 0x029483 (0xfd90 bytes) 000634 1786748466.882289 mapwrite 0x01f3c5 thru 0x02dd7b (0xe9b7 bytes) 000635 1786748466.985951 read 0x02b35f thru 0x02dd7b (0x2a1d bytes) 000637 1786748466.987207 write 0x001f98 thru 0x00961c (0x7685 bytes) 000638 1786748466.989405 write 0x02b6af thru 0x02d0b7 (0x1a09 bytes) 000639 1786748466.991131 read 0x00ef1a thru 0x01d294 (0xe37b bytes) 000640 1786748466.992342 mapread 0x01ce8e thru 0x02c1a7 (0xf31a bytes) 000641 1786748466.994396 read 0x01df4a thru 0x0253cc (0x7483 bytes) 000643 1786748466.996096 mapread 0x01c877 thru 0x023289 (0x6a13 bytes) 000645 1786748466.998779 trunc from 0x02dd7c to 0x02729d (0x6ade bytes) 000646 1786748467.019082 read 0x01d9e9 thru 0x023a83 (0x609b bytes) 000649 1786748467.022379 write 0x025cc6 thru 0x027aa1 (0x1ddc bytes) 000650 1786748467.023978 trunc from 0x027aa2 to 0x024e89 (0x2c18 bytes) 000652 1786748467.045940 write 0x02d049 thru 0x02fafb (0x2ab3 bytes) 000654 1786748467.048163 read 0x026c01 thru 0x02f7ef (0x8bef bytes) 000658 1786748467.058564 write 0x02fcaf thru 0x03a03b (0xa38d bytes) 000659 1786748467.060576 trunc from 0x03a03c to 0x03a065 (0x2a bytes) 000661 1786748467.083202 trunc from 0x03a066 to 0x01818c (0x21ed9 bytes) 000662 1786748467.099148 trunc from 0x01818d to 0x034ff3 (0x1ce67 bytes) 000665 1786748467.120987 MIRROR_RESYNC 7 mirrors 000666 1786748468.774530 MIRROR_RESYNC 7 mirrors 000667 1786748469.270567 mapread 0x023c3e thru 0x030206 (0xc5c9 bytes) 000670 1786748469.333139 read 0x0139be thru 0x01c39b (0x89de bytes) 000671 1786748469.351676 trunc from 0x034ff4 to 0x0097ea (0x2b809 bytes) 000672 1786748469.480471 mapwrite 0x016e3b thru 0x01b678 (0x483e bytes) 000674 1786748469.622702 trunc from 0x01b679 to 0x00a004 (0x11674 bytes) 000675 1786748469.646088 trunc from 0x00a005 to 0x028f0a (0x1ef06 bytes) 000681 1786748469.669568 mapread 0x014b6b thru 0x017e71 (0x3307 bytes) 000683 1786748469.680320 mapread 0x00897f thru 0x00e1bc (0x583e bytes) 000684 1786748469.689371 read 0x00a695 thru 0x012e17 (0x8783 bytes) 000687 1786748469.691718 read 0x004c08 thru 0x00e190 (0x9589 bytes) 000689 1786748469.693528 mapread 0x0279ae thru 0x028f0a (0x155d bytes) 000691 1786748469.704807 trunc from 0x028f0b to 0x013342 (0x15bc8 bytes) 000693 1786748469.719783 write 0x024937 thru 0x0297ab (0x4e75 bytes) 000694 1786748469.726324 MIRROR_RESYNC 7 mirrors 000697 1786748472.468269 trunc from 0x0297ac to 0x00591e (0x23e8d bytes) 000698 1786748472.546139 mapread 0x003df1 thru 0x00591e (0x1b2e bytes) 000701 1786748472.569657 read 0x001f68 thru 0x00591e (0x39b7 bytes) 000702 1786748472.570484 mapread 0x000e92 thru 0x00591e (0x4a8d bytes) 000704 1786748472.577298 write 0x0123a4 thru 0x01f549 (0xd1a6 bytes) 000707 1786748472.581918 write 0x026ab6 thru 0x027139 (0x684 bytes) 000708 1786748472.583302 write 0x037725 thru 0x03fe4d (0x8729 bytes) 000710 1786748472.585482 mapread 0x011e3b thru 0x016ce8 (0x4eae bytes) 000711 1786748472.594886 mapread 0x0163e2 thru 0x02241b (0xc03a bytes) 000712 1786748472.602456 mapread 0x02603d thru 0x0306a5 (0xa669 bytes) 000714 1786748472.614224 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe4e): Mirror 14: 0x9b3d3c3d (stale) Mirror 19: 0x9b3d3c3d (stale) Mirror 20: 0x9b3d3c3d (stale) Mirror 21: 0x9b3d3c3d (stale) Mirror 23: 0x9b3d3c3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe4e): Mirror 14: 0x9b3d3c3d (stale) Mirror 19: 0x9b3d3c3d (stale) Mirror 20: 0x9b3d3c3d (stale) Mirror 21: 0x9b3d3c3d (stale) Mirror 23: 0x9b3d3c3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe4e: skipped 000715 1786748473.039472 read 0x00d940 thru 0x018286 (0xa947 bytes) 000717 1786748473.050039 write 0x033c70 thru 0x036b26 (0x2eb7 bytes) 000718 1786748473.067464 read 0x001b00 thru 0x00871c (0x6c1d bytes) 000719 1786748473.083083 MIRROR_RESYNC 6 mirrors 000720 1786748475.470959 mapwrite 0x034b49 thru 0x03ab1a (0x5fd2 bytes) 000722 1786748475.685100 write 0x0319a7 thru 0x035e25 (0x447f bytes) 000723 1786748475.687849 mapread 0x03acc9 thru 0x03ae8d (0x1c5 bytes) 000724 1786748475.690888 mapwrite 0x029eb2 thru 0x03429e (0xa3ed bytes) 000725 1786748475.789641 read 0x010dd8 thru 0x019444 (0x866d bytes) 000727 1786748475.811981 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fe4e): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe4e): Mirror 14: 0x2f67cd18 (stale) Mirror 19: 0x2f67cd18 (stale) Mirror 20: 0x2f67cd18 (stale) Mirror 21: 0x2f67cd18 (stale) Mirror 23: 0x2f67cd18 (stale) CRC-32 checksum value for chunk [0, 0x3fe4e): Mirror 1: 0x8cb747c8 Mirror 24: 0x8cb747c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe4e: skipped 000728 1786748476.859336 mapwrite 0x02ce51 thru 0x031d28 (0x4ed8 bytes) 000730 1786748477.073557 trunc from 0x03fe4e to 0x02af4c (0x14f01 bytes) 000732 1786748477.109005 trunc from 0x02af4d to 0x019226 (0x11d26 bytes) 000734 1786748477.142049 mapwrite 0x036d9b thru 0x03bedb (0x5141 bytes) 000735 1786748477.277916 mapwrite 0x02890e thru 0x02c510 (0x3c03 bytes) 000737 1786748477.363116 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3bedc): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bedc): Mirror 14: 0x615072c7 (stale) Mirror 19: 0x615072c7 (stale) Mirror 20: 0x615072c7 (stale) Mirror 21: 0x615072c7 (stale) Mirror 23: 0x615072c7 (stale) Mirror 24: 0x7491cac2 (stale) CRC-32 checksum value for chunk [0, 0x3bedc): Mirror 1: 0x4355d82 Mirror 25: 0x4355d82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bedc: skipped 000738 1786748478.285497 mapread 0x021896 thru 0x0265be (0x4d29 bytes) 000740 1786748478.340715 trunc from 0x03bedc to 0x008418 (0x33ac3 bytes) 000744 1786748478.459720 write 0x004015 thru 0x006739 (0x2725 bytes) 000746 1786748478.470424 mapwrite 0x006eda thru 0x00a5d4 (0x36fb bytes) 000749 1786748478.577563 write 0x007664 thru 0x00db66 (0x6503 bytes) 000750 1786748478.585150 MIRROR_RESYNC 8 mirrors 000751 1786748479.867184 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000754 1786748480.400849 mapwrite 0x004b9e thru 0x00bfac (0x740f bytes) 000755 1786748480.733321 read 0x003796 thru 0x007fb8 (0x4823 bytes) 000756 1786748480.740341 mapread 0x00c8de thru 0x00db66 (0x1289 bytes) 000757 1786748480.743925 mapread 0x001d2d thru 0x00c65d (0xa931 bytes) 000759 1786748480.749281 read 0x00bcd3 thru 0x00db66 (0x1e94 bytes) 000761 1786748480.752790 write 0x02663c thru 0x02f8d7 (0x929c bytes) 000762 1786748480.760974 mapread 0x021a14 thru 0x02ecad (0xd29a bytes) 000764 1786748480.800508 mapread 0x02a9bf thru 0x02f8d7 (0x4f19 bytes) 000766 1786748480.803791 write 0x01f578 thru 0x021630 (0x20b9 bytes) 000768 1786748480.812869 trunc from 0x02f8d8 to 0x01af5a (0x1497d bytes) 000769 1786748480.851728 mapwrite 0x03c8cf thru 0x03ffff (0x3731 bytes) truncating to largest ever: 0x3fc85 000770 1786748481.054968 trunc from 0x040000 to 0x03fc84 (0x37b bytes) 000771 1786748481.103031 mapread 0x021a62 thru 0x0301a9 (0xe748 bytes) 000773 1786748481.265426 write 0x03aad6 thru 0x03ffff (0x552a bytes) 000774 1786748481.270750 mapwrite 0x016bda thru 0x0202ed (0x9714 bytes) 000778 1786748481.392013 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9b13786f (stale) Mirror 21: 0x9b13786f (stale) Mirror 23: 0x9b13786f (stale) Mirror 24: 0x9b13786f (stale) Mirror 25: 0x9b13786f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9b13786f (stale) Mirror 21: 0x9b13786f (stale) Mirror 23: 0x9b13786f (stale) Mirror 24: 0x9b13786f (stale) Mirror 25: 0x9b13786f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000779 1786748481.886761 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9b13786f (stale) Mirror 21: 0x9b13786f (stale) Mirror 23: 0x9b13786f (stale) Mirror 24: 0x9b13786f (stale) Mirror 25: 0x9b13786f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7dfa2a7b Mirror 26: 0x7dfa2a7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1786748483.025417 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9b13786f (stale) Mirror 21: 0x9b13786f (stale) Mirror 23: 0x9b13786f (stale) Mirror 24: 0x9b13786f (stale) Mirror 25: 0x9b13786f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7dfa2a7b Mirror 26: 0x7dfa2a7b Mirror 27: 0x7dfa2a7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1786748483.925001 mapwrite 0x012db4 thru 0x013837 (0xa84 bytes) 000782 1786748484.132901 trunc from 0x040000 to 0x0216fb (0x1e904 bytes) 000783 1786748484.170446 mapwrite 0x03ce7a thru 0x03fdeb (0x2f72 bytes) 000786 1786748484.265213 MIRROR_RESYNC 8 mirrors 000787 1786748486.215946 MIRROR_EXTEND to 9 mirrors 000790 1786748487.214397 mapwrite 0x020077 thru 0x0296d9 (0x9663 bytes) 000791 1786748487.418785 trunc from 0x03fdec to 0x02f978 (0x10473 bytes) 000792 1786748487.443954 mapwrite 0x03f2b8 thru 0x03ffff (0xd48 bytes) 000793 1786748487.511841 trunc from 0x040000 to 0x02970e (0x168f1 bytes) 000795 1786748487.527580 mapread 0x01ea07 thru 0x021353 (0x294d bytes) 000798 1786748487.536133 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2970f): Mirror 14: 0xc79e5eaa (stale) Mirror 21: 0xc79e5eaa (stale) Mirror 23: 0xc79e5eaa (stale) Mirror 24: 0xc79e5eaa (stale) Mirror 25: 0xc79e5eaa (stale) Mirror 27: 0xc79e5eaa (stale) Mirror 28: 0xc79e5eaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2970f): Mirror 14: 0xc79e5eaa (stale) Mirror 21: 0xc79e5eaa (stale) Mirror 23: 0xc79e5eaa (stale) Mirror 24: 0xc79e5eaa (stale) Mirror 25: 0xc79e5eaa (stale) Mirror 27: 0xc79e5eaa (stale) Mirror 28: 0xc79e5eaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2970f: skipped 000799 1786748488.079090 read 0x01fe39 thru 0x023c91 (0x3e59 bytes) 000801 1786748488.093456 trunc from 0x02970f to 0x002e0c (0x26902 bytes) 000803 1786748488.110982 mapwrite 0x0332c6 thru 0x03a379 (0x70b4 bytes) 000804 1786748488.188033 mapwrite 0x02b8d6 thru 0x02f191 (0x38bc bytes) 000805 1786748488.249552 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a37a): Mirror 21: 0xec6991f7 (stale) Mirror 23: 0xec6991f7 (stale) Mirror 24: 0xec6991f7 (stale) Mirror 25: 0xec6991f7 (stale) Mirror 27: 0xec6991f7 (stale) Mirror 28: 0xec6991f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37a): Mirror 21: 0xec6991f7 (stale) Mirror 23: 0xec6991f7 (stale) Mirror 24: 0xec6991f7 (stale) Mirror 25: 0xec6991f7 (stale) Mirror 27: 0xec6991f7 (stale) Mirror 28: 0xec6991f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37a: skipped 000808 1786748488.698500 trunc from 0x03a37a to 0x039de7 (0x592 bytes) 000811 1786748488.735306 mapwrite 0x011006 thru 0x01bf44 (0xaf3f bytes) 000812 1786748488.807405 read 0x01a8f4 thru 0x021ba2 (0x72af bytes) 000813 1786748488.816149 read 0x028238 thru 0x0340a4 (0xbe6d bytes) 000817 1786748488.828879 trunc from 0x039de8 to 0x014ef2 (0x24ef5 bytes) 000819 1786748488.850866 mapread 0x0023a2 thru 0x0115e4 (0xf243 bytes) 000820 1786748488.852731 mapwrite 0x036ef5 thru 0x03d271 (0x637d bytes) 000821 1786748488.939043 trunc from 0x03d272 to 0x00da56 (0x2f81b bytes) 000822 1786748488.957985 read 0x00016b thru 0x0057d8 (0x566e bytes) 000824 1786748488.960374 mapread 0x007ca7 thru 0x00da56 (0x5db0 bytes) 000825 1786748488.962302 trunc from 0x00da57 to 0x00bb1f (0x1f37 bytes) 000827 1786748488.996173 trunc from 0x00bb20 to 0x01f162 (0x13643 bytes) 000828 1786748489.023994 mapread 0x01b89c thru 0x01e2cf (0x2a34 bytes) 000829 1786748489.033829 mapread 0x009a85 thru 0x00cd64 (0x32e0 bytes) 000830 1786748489.036936 mapread 0x00df5e thru 0x00e1a2 (0x245 bytes) 000833 1786748489.043672 mapread 0x006eaa thru 0x00a865 (0x39bc bytes) 000835 1786748489.045786 trunc from 0x01f163 to 0x01cb53 (0x260f bytes) 000836 1786748489.075633 mapwrite 0x014c77 thru 0x01c548 (0x78d2 bytes) 000838 1786748489.162002 mapread 0x01597f thru 0x01cb53 (0x71d5 bytes) 000839 1786748489.163867 mapread 0x011dac thru 0x012be6 (0xe3b bytes) 000842 1786748489.169167 mapwrite 0x02e0ec thru 0x02eb96 (0xaab bytes) 000843 1786748489.253739 mapwrite 0x00fbdf thru 0x012eba (0x32dc bytes) 000846 1786748489.341010 read 0x0126ec thru 0x01f763 (0xd078 bytes) 000848 1786748489.361336 MIRROR_RESYNC 7 mirrors 000849 1786748491.245924 mapwrite 0x0302d0 thru 0x03ed78 (0xeaa9 bytes) 000851 1786748491.507838 trunc from 0x03ed79 to 0x01f8e8 (0x1f490 bytes) 000854 1786748491.542447 trunc from 0x01f8e9 to 0x0125f6 (0xd2f2 bytes) 000855 1786748491.569412 read 0x0097be thru 0x00bb59 (0x239c bytes) 000860 1786748491.591574 write 0x01920c thru 0x01ea15 (0x580a bytes) 000861 1786748491.608431 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1ea16): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea16): Mirror 21: 0x1833bdbe (stale) Mirror 23: 0x1833bdbe (stale) Mirror 24: 0x1833bdbe (stale) Mirror 25: 0x1833bdbe (stale) Mirror 27: 0x1833bdbe (stale) Mirror 28: 0x1833bdbe (stale) CRC-32 checksum value for chunk [0, 0x1ea16): Mirror 1: 0xb86da4f1 Mirror 29: 0xb86da4f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea16: skipped 000863 1786748492.761958 write 0x010799 thru 0x01fca6 (0xf50e bytes) 000864 1786748492.840918 trunc from 0x01fca7 to 0x03239d (0x126f7 bytes) 000867 1786748492.869221 mapread 0x01f81a thru 0x0238ef (0x40d6 bytes) 000868 1786748492.891978 write 0x03f29d thru 0x03ffff (0xd63 bytes) 000872 1786748492.899885 MIRROR_RESYNC 8 mirrors 000873 1786748495.147644 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000875 1786748495.613841 mapwrite 0x02afe7 thru 0x031680 (0x669a bytes) 000876 1786748495.929431 read 0x02755a thru 0x0312c3 (0x9d6a bytes) 000877 1786748495.930876 read 0x023444 thru 0x031051 (0xdc0e bytes) 000879 1786748495.953033 MIRROR_RESYNC 7 mirrors 000880 1786748498.471434 mapread 0x01772b thru 0x02622b (0xeb01 bytes) 000881 1786748498.515117 trunc from 0x040000 to 0x0150de (0x2af21 bytes) 000882 1786748498.624669 read 0x0115c3 thru 0x0150de (0x3b1c bytes) 000883 1786748498.638999 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x150df): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x150df): Mirror 21: 0xd7bb3dfd (stale) Mirror 23: 0xd7bb3dfd (stale) Mirror 24: 0xd7bb3dfd (stale) Mirror 27: 0xd7bb3dfd (stale) Mirror 28: 0xd7bb3dfd (stale) Mirror 29: 0xd7bb3dfd (stale) CRC-32 checksum value for chunk [0, 0x150df): Mirror 1: 0xd7bb3dfd Mirror 30: 0xd7bb3dfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x150df: skipped 000887 1786748500.000426 mapread 0x009886 thru 0x00cbff (0x337a bytes) 000888 1786748500.047362 trunc from 0x0150df to 0x038445 (0x23367 bytes) 000889 1786748500.162767 mapwrite 0x0272c3 thru 0x02ce9b (0x5bd9 bytes) 000890 1786748500.231635 MIRROR_RESYNC 8 mirrors 000891 1786748503.112639 MIRROR_RESYNC 8 mirrors 000892 1786748503.791535 mapwrite 0x022be6 thru 0x02e5c5 (0xb9e0 bytes) 000895 1786748503.972392 mapwrite 0x00602d thru 0x011627 (0xb5fb bytes) 000896 1786748504.048237 mapwrite 0x02d4a7 thru 0x035588 (0x80e2 bytes) 000898 1786748504.112632 write 0x000522 thru 0x00f200 (0xecdf bytes) 000899 1786748504.119696 read 0x01d4d8 thru 0x0209b2 (0x34db bytes) 000905 1786748504.130061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38446): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38446): Mirror 21: 0x9196ef92 (stale) Mirror 23: 0x9196ef92 (stale) Mirror 24: 0x9196ef92 (stale) Mirror 27: 0x9196ef92 (stale) Mirror 28: 0x9196ef92 (stale) Mirror 29: 0x9196ef92 (stale) Mirror 30: 0x9196ef92 (stale) CRC-32 checksum value for chunk [0, 0x38446): Mirror 1: 0x15581c8 Mirror 31: 0x15581c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38446: skipped 000906 1786748505.282808 read 0x018375 thru 0x01e985 (0x6611 bytes) 000907 1786748505.300202 mapread 0x01f159 thru 0x024632 (0x54da bytes) 000909 1786748505.349354 mapread 0x02ddc3 thru 0x0378d5 (0x9b13 bytes) 000910 1786748505.391295 mapwrite 0x00ef97 thru 0x01ed83 (0xfded bytes) 000911 1786748505.708824 mapread 0x007db7 thru 0x00fd22 (0x7f6c bytes) 000913 1786748505.718374 MIRROR_RESYNC 9 mirrors 000914 1786748507.875204 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000915 1786748508.542010 mapread 0x02a137 thru 0x037dae (0xdc78 bytes) 000916 1786748508.619479 MIRROR_RESYNC 8 mirrors 000917 1786748509.577097 mapwrite 0x03d91a thru 0x03ffff (0x26e6 bytes) 000920 1786748509.758756 read 0x010104 thru 0x011625 (0x1522 bytes) 000921 1786748509.764917 trunc from 0x040000 to 0x026850 (0x197af bytes) 000922 1786748509.795998 read 0x00d440 thru 0x013642 (0x6203 bytes) 000923 1786748509.808321 write 0x03d7c2 thru 0x03ffff (0x283e bytes) 000926 1786748509.812157 mapwrite 0x032231 thru 0x03ffff (0xddcf bytes) 000929 1786748509.912574 write 0x003ec6 thru 0x00f276 (0xb3b1 bytes) 000930 1786748509.922894 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb186c7b5 (stale) Mirror 24: 0xb186c7b5 (stale) Mirror 27: 0xb186c7b5 (stale) Mirror 28: 0xb186c7b5 (stale) Mirror 29: 0xb186c7b5 (stale) Mirror 30: 0xb186c7b5 (stale) Mirror 31: 0xb186c7b5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaddefb95 Mirror 32: 0xaddefb95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000931 1786748511.677253 trunc from 0x040000 to 0x0372e7 (0x8d18 bytes) 000933 1786748511.800964 MIRROR_RESYNC 9 mirrors 000934 1786748514.452954 MIRROR_EXTEND to 10 mirrors 000936 1786748515.608571 mapwrite 0x006c98 thru 0x00abce (0x3f37 bytes) 000937 1786748515.934778 mapwrite 0x021a40 thru 0x02b2a1 (0x9862 bytes) 000939 1786748516.058295 trunc from 0x0372e8 to 0x00273f (0x34ba8 bytes) 000940 1786748516.096824 trunc from 0x002740 to 0x03f90f (0x3d1d0 bytes) 000941 1786748516.121779 mapread 0x03b0d8 thru 0x03f90f (0x4838 bytes) 000943 1786748516.144732 mapwrite 0x021696 thru 0x027b17 (0x6482 bytes) 000944 1786748516.307046 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f910): Mirror 23: 0x2135e389 (stale) Mirror 27: 0x2135e389 (stale) Mirror 28: 0x2135e389 (stale) Mirror 29: 0x2135e389 (stale) Mirror 30: 0x2135e389 (stale) Mirror 31: 0x2135e389 (stale) Mirror 32: 0x2135e389 (stale) Mirror 33: 0x2135e389 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f910): Mirror 23: 0x2135e389 (stale) Mirror 27: 0x2135e389 (stale) Mirror 28: 0x2135e389 (stale) Mirror 29: 0x2135e389 (stale) Mirror 30: 0x2135e389 (stale) Mirror 31: 0x2135e389 (stale) Mirror 32: 0x2135e389 (stale) Mirror 33: 0x2135e389 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f910: skipped 000945 1786748517.286534 trunc from 0x03f910 to 0x02e332 (0x115dd bytes) 000947 1786748517.339369 read 0x014c57 thru 0x01a420 (0x57ca bytes) 000949 1786748517.358342 trunc from 0x02e333 to 0x02918d (0x51a5 bytes) 000950 1786748517.401729 mapread 0x02113e thru 0x024cdc (0x3b9f bytes) 000951 1786748517.424131 mapwrite 0x03ef54 thru 0x03ffff (0x10ac bytes) 000952 1786748517.510971 trunc from 0x040000 to 0x01b78f (0x24870 bytes) 000953 1786748517.540580 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b790): Mirror 23: 0x110ca909 (stale) Mirror 27: 0x110ca909 (stale) Mirror 28: 0x110ca909 (stale) Mirror 29: 0x110ca909 (stale) Mirror 30: 0x110ca909 (stale) Mirror 31: 0x110ca909 (stale) Mirror 32: 0x110ca909 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b790): Mirror 23: 0x110ca909 (stale) Mirror 27: 0x110ca909 (stale) Mirror 28: 0x110ca909 (stale) Mirror 29: 0x110ca909 (stale) Mirror 30: 0x110ca909 (stale) Mirror 31: 0x110ca909 (stale) Mirror 32: 0x110ca909 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b790: skipped 000954 1786748518.007877 trunc from 0x01b790 to 0x02ec44 (0x134b5 bytes) 000955 1786748518.021516 mapread 0x01a80f thru 0x028757 (0xdf49 bytes) 000957 1786748518.053746 trunc from 0x02ec45 to 0x028604 (0x6640 bytes) 000958 1786748518.080382 MIRROR_RESYNC 8 mirrors 000960 1786748520.166206 trunc from 0x028605 to 0x01c567 (0xc09d bytes) 000961 1786748520.300100 write 0x03bf5d thru 0x03ffff (0x40a3 bytes) 000962 1786748520.312153 MIRROR_RESYNC 8 mirrors 000964 1786748522.725036 trunc from 0x040000 to 0x02448c (0x1bb73 bytes) 000965 1786748522.868466 write 0x026c23 thru 0x02b531 (0x490f bytes) 000968 1786748522.880129 mapwrite 0x0027ab thru 0x00953d (0x6d93 bytes) 000969 1786748523.048778 write 0x02fb81 thru 0x03dac7 (0xdf47 bytes) 000971 1786748523.054963 mapwrite 0x00f5ac thru 0x016487 (0x6edc bytes) 000975 1786748523.214308 read 0x032e1d thru 0x037a0a (0x4bee bytes) 000977 1786748523.216675 write 0x01b5d8 thru 0x02354a (0x7f73 bytes) 000978 1786748523.239582 trunc from 0x03dac8 to 0x00406a (0x39a5d bytes) 000979 1786748523.276854 write 0x0180c3 thru 0x0194ac (0x13ea bytes) 000980 1786748523.283522 write 0x0309a5 thru 0x0395b9 (0x8c15 bytes) 000981 1786748523.287852 mapread 0x02e9d1 thru 0x035d5c (0x738c bytes) 000982 1786748523.310106 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x395ba): Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395ba): Mirror 23: 0x9dcd465c (stale) Mirror 27: 0x9dcd465c (stale) Mirror 28: 0x9dcd465c (stale) Mirror 29: 0x9dcd465c (stale) Mirror 30: 0x9dcd465c (stale) Mirror 31: 0x9dcd465c (stale) Mirror 32: 0x9dcd465c (stale) CRC-32 checksum value for chunk [0, 0x395ba): Mirror 1: 0x6d45c2be Mirror 33: 0x6d45c2be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395ba: skipped 000983 1786748524.376428 trunc from 0x0395ba to 0x03abe9 (0x1630 bytes) 000984 1786748524.505184 read 0x0271ff thru 0x0288bc (0x16be bytes) 000986 1786748524.514112 mapread 0x020bdd thru 0x0240f7 (0x351b bytes) 000987 1786748524.535232 mapwrite 0x015857 thru 0x019906 (0x40b0 bytes) 000992 1786748524.588774 mapread 0x008b60 thru 0x00f0e9 (0x658a bytes) 000994 1786748524.600993 read 0x001aad thru 0x010588 (0xeadc bytes) 000995 1786748524.605981 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3abea): Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abea): Mirror 23: 0xac386d39 (stale) Mirror 27: 0xac386d39 (stale) Mirror 28: 0xac386d39 (stale) Mirror 29: 0xac386d39 (stale) Mirror 30: 0xac386d39 (stale) Mirror 31: 0xac386d39 (stale) Mirror 32: 0xac386d39 (stale) Mirror 33: 0xbf5b6306 (stale) CRC-32 checksum value for chunk [0, 0x3abea): Mirror 1: 0x52ef937b Mirror 34: 0x52ef937b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abea: skipped 000997 1786748526.029085 trunc from 0x03abea to 0x03043f (0xa7aa bytes) 000998 1786748526.217044 mapwrite 0x021f02 thru 0x02a095 (0x8194 bytes) 001000 1786748526.335716 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30440): Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30440): Mirror 23: 0xa1a4e829 (stale) Mirror 27: 0xa1a4e829 (stale) Mirror 28: 0xa1a4e829 (stale) Mirror 29: 0xa1a4e829 (stale) Mirror 30: 0xa1a4e829 (stale) Mirror 31: 0xa1a4e829 (stale) Mirror 32: 0xa1a4e829 (stale) Mirror 33: 0x11a50c04 (stale) Mirror 34: 0xdffe1984 (stale) CRC-32 checksum value for chunk [0, 0x30440): Mirror 1: 0x181af2e4 Mirror 35: 0x181af2e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30440: skipped All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 19:02:14 (1786748534) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 19:02:18 (1786748538) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324604 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206293 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 19:02:28 (1786748548) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8226592 .. Extending file size to 3227968 .. done Extending file size to 7626400 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8942624 .. Extending file size to 9107008 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57472 57473 57474 57476 57478 done 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) 192.168.204.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:03:51 (1786748631) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4081248 .. Extending file size to 5625792 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7161216 .. Extending file size to 7477280 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7027072 .. Extending file size to 7535008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9043072 .. Extending file size to 8252000 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9171136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59069 59070 59072 59074 59076 done 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) 192.168.204.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:05:09 (1786748709) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139738 s, 15.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.392185 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc676877c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:05:28 (1786748728) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.081406 s, 25.8 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 19:05:37 (1786748737) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:05:46 (1786748746) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199526 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:05:54 (1786748754) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162454 s, 12.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:06:03 (1786748763) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176986 s, 11.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:06:38 (1786748798) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558585 s, 18.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:07:39 (1786748859) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.399584 s, 5.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:08:41 (1786748921) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. Creating new pool test_204f oleg426-server: Pool lustre.test_204f created Adding targets to pool oleg426-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg426-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245008 s, 8.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg426-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg426-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg426-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:09:58 (1786748998) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:10:06 (1786749006) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:10:15 (1786749015) oleg426-server: Pool lustre.test_206 created Adding targets to pool oleg426-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg426-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg426-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg426-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg426-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:10:40 (1786749040) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667334 s, 15.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040211 s, 26.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2ad11b7a Mirror 3: 0x2ad11b7a PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:10:52 (1786749052) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:10:55 (1786749055) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:10:59 (1786749059) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.118548 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1440000 Read this (4 bytes): BBBB 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:11:39 (1786749099) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0500251 s, 164 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:11:47 (1786749107) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385388 s, 27.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834467 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 19:11:49.000000000 -0400 Modify: 2026-08-14 19:11:49.000000000 -0400 Change: 2026-08-14 19:11:50.000000000 -0400 Birth: 2026-08-14 19:11:48.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:11:55 (1786749115) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:11:58 (1786749118) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.499055 s, 82.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.112462 s, 36.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 19:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x225:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 19:12 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:12:06 (1786749126) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:12:13 (1786749133) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1813 sec =========== 19:12:21 (1786749141) === sanity-flr: start cleanup 19:12:23 (1786749143) === === sanity-flr: finish cleanup 19:12:31 (1786749151) ===