-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 11:03:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:03:24 (1786719804) === oleg420-client.virtnet: executing check_config_client /mnt/lustre oleg420-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg420-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff900009436000.idle_timeout=debug osc.lustre-OST0001-osc-ffff900009436000.idle_timeout=debug disable quota as required oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:04:04 (1786719844) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7637] == sanity-flr test 0a: lfs mirror create with -N option == 11:04:07 (1786719847) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:04:27 (1786719867) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:04:31 (1786719871) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:04:35 (1786719875) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:04:59 (1786719899) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:05:02 (1786719902) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 11:05:05 (1786719905) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff900009436000.stats=clear osc.lustre-OST0001-osc-ffff900009436000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:05:13 (1786719913) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:05:24 (1786719924) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:05:34 (1786719934) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:05:38 (1786719938) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:05:48 (1786719948) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (14s) == sanity-flr test 2: create components from existing files ========================================================== 11:06:02 (1786719962) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:06:11 (1786719971) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:06:14 (1786719974) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:06:22 (1786719982) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:06:34 (1786719994) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:06:42 (1786720002) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.88138 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168456 s, 6.2 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:06:54 (1786720014) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.304678 s, 37.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:07:03 (1786720023) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:07:14 (1786720034) Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:08:31 (1786720111) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.561232 s, 13.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:09:14 (1786720154) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:10:40 (1786720240) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.27047 s, 8.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 11:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.463s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.034s user 0m0.000s sys 0m0.196s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.262s user 0m0.001s sys 0m0.204s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900009436000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:12:25 (1786720345) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:12:30 (1786720350) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:12:33 (1786720353) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 11:12:36 (1786720356) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 11:12:44 (1786720364) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.636552 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:12:58 (1786720378) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344759 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235832 s, 44.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:13:05 (1786720385) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:13:13 (1786720393) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107306 s, 954 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00416084 s, 246 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126766 s, 808 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00302111 s, 339 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 11:13:27 (1786720407) create test files with size 15754845 .. e9c089e265a2a4a487ac4814976b4d72 - cef6cac77759c2b1fe71bcfc8027758f - 44d72b62cda5d8bf0f641700ed534d4f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 11:13:58 (1786720438) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:14:18 (1786720458) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:14:24 (1786720464) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157731 s, 6.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382132 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 11:14:34 (1786720474) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217652 s, 9.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.28519 s, 14.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204284 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 11:14:52 (1786720492) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 11:14:56 (1786720496) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:14:59 (1786720499) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359648 s, 29.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:15:07 (1786720507) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:15:11 (1786720511) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:15:21 (1786720521) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.335779 s, 31.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:15:32 (1786720532) llite.lustre-ffff900002ea9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300536 s, 34.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9000109d7800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:15:44 (1786720544) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:16:17 (1786720577) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:16:50 (1786720610) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 11:16:53 (1786720613) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.253472 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199458 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1443aad5 (nosync,stale) Mirror 2: 0x1443aad5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 54e8131d4926a4d2765c26cf08dc8e26 - mirror 1 checksum: f320942643eec8f9b46a398e83a668d6 - mirror 2 checksum: 54e8131d4926a4d2765c26cf08dc8e26 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T11:16:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T11:16:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 54e8131d4926a4d2765c26cf08dc8e26 - mirror 1 checksum: f320942643eec8f9b46a398e83a668d6 - mirror 2 checksum: f320942643eec8f9b46a398e83a668d6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T11:16:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T11:16:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:17:06 (1786720626) Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 11:17:14 (1786720634) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server Failover mds1 to oleg420-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:46 (1786720666) targets are mounted 11:17:46 (1786720666) facet_failover done oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275676 s, 38.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:18:09 (1786720689) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:18:13 (1786720693) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:18:17 (1786720697) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0068826 s, 2.4 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:18:26 (1786720706) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:18:30 (1786720710) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0207953 s, 197 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0685469 s, 59.8 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:18:39 (1786720719) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:19:06 (1786720746) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:19:39 (1786720779) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:20:05 (1786720805) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:20:32 (1786720832) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 11:20:42 (1786720842) mirror create pid 54336 mirror split pid 54338 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54336 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54338 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 11:21:50 (1786720910) Seed set to 912 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786720913.168281 MIRROR_EXTEND to 2 mirrors skipping zero size read truncating to largest ever: 0x129e9 000004 1786720913.772346 trunc from 00000000 to 0x0129e8 (0x129e9 bytes) 000007 1786720913.917383 read 0x0073ba thru 0x00e154 (0x6d9b bytes) 000009 1786720913.937257 mapread 0x00901a thru 0x010b85 (0x7b6c bytes) 000010 1786720913.960580 mapread 0x012178 thru 0x0129e8 (0x871 bytes) 000011 1786720913.977200 mapwrite 0x008b6b thru 0x01364d (0xaae3 bytes) 000012 1786720914.144507 read 0x00bdba thru 0x00ca49 (0xc90 bytes) 000013 1786720914.146359 mapwrite 0x00b4fe thru 0x01b4e7 (0xffea bytes) 000014 1786720914.273345 write 0x01ad8e thru 0x01c8a1 (0x1b14 bytes) 000016 1786720914.278772 mapwrite 0x018f5d thru 0x027d56 (0xedfa bytes) 000018 1786720914.402342 write 0x015a33 thru 0x0205af (0xab7d bytes) 000019 1786720914.409430 write 0x006f82 thru 0x0146bf (0xd73e bytes) 000021 1786720914.418633 read 0x0107ff thru 0x019614 (0x8e16 bytes) 000022 1786720914.420891 read 0x00f32f thru 0x01b025 (0xbcf7 bytes) 000023 1786720914.424962 mapwrite 0x003b72 thru 0x00a4d1 (0x6960 bytes) 000025 1786720914.541811 read 0x001e14 thru 0x0095ab (0x7798 bytes) 000027 1786720914.569122 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27d57): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27d57): Mirror 2: 0x5149c1e9 (stale) CRC-32 checksum value for chunk [0, 0x27d57): Mirror 1: 0x68c8f2d2 Mirror 3: 0x68c8f2d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27d57: skipped 000028 1786720916.023458 MIRROR_RESYNC 3 mirrors 000031 1786720917.254574 MIRROR_EXTEND to 4 mirrors 000033 1786720918.206860 read 0x0254a1 thru 0x027d56 (0x28b6 bytes) 000035 1786720918.213886 read 0x008519 thru 0x00a930 (0x2418 bytes) 000036 1786720918.224892 mapwrite 0x000429 thru 0x00b44f (0xb027 bytes) 000038 1786720918.577306 MIRROR_RESYNC 4 mirrors 000039 1786720919.948856 MIRROR_EXTEND to 5 mirrors 000040 1786720921.249472 write 0x027017 thru 0x035df8 (0xede2 bytes) 000042 1786720921.446335 write 0x028371 thru 0x031e55 (0x9ae5 bytes) 000043 1786720921.455775 mapread 0x033495 thru 0x035df8 (0x2964 bytes) truncating to largest ever: 0x3e27f 000044 1786720921.457606 trunc from 0x035df9 to 0x03e27e (0x8486 bytes) 000045 1786720921.480982 mapread 0x024054 thru 0x029665 (0x5612 bytes) 000046 1786720921.514202 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e27f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e27f): Mirror 2: 0xc4df1017 (stale) Mirror 3: 0xc4df1017 (stale) Mirror 4: 0xc4df1017 (stale) Mirror 5: 0xc4df1017 (stale) CRC-32 checksum value for chunk [0, 0x3e27f): Mirror 1: 0x10b525ba Mirror 6: 0x10b525ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e27f: skipped 000049 1786720923.128838 mapwrite 0x002abb thru 0x010922 (0xde68 bytes) 000052 1786720923.464498 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e27f): Mirror 3: 0xc4df1017 (stale) Mirror 4: 0xc4df1017 (stale) Mirror 5: 0xc4df1017 (stale) Mirror 6: 0x10b525ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e27f): Mirror 3: 0xc4df1017 (stale) Mirror 4: 0xc4df1017 (stale) Mirror 5: 0xc4df1017 (stale) Mirror 6: 0x10b525ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e27f: skipped 000054 1786720924.215236 mapwrite 0x02bdbb thru 0x032de7 (0x702d bytes) 000055 1786720924.321879 write 0x0161d0 thru 0x0196d4 (0x3505 bytes) 000057 1786720924.349164 mapread 0x03254a thru 0x03bf89 (0x9a40 bytes) 000058 1786720924.378768 mapwrite 0x026746 thru 0x0361cb (0xfa86 bytes) 000063 1786720924.507031 write 0x02dab4 thru 0x0359ab (0x7ef8 bytes) 000065 1786720924.510536 trunc from 0x03e27f to 0x01c1e0 (0x2209e bytes) 000066 1786720924.548056 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c1e1): Mirror 4: 0x60558fbf (stale) Mirror 5: 0x60558fbf (stale) Mirror 6: 0x60558fbf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1e1): Mirror 4: 0x60558fbf (stale) Mirror 5: 0x60558fbf (stale) Mirror 6: 0x60558fbf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1e1: skipped 000068 1786720925.204143 mapread 0x006aa7 thru 0x00c1a4 (0x56fe bytes) 000070 1786720925.230651 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c1e1): Mirror 5: 0x60558fbf (stale) Mirror 6: 0x60558fbf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1e1): Mirror 5: 0x60558fbf (stale) Mirror 6: 0x60558fbf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1e1: skipped 000071 1786720925.773668 trunc from 0x01c1e1 to 0x02c66c (0x1048c bytes) 000074 1786720925.797036 write 0x0157b5 thru 0x01d949 (0x8195 bytes) 000075 1786720925.816746 mapwrite 0x01667e thru 0x01ebed (0x8570 bytes) 000076 1786720925.888715 mapread 0x014dae thru 0x024271 (0xf4c4 bytes) 000077 1786720925.924523 trunc from 0x02c66d to 0x026548 (0x6124 bytes) 000080 1786720925.957048 mapread 0x013f07 thru 0x01f43c (0xb536 bytes) 000081 1786720925.977622 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26549): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26549): Mirror 5: 0x8c4f33f6 (stale) Mirror 6: 0x8c4f33f6 (stale) CRC-32 checksum value for chunk [0, 0x26549): Mirror 1: 0x17b2aab9 Mirror 7: 0x17b2aab9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26549: skipped 000082 1786720927.361975 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26549): Mirror 6: 0x8c4f33f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26549): Mirror 6: 0x8c4f33f6 (stale) CRC-32 checksum value for chunk [0, 0x26549): Mirror 1: 0x17b2aab9 Mirror 7: 0x17b2aab9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26549: skipped 000085 1786720928.343919 trunc from 0x026549 to 0x016f18 (0xf630 bytes) 000086 1786720928.477216 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16f19): Mirror 6: 0xc82c59b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f19): Mirror 6: 0xc82c59b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f19: skipped 000087 1786720928.972445 write 0x0316be thru 0x03a933 (0x9276 bytes) 000088 1786720928.975393 trunc from 0x03a934 to 0x038a3f (0x1ef4 bytes) 000089 1786720928.999212 trunc from 0x038a40 to 0x037baa (0xe95 bytes) 000090 1786720929.036505 read 0x00a27c thru 0x0192f3 (0xf078 bytes) 000091 1786720929.058918 trunc from 0x037bab to 0x013c9d (0x23f0d bytes) 000092 1786720929.086892 write 0x0267eb thru 0x029b71 (0x3387 bytes) 000094 1786720929.106748 MIRROR_RESYNC 2 mirrors 000095 1786720930.604373 mapwrite 0x03de31 thru 0x03e8d6 (0xaa6 bytes) 000097 1786720930.798459 trunc from 0x03e8d7 to 0x013f3d (0x2a999 bytes) 000103 1786720930.836499 write 0x02cacb thru 0x031f22 (0x5458 bytes) 000105 1786720930.842650 read 0x02ef34 thru 0x031f22 (0x2fef bytes) 000106 1786720930.846950 MIRROR_RESYNC 2 mirrors 000107 1786720932.397000 mapwrite 0x0358da thru 0x03c12b (0x6852 bytes) 000110 1786720932.693751 read 0x007683 thru 0x00856a (0xee8 bytes) 000111 1786720932.701244 mapwrite 0x006fb4 thru 0x00ba64 (0x4ab1 bytes) 000114 1786720932.793679 mapwrite 0x002a4b thru 0x00663c (0x3bf2 bytes) 000115 1786720932.884273 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c12c): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c12c): Mirror 6: 0x4bc2c271 (stale) CRC-32 checksum value for chunk [0, 0x3c12c): Mirror 1: 0x6870ca0c Mirror 7: 0x6870ca0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c12c: skipped 000117 1786720934.528046 read 0x03b1ab thru 0x03c12b (0xf81 bytes) 000118 1786720934.545438 mapwrite 0x0078e3 thru 0x012fe0 (0xb6fe bytes) 000119 1786720934.829867 trunc from 0x03c12c to 0x0001b1 (0x3bf7a bytes) 000120 1786720934.860554 trunc from 0x0001b2 to 0x013c06 (0x13a55 bytes) 000121 1786720934.890123 write 0x022d6b thru 0x02773d (0x49d3 bytes) 000122 1786720934.891553 mapread 0x01131c thru 0x01f0ad (0xdd92 bytes) 000124 1786720934.917065 mapwrite 0x032207 thru 0x035c29 (0x3a23 bytes) 000125 1786720935.054434 trunc from 0x035c2a to 0x03c188 (0x655f bytes) 000126 1786720935.094053 trunc from 0x03c189 to 0x014cbe (0x274ca bytes) 000128 1786720935.145835 mapwrite 0x01a8e6 thru 0x027feb (0xd706 bytes) 000129 1786720935.322474 mapread 0x018cba thru 0x019e5f (0x11a6 bytes) 000130 1786720935.326131 read 0x021659 thru 0x0243cd (0x2d75 bytes) 000131 1786720935.342657 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27fec): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27fec): Mirror 6: 0x3bc68cb3 (stale) Mirror 7: 0x65d44243 (stale) CRC-32 checksum value for chunk [0, 0x27fec): Mirror 1: 0x400d4897 Mirror 8: 0x400d4897 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27fec: skipped 000132 1786720936.395611 trunc from 0x027fec to 0x00c946 (0x1b6a5 bytes) 000133 1786720936.596210 read 0x00a551 thru 0x00c946 (0x23f6 bytes) 000135 1786720936.604369 read 0x00024d thru 0x00330b (0x30bf bytes) 000136 1786720936.620006 trunc from 0x00c947 to 0x02b6ab (0x1ed65 bytes) 000138 1786720936.656872 read 0x027fb4 thru 0x02b6ab (0x36f8 bytes) 000139 1786720936.670724 trunc from 0x02b6ac to 0x01e40a (0xd2a1 bytes) 000140 1786720936.693899 trunc from 0x01e40b to 0x01913e (0x52cc bytes) 000141 1786720936.728892 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1913f): Mirror 6: 0xcbb8e478 (stale) Mirror 8: 0x9b0059da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1913f): Mirror 6: 0xcbb8e478 (stale) Mirror 8: 0x9b0059da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1913f: skipped 000142 1786720937.128148 MIRROR_RESYNC 3 mirrors 000143 1786720938.200237 trunc from 0x01913f to 0x03d5f8 (0x244ba bytes) 000145 1786720938.287017 write 0x02e8cb thru 0x0326a6 (0x3ddc bytes) 000146 1786720938.299152 trunc from 0x03d5f9 to 0x011c3e (0x2b9ba bytes) 000149 1786720938.330926 mapread 0x00ceb6 thru 0x011c3e (0x4d89 bytes) 000151 1786720938.364562 read 0x0077e3 thru 0x011c3e (0xa45c bytes) 000152 1786720938.365359 mapwrite 0x02ef63 thru 0x036774 (0x7812 bytes) 000153 1786720938.488184 MIRROR_RESYNC 3 mirrors 000155 1786720939.818186 trunc from 0x036775 to 0x00f3c1 (0x273b3 bytes) 000156 1786720939.947019 trunc from 0x00f3c2 to 0x02ff67 (0x20ba6 bytes) 000157 1786720939.985473 mapwrite 0x01e030 thru 0x028dd7 (0xada8 bytes) 000158 1786720940.106673 trunc from 0x02ff68 to 0x00e29b (0x21ccc bytes) 000159 1786720940.141369 MIRROR_RESYNC 3 mirrors 000160 1786720941.436727 mapwrite 0x02dbf3 thru 0x039a84 (0xbe92 bytes) 000163 1786720941.725050 read 0x00ec1d thru 0x019314 (0xa6f8 bytes) 000165 1786720941.781829 mapread 0x00d776 thru 0x01a491 (0xcd1c bytes) 000166 1786720941.807469 trunc from 0x039a85 to 0x01c273 (0x1d811 bytes) 000167 1786720941.866146 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c274): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c274): Mirror 6: 0x9b2e4523 (stale) Mirror 8: 0x9b2e4523 (stale) CRC-32 checksum value for chunk [0, 0x1c274): Mirror 1: 0x9b2e4523 Mirror 9: 0x9b2e4523 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c274: skipped 000170 1786720943.544288 mapread 0x013e72 thru 0x0183bb (0x454a bytes) 000171 1786720943.595852 mapread 0x015c5d thru 0x017d16 (0x20ba bytes) 000172 1786720943.615757 mapread 0x0098ef thru 0x00ffc1 (0x66d3 bytes) 000175 1786720943.666240 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c274): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c274): Mirror 6: 0x9b2e4523 (stale) Mirror 8: 0x9b2e4523 (stale) CRC-32 checksum value for chunk [0, 0x1c274): Mirror 1: 0x9b2e4523 Mirror 9: 0x9b2e4523 Mirror 10: 0x9b2e4523 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c274: skipped 000176 1786720945.002353 mapwrite 0x037cd0 thru 0x03c1ef (0x4520 bytes) 000182 1786720945.294473 mapread 0x031695 thru 0x03c1ef (0xab5b bytes) 000184 1786720945.305544 read 0x00c414 thru 0x01b1b4 (0xeda1 bytes) 000186 1786720945.317759 mapwrite 0x0364ce thru 0x03ce2a (0x695d bytes) 000188 1786720945.424035 mapread 0x0307dc thru 0x0358c9 (0x50ee bytes) 000190 1786720945.432257 mapwrite 0x02d027 thru 0x0390f5 (0xc0cf bytes) 000191 1786720945.546609 mapwrite 0x0195ae thru 0x026088 (0xcadb bytes) 000192 1786720945.682010 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce2b): Mirror 8: 0x1605748 (stale) Mirror 9: 0x1605748 (stale) Mirror 10: 0x1605748 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce2b): Mirror 8: 0x1605748 (stale) Mirror 9: 0x1605748 (stale) Mirror 10: 0x1605748 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce2b: skipped 000193 1786720946.108154 mapread 0x015b54 thru 0x020948 (0xadf5 bytes) 000194 1786720946.158323 mapread 0x01c7c4 thru 0x01c801 (0x3e bytes) 000195 1786720946.159387 read 0x01e899 thru 0x0280a6 (0x980e bytes) 000196 1786720946.160074 write 0x006b73 thru 0x01386e (0xccfc bytes) 000197 1786720946.181435 trunc from 0x03ce2b to 0x003da0 (0x3908a bytes) 000198 1786720946.213052 trunc from 0x003da1 to 0x02844e (0x246ae bytes) 000199 1786720946.241501 trunc from 0x02844f to 0x000c13 (0x2783b bytes) 000200 1786720946.302742 read 0x000ba5 thru 0x000c13 (0x6f bytes) 000201 1786720946.318046 read 0x000699 thru 0x000c13 (0x57b bytes) 000202 1786720946.318936 read 0x0001a7 thru 0x000c13 (0xa6d bytes) 000203 1786720946.321354 mapwrite 0x018c43 thru 0x02620e (0xd5cc bytes) 000204 1786720946.467515 read 0x006737 thru 0x0162d6 (0xfba0 bytes) 000206 1786720946.471121 trunc from 0x02620f to 0x02c939 (0x672b bytes) 000208 1786720946.501069 write 0x01c1fe thru 0x023841 (0x7644 bytes) 000209 1786720946.502445 mapread 0x01e155 thru 0x01f1c9 (0x1075 bytes) 000210 1786720946.510400 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c93a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c93a): Mirror 8: 0xf277fe30 (stale) Mirror 9: 0xf277fe30 (stale) Mirror 10: 0xf277fe30 (stale) CRC-32 checksum value for chunk [0, 0x2c93a): Mirror 1: 0x640cb74f Mirror 11: 0x640cb74f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c93a: skipped 000213 1786720948.122200 trunc from 0x02c93a to 0x0299f3 (0x2f46 bytes) 000216 1786720948.331189 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x299f4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x299f4): Mirror 8: 0x4d506e7c (stale) Mirror 9: 0x4d506e7c (stale) Mirror 10: 0x4d506e7c (stale) Mirror 11: 0x72a9c80e (stale) CRC-32 checksum value for chunk [0, 0x299f4): Mirror 1: 0x72a9c80e Mirror 12: 0x72a9c80e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x299f4: skipped 000217 1786720950.414596 write 0x033260 thru 0x03ccad (0x9a4e bytes) 000218 1786720950.568666 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ccae): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ccae): Mirror 8: 0xe19443ac (stale) Mirror 9: 0xe19443ac (stale) Mirror 10: 0xe19443ac (stale) Mirror 11: 0x29f64bea (stale) Mirror 12: 0x29f64bea (stale) CRC-32 checksum value for chunk [0, 0x3ccae): Mirror 1: 0xf23c6176 Mirror 13: 0xf23c6176 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ccae: skipped 000219 1786720952.007326 write 0x034997 thru 0x037e64 (0x34ce bytes) 000221 1786720952.179570 trunc from 0x03ccae to 0x03cc56 (0x57 bytes) 000222 1786720952.217139 mapread 0x03164b thru 0x03913d (0x7af3 bytes) 000223 1786720952.253631 mapwrite 0x03177f thru 0x03f67e (0xdf00 bytes) 000224 1786720952.384031 read 0x02debf thru 0x02e14d (0x28f bytes) 000225 1786720952.384742 trunc from 0x03f67f to 0x003f99 (0x3b6e5 bytes) 000227 1786720952.430217 mapwrite 0x003405 thru 0x00865a (0x5256 bytes) 000228 1786720952.595219 trunc from 0x00865b to 0x023403 (0x1ada9 bytes) 000229 1786720952.632440 mapwrite 0x02baae thru 0x02f20c (0x375f bytes) 000231 1786720952.733596 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f20d): Mirror 8: 0x9b140bec (stale) Mirror 9: 0x9b140bec (stale) Mirror 10: 0x9b140bec (stale) Mirror 11: 0xfcb2d0f8 (stale) Mirror 13: 0xfcb2d0f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f20d): Mirror 8: 0x9b140bec (stale) Mirror 9: 0x9b140bec (stale) Mirror 10: 0x9b140bec (stale) Mirror 11: 0xfcb2d0f8 (stale) Mirror 13: 0xfcb2d0f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f20d: skipped 000232 1786720953.361034 read 0x01e17c thru 0x01ec19 (0xa9e bytes) 000234 1786720953.373230 write 0x03cee6 thru 0x03ffff (0x311a bytes) 000235 1786720953.375959 read 0x01e856 thru 0x02330f (0x4aba bytes) 000238 1786720953.420000 mapwrite 0x03e288 thru 0x03eefb (0xc74 bytes) 000239 1786720953.499474 read 0x03f423 thru 0x03ffff (0xbdd bytes) truncating to largest ever: 0x3f06f 000240 1786720953.501479 trunc from 0x040000 to 0x03f06e (0xf91 bytes) 000241 1786720953.530695 mapwrite 0x038865 thru 0x03ffff (0x779b bytes) 000244 1786720953.626422 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0x929b516 (stale) Mirror 13: 0x51768c83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0x929b516 (stale) Mirror 13: 0x51768c83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1786720954.313579 write 0x030010 thru 0x03726e (0x725f bytes) 000247 1786720954.330523 read 0x03382b thru 0x038ad5 (0x52ab bytes) 000249 1786720954.338931 write 0x01d7c7 thru 0x0230a1 (0x58db bytes) 000250 1786720954.353668 write 0x03f6c7 thru 0x03ffff (0x939 bytes) 000252 1786720954.371316 mapwrite 0x01de4a thru 0x02c070 (0xe227 bytes) 000253 1786720954.539291 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0x929b516 (stale) Mirror 13: 0x51768c83 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xecdf8637 Mirror 14: 0xecdf8637 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1786720956.222187 mapread 0x024112 thru 0x027504 (0x33f3 bytes) 000256 1786720956.258529 mapread 0x01c809 thru 0x02a440 (0xdc38 bytes) 000259 1786720956.307382 MIRROR_RESYNC 6 mirrors 000261 1786720958.556731 write 0x00dfda thru 0x00e12f (0x156 bytes) 000263 1786720958.719197 trunc from 0x040000 to 0x020d6c (0x1f293 bytes) 000266 1786720958.760022 read 0x011feb thru 0x019c77 (0x7c8d bytes) 000268 1786720958.776186 mapread 0x0204b0 thru 0x020d6c (0x8bd bytes) 000270 1786720958.785628 write 0x03d5da thru 0x03ffff (0x2a26 bytes) 000271 1786720958.788881 mapread 0x018504 thru 0x02546d (0xcf6a bytes) 000272 1786720958.818688 mapwrite 0x017c3f thru 0x01cd47 (0x5109 bytes) 000274 1786720958.872436 read 0x0129e1 thru 0x0154b6 (0x2ad6 bytes) 000275 1786720958.873232 trunc from 0x040000 to 0x03a1ee (0x5e11 bytes) 000276 1786720958.896031 write 0x031f85 thru 0x034391 (0x240d bytes) 000277 1786720958.925873 mapread 0x02e01a thru 0x03a1ee (0xc1d5 bytes) 000281 1786720958.949200 read 0x00f60c thru 0x016b60 (0x7555 bytes) 000283 1786720958.957982 mapread 0x032c37 thru 0x037dd0 (0x519a bytes) 000284 1786720958.969180 read 0x008ec0 thru 0x01255e (0x969f bytes) 000288 1786720958.992872 write 0x03531b thru 0x0378e0 (0x25c6 bytes) 000289 1786720959.001546 mapwrite 0x022c43 thru 0x028dcd (0x618b bytes) 000291 1786720959.124365 trunc from 0x03a1ef to 0x03b4e1 (0x12f3 bytes) 000293 1786720959.149169 mapwrite 0x01d66f thru 0x021c0c (0x459e bytes) 000294 1786720959.210130 mapwrite 0x01bac5 thru 0x0230d8 (0x7614 bytes) 000295 1786720959.308127 mapread 0x03a8d2 thru 0x03b4e1 (0xc10 bytes) 000298 1786720959.317138 read 0x01ca1c thru 0x01cde9 (0x3ce bytes) 000299 1786720959.319057 trunc from 0x03b4e2 to 0x013649 (0x27e98 bytes) 000300 1786720959.339581 mapread 0x00e602 thru 0x013649 (0x5048 bytes) 000303 1786720959.348422 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1364a): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1364a): Mirror 9: 0x9cce5df5 (stale) Mirror 10: 0x9cce5df5 (stale) Mirror 11: 0x9cce5df5 (stale) Mirror 13: 0x9cce5df5 (stale) Mirror 14: 0x9cce5df5 (stale) CRC-32 checksum value for chunk [0, 0x1364a): Mirror 1: 0xe65a877 Mirror 15: 0xe65a877 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1364a: skipped 000304 1786720961.007759 MIRROR_RESYNC 7 mirrors 000307 1786720963.807631 mapwrite 0x01ba2f thru 0x029bf2 (0xe1c4 bytes) 000308 1786720964.032491 write 0x019142 thru 0x0264d2 (0xd391 bytes) 000309 1786720964.038988 mapwrite 0x01c785 thru 0x01e485 (0x1d01 bytes) 000310 1786720964.085033 read 0x0227d0 thru 0x029bf2 (0x7423 bytes) 000311 1786720964.086652 trunc from 0x029bf3 to 0x02cac7 (0x2ed5 bytes) 000314 1786720964.132096 read 0x00ca19 thru 0x01aa38 (0xe020 bytes) 000316 1786720964.155877 write 0x01b4a7 thru 0x021218 (0x5d72 bytes) 000317 1786720964.164235 read 0x00d087 thru 0x018fa4 (0xbf1e bytes) 000319 1786720964.168832 mapwrite 0x01bb93 thru 0x0259e5 (0x9e53 bytes) 000322 1786720964.237759 read 0x01d9e8 thru 0x025b95 (0x81ae bytes) 000323 1786720964.239667 mapwrite 0x03f9a9 thru 0x03fdc4 (0x41c bytes) 000324 1786720964.420535 write 0x016b20 thru 0x01b55a (0x4a3b bytes) 000328 1786720964.427450 mapwrite 0x005ba3 thru 0x00da58 (0x7eb6 bytes) 000330 1786720964.529053 mapread 0x0315ff thru 0x03d77e (0xc180 bytes) 000333 1786720964.541716 trunc from 0x03fdc5 to 0x00b2b6 (0x34b0e bytes) 000334 1786720964.588044 mapwrite 0x006cb0 thru 0x006ff8 (0x349 bytes) 000336 1786720964.656440 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb2b7): Mirror 9: 0x54cb32f1 (stale) Mirror 10: 0x54cb32f1 (stale) Mirror 11: 0x54cb32f1 (stale) Mirror 13: 0x54cb32f1 (stale) Mirror 15: 0x54cb32f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb2b7): Mirror 9: 0x54cb32f1 (stale) Mirror 10: 0x54cb32f1 (stale) Mirror 11: 0x54cb32f1 (stale) Mirror 13: 0x54cb32f1 (stale) Mirror 15: 0x54cb32f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb2b7: skipped 000337 1786720965.094709 read 0x001db5 thru 0x00acfe (0x8f4a bytes) 000339 1786720965.106237 trunc from 0x00b2b7 to 0x01a8b6 (0xf600 bytes) 000340 1786720965.137102 mapwrite 0x025b25 thru 0x030f9b (0xb477 bytes) 000341 1786720965.252055 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30f9c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f9c): Mirror 9: 0xe9736673 (stale) Mirror 10: 0xe9736673 (stale) Mirror 11: 0xe9736673 (stale) Mirror 13: 0xe9736673 (stale) Mirror 15: 0xe9736673 (stale) CRC-32 checksum value for chunk [0, 0x30f9c): Mirror 1: 0xb27ea556 Mirror 16: 0xb27ea556 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f9c: skipped 000342 1786720966.727468 write 0x017b4a thru 0x01e5a7 (0x6a5e bytes) 000345 1786720966.942371 trunc from 0x030f9c to 0x036714 (0x5779 bytes) 000347 1786720967.053400 write 0x00fd24 thru 0x013fbc (0x4299 bytes) 000348 1786720967.081510 read 0x01c790 thru 0x01cb8a (0x3fb bytes) 000352 1786720967.084822 trunc from 0x036715 to 0x030e84 (0x5890 bytes) 000356 1786720967.125370 trunc from 0x030e85 to 0x039e00 (0x8f7c bytes) 000357 1786720967.154778 write 0x00ae20 thru 0x00b82f (0xa10 bytes) 000358 1786720967.193765 trunc from 0x039e01 to 0x0271e8 (0x12c18 bytes) 000359 1786720967.238908 read 0x009821 thru 0x015d3b (0xc51b bytes) 000360 1786720967.256507 mapread 0x010ff0 thru 0x01394e (0x295f bytes) 000361 1786720967.258398 mapread 0x01d854 thru 0x0254ad (0x7c5a bytes) 000365 1786720967.265570 trunc from 0x0271e9 to 0x029754 (0x256c bytes) 000374 1786720967.339496 mapread 0x01374d thru 0x01bcbb (0x856f bytes) 000376 1786720967.345496 mapread 0x015e0a thru 0x021325 (0xb51c bytes) 000377 1786720967.366671 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29755): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29755): Mirror 9: 0x5afdd681 (stale) Mirror 10: 0x5afdd681 (stale) Mirror 11: 0x5afdd681 (stale) Mirror 13: 0x5afdd681 (stale) Mirror 15: 0x5afdd681 (stale) Mirror 16: 0xc2ec8c2f (stale) CRC-32 checksum value for chunk [0, 0x29755): Mirror 1: 0x38b7a254 Mirror 17: 0x38b7a254 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29755: skipped 000378 1786720969.306104 mapwrite 0x024838 thru 0x02eceb (0xa4b4 bytes) 000379 1786720969.572385 mapread 0x023346 thru 0x02eceb (0xb9a6 bytes) 000380 1786720969.575893 write 0x009629 thru 0x00bbe7 (0x25bf bytes) 000383 1786720969.601744 write 0x036b67 thru 0x03ffff (0x9499 bytes) 000385 1786720969.611331 mapread 0x03b23c thru 0x03ffff (0x4dc4 bytes) 000387 1786720969.638033 MIRROR_RESYNC 8 mirrors 000389 1786720975.188668 trunc from 0x040000 to 0x00ccd0 (0x3332f bytes) 000391 1786720975.383255 write 0x0156a9 thru 0x01c8f3 (0x724b bytes) 000393 1786720975.388581 mapread 0x00ecde thru 0x01c8f3 (0xdc16 bytes) 000394 1786720975.431731 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c8f4): Mirror 10: 0x2bf268c8 (stale) Mirror 11: 0x2bf268c8 (stale) Mirror 13: 0x2bf268c8 (stale) Mirror 15: 0x2bf268c8 (stale) Mirror 16: 0x2bf268c8 (stale) Mirror 17: 0x2bf268c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c8f4): Mirror 10: 0x2bf268c8 (stale) Mirror 11: 0x2bf268c8 (stale) Mirror 13: 0x2bf268c8 (stale) Mirror 15: 0x2bf268c8 (stale) Mirror 16: 0x2bf268c8 (stale) Mirror 17: 0x2bf268c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c8f4: skipped 000395 1786720976.097180 mapwrite 0x03b703 thru 0x03ffff (0x48fd bytes) 000396 1786720976.291360 MIRROR_RESYNC 7 mirrors 000397 1786720980.467751 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000398 1786720980.937587 MIRROR_EXTEND to 7 mirrors 000400 1786720982.223384 mapread 0x00005d thru 0x007a5b (0x79ff bytes) 000401 1786720982.253364 mapwrite 0x00c6a9 thru 0x00e2cc (0x1c24 bytes) 000403 1786720982.443857 MIRROR_RESYNC 7 mirrors 000407 1786720985.603939 trunc from 0x040000 to 0x0338f8 (0xc707 bytes) 000408 1786720985.767568 read 0x000c83 thru 0x004afd (0x3e7b bytes) 000409 1786720985.780958 write 0x027962 thru 0x032331 (0xa9d0 bytes) 000410 1786720985.811222 mapwrite 0x03cc53 thru 0x03f1dd (0x258b bytes) 000411 1786720985.933361 mapread 0x02f808 thru 0x03f1dd (0xf9d6 bytes) 000412 1786720985.945333 mapread 0x005ee5 thru 0x01046b (0xa587 bytes) 000414 1786720985.963406 mapread 0x03264e thru 0x03e453 (0xbe06 bytes) 000415 1786720985.966381 mapwrite 0x039bec thru 0x03cc26 (0x303b bytes) 000416 1786720986.053604 read 0x0383b9 thru 0x03eb1b (0x6763 bytes) 000420 1786720986.074923 read 0x02d30c thru 0x03841c (0xb111 bytes) 000421 1786720986.077366 mapwrite 0x01b229 thru 0x022748 (0x7520 bytes) 000422 1786720986.178718 trunc from 0x03f1de to 0x0266ac (0x18b31 bytes) 000423 1786720986.224149 mapread 0x021376 thru 0x023b12 (0x279d bytes) 000424 1786720986.225501 write 0x0220f9 thru 0x030152 (0xe05a bytes) 000425 1786720986.229531 trunc from 0x030153 to 0x024b90 (0xb5c2 bytes) 000429 1786720986.259993 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: 0x38acf53b (stale) Mirror 11: 0x38acf53b (stale) Mirror 13: 0x38acf53b (stale) Mirror 16: 0x38acf53b (stale) Mirror 17: 0x38acf53b (stale) Mirror 18: 0x38acf53b (stale) CRC-32 checksum value for chunk [0, 0x24b91): Mirror 1: 0xb96b8290 Mirror 19: 0xb96b8290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b91: skipped 000430 1786720987.991926 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: 0x38acf53b (stale) Mirror 11: 0x38acf53b (stale) Mirror 13: 0x38acf53b (stale) Mirror 16: 0x38acf53b (stale) Mirror 17: 0x38acf53b (stale) Mirror 18: 0x38acf53b (stale) CRC-32 checksum value for chunk [0, 0x24b91): Mirror 1: 0xb96b8290 Mirror 19: 0xb96b8290 Mirror 20: 0xb96b8290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b91: skipped 000432 1786720989.468518 read 0x013b97 thru 0x02005c (0xc4c6 bytes) 000434 1786720989.480618 mapread 0x01fdef thru 0x024b90 (0x4da2 bytes) 000435 1786720989.495850 mapwrite 0x000190 thru 0x00501a (0x4e8b bytes) 000436 1786720989.720866 mapwrite 0x0024b0 thru 0x0122a3 (0xfdf4 bytes) 000438 1786720989.815997 mapread 0x0063fb thru 0x0160d9 (0xfcdf bytes) 000439 1786720989.817774 read 0x01eeb4 thru 0x024b90 (0x5cdd bytes) 000440 1786720989.837541 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: 0x38acf53b (stale) Mirror 11: 0x38acf53b (stale) Mirror 13: 0x38acf53b (stale) Mirror 16: 0x38acf53b (stale) Mirror 17: 0x38acf53b (stale) Mirror 18: 0x38acf53b (stale) Mirror 19: 0xb96b8290 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b91): Mirror 10: 0x38acf53b (stale) Mirror 11: 0x38acf53b (stale) Mirror 13: 0x38acf53b (stale) Mirror 16: 0x38acf53b (stale) Mirror 17: 0x38acf53b (stale) Mirror 18: 0x38acf53b (stale) Mirror 19: 0xb96b8290 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b91: skipped 000443 1786720990.496928 read 0x003092 thru 0x00a148 (0x70b7 bytes) 000444 1786720990.517764 read 0x00018c thru 0x0067e2 (0x6657 bytes) 000446 1786720990.548072 write 0x036f24 thru 0x03882c (0x1909 bytes) 000447 1786720990.556496 mapread 0x02d810 thru 0x03882c (0xb01d bytes) 000448 1786720990.573363 write 0x01e863 thru 0x02ac0d (0xc3ab bytes) 000449 1786720990.577703 read 0x00012b thru 0x009158 (0x902e bytes) 000450 1786720990.584383 MIRROR_RESYNC 8 mirrors 000451 1786720993.630081 trunc from 0x03882d to 0x003e81 (0x349ab bytes) 000452 1786720993.709234 write 0x026d21 thru 0x032425 (0xb705 bytes) 000454 1786720993.718547 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32426): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32426): Mirror 10: 0x23845268 (stale) Mirror 11: 0x23845268 (stale) Mirror 13: 0x23845268 (stale) Mirror 16: 0x23845268 (stale) Mirror 17: 0x23845268 (stale) Mirror 18: 0x23845268 (stale) Mirror 19: 0x23845268 (stale) CRC-32 checksum value for chunk [0, 0x32426): Mirror 1: 0xcb68eca Mirror 20: 0xcb68eca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32426: skipped 000455 1786720995.188400 mapread 0x02d4c5 thru 0x032425 (0x4f61 bytes) 000456 1786720995.331666 MIRROR_RESYNC 9 mirrors 000457 1786720997.562847 mapwrite 0x038661 thru 0x03ffff (0x799f bytes) 000459 1786720997.778413 trunc from 0x040000 to 0x0012aa (0x3ed55 bytes) 000461 1786720997.809095 mapread 0x001201 thru 0x0012aa (0xaa bytes) 000464 1786720997.820397 write 0x00efc9 thru 0x013478 (0x44b0 bytes) 000466 1786720997.823657 write 0x0239e9 thru 0x027539 (0x3b51 bytes) 000467 1786720997.831300 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2753a): Mirror 11: 0xa68d4658 (stale) Mirror 13: 0xa68d4658 (stale) Mirror 16: 0xa68d4658 (stale) Mirror 17: 0xa68d4658 (stale) Mirror 18: 0xa68d4658 (stale) Mirror 19: 0xa68d4658 (stale) Mirror 20: 0xa68d4658 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2753a): Mirror 11: 0xa68d4658 (stale) Mirror 13: 0xa68d4658 (stale) Mirror 16: 0xa68d4658 (stale) Mirror 17: 0xa68d4658 (stale) Mirror 18: 0xa68d4658 (stale) Mirror 19: 0xa68d4658 (stale) Mirror 20: 0xa68d4658 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2753a: skipped 000468 1786720998.533316 trunc from 0x02753a to 0x036e87 (0xf94e bytes) 000470 1786720998.571165 write 0x03d4d9 thru 0x03ffff (0x2b27 bytes) 000472 1786720998.573962 read 0x029132 thru 0x02d209 (0x40d8 bytes) 000473 1786720998.598024 mapread 0x00f27e thru 0x011df3 (0x2b76 bytes) 000474 1786720998.646973 mapwrite 0x01777d thru 0x01f116 (0x799a bytes) 000478 1786720998.794997 MIRROR_RESYNC 8 mirrors 000479 1786721003.549321 mapwrite 0x00a81e thru 0x013372 (0x8b55 bytes) 000480 1786721003.810616 mapread 0x02239e thru 0x026936 (0x4599 bytes) 000483 1786721003.835033 MIRROR_RESYNC 8 mirrors 000484 1786721008.151770 mapwrite 0x028230 thru 0x02bade (0x38af bytes) 000485 1786721008.515586 read 0x026f69 thru 0x030b66 (0x9bfe bytes) 000487 1786721008.519976 mapwrite 0x02b540 thru 0x036820 (0xb2e1 bytes) 000489 1786721008.700596 mapread 0x0291c5 thru 0x0343dd (0xb219 bytes) 000490 1786721008.707264 read 0x00829d thru 0x008b4f (0x8b3 bytes) 000491 1786721008.741495 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb433e896 (stale) Mirror 16: 0xb433e896 (stale) Mirror 17: 0xb433e896 (stale) Mirror 18: 0xb433e896 (stale) Mirror 19: 0xb433e896 (stale) Mirror 20: 0xb433e896 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb433e896 (stale) Mirror 16: 0xb433e896 (stale) Mirror 17: 0xb433e896 (stale) Mirror 18: 0xb433e896 (stale) Mirror 19: 0xb433e896 (stale) Mirror 20: 0xb433e896 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1786721009.606355 trunc from 0x040000 to 0x00b102 (0x34efd bytes) 000495 1786721009.628686 mapread 0x0058b4 thru 0x00b102 (0x584f bytes) 000496 1786721009.705088 trunc from 0x00b103 to 0x03cb6a (0x31a68 bytes) 000497 1786721009.751260 read 0x00c798 thru 0x0160ea (0x9953 bytes) 000498 1786721009.763616 read 0x025a61 thru 0x025e55 (0x3f5 bytes) 000499 1786721009.793602 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cb6b): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb6b): Mirror 11: 0xa7bae297 (stale) Mirror 16: 0xa7bae297 (stale) Mirror 17: 0xa7bae297 (stale) Mirror 18: 0xa7bae297 (stale) Mirror 19: 0xa7bae297 (stale) Mirror 20: 0xa7bae297 (stale) CRC-32 checksum value for chunk [0, 0x3cb6b): Mirror 1: 0xaec5af06 Mirror 21: 0xaec5af06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb6b: skipped 000500 1786721011.966250 read 0x020075 thru 0x024121 (0x40ad bytes) 000502 1786721012.041102 MIRROR_RESYNC 8 mirrors 000505 1786721014.814032 mapwrite 0x00e021 thru 0x014c54 (0x6c34 bytes) 000513 1786721015.252699 trunc from 0x03cb6b to 0x01013a (0x2ca30 bytes) 000517 1786721015.394596 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1013b): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1013b): Mirror 11: 0xc9edd678 (stale) Mirror 16: 0xc9edd678 (stale) Mirror 17: 0xc9edd678 (stale) Mirror 18: 0xc9edd678 (stale) Mirror 19: 0xc9edd678 (stale) Mirror 20: 0xc9edd678 (stale) Mirror 21: 0xc9edd678 (stale) CRC-32 checksum value for chunk [0, 0x1013b): Mirror 1: 0xe66cc51b Mirror 22: 0xe66cc51b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1013b: skipped 000518 1786721017.325613 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1013b): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1013b): Mirror 11: 0xc9edd678 (stale) Mirror 16: 0xc9edd678 (stale) Mirror 17: 0xc9edd678 (stale) Mirror 18: 0xc9edd678 (stale) Mirror 19: 0xc9edd678 (stale) Mirror 20: 0xc9edd678 (stale) Mirror 21: 0xc9edd678 (stale) CRC-32 checksum value for chunk [0, 0x1013b): Mirror 1: 0xe66cc51b Mirror 22: 0xe66cc51b Mirror 23: 0xe66cc51b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1013b: skipped 000519 1786721018.925006 write 0x03fd93 thru 0x03ffff (0x26d bytes) 000521 1786721019.129478 mapread 0x00a440 thru 0x00c4d8 (0x2099 bytes) 000522 1786721019.160292 mapread 0x03c54f thru 0x03ffff (0x3ab1 bytes) 000523 1786721019.169414 trunc from 0x040000 to 0x00b618 (0x349e7 bytes) 000525 1786721019.210356 trunc from 0x00b619 to 0x02b165 (0x1fb4d bytes) 000526 1786721019.241380 trunc from 0x02b166 to 0x02a5d1 (0xb94 bytes) 000528 1786721019.263113 mapread 0x0289c8 thru 0x02a5d1 (0x1c0a bytes) 000529 1786721019.289117 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2a5d2): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a5d2): Mirror 11: 0xfd32c2a4 (stale) Mirror 16: 0xfd32c2a4 (stale) Mirror 17: 0xfd32c2a4 (stale) Mirror 18: 0xfd32c2a4 (stale) Mirror 19: 0xfd32c2a4 (stale) Mirror 20: 0xfd32c2a4 (stale) Mirror 21: 0xfd32c2a4 (stale) Mirror 22: 0x58224f15 (stale) Mirror 23: 0x58224f15 (stale) CRC-32 checksum value for chunk [0, 0x2a5d2): Mirror 1: 0xfd32c2a4 Mirror 24: 0xfd32c2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a5d2: skipped 000530 1786721021.199328 write 0x01539c thru 0x024638 (0xf29d bytes) 000531 1786721021.280298 write 0x03eb25 thru 0x03ffff (0x14db bytes) 000533 1786721021.283145 write 0x014e89 thru 0x020189 (0xb301 bytes) 000534 1786721021.288960 read 0x023480 thru 0x0300dc (0xcc5d bytes) 000535 1786721021.298256 mapread 0x031274 thru 0x03d625 (0xc3b2 bytes) 000536 1786721021.338831 MIRROR_RESYNC 11 mirrors 000537 1786721026.418555 mapwrite 0x022803 thru 0x02ec4d (0xc44b bytes) 000538 1786721026.704701 trunc from 0x040000 to 0x02ab27 (0x154d8 bytes) 000541 1786721026.748233 read 0x028042 thru 0x02a086 (0x2045 bytes) 000544 1786721026.750511 write 0x0078d3 thru 0x008742 (0xe70 bytes) 000545 1786721026.765615 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ab28): Mirror 11: 0x53a1bc9f (stale) Mirror 16: 0x53a1bc9f (stale) Mirror 18: 0x53a1bc9f (stale) Mirror 19: 0x53a1bc9f (stale) Mirror 20: 0x53a1bc9f (stale) Mirror 21: 0x53a1bc9f (stale) Mirror 22: 0x53a1bc9f (stale) Mirror 23: 0x53a1bc9f (stale) Mirror 24: 0x53a1bc9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab28): Mirror 11: 0x53a1bc9f (stale) Mirror 16: 0x53a1bc9f (stale) Mirror 18: 0x53a1bc9f (stale) Mirror 19: 0x53a1bc9f (stale) Mirror 20: 0x53a1bc9f (stale) Mirror 21: 0x53a1bc9f (stale) Mirror 22: 0x53a1bc9f (stale) Mirror 23: 0x53a1bc9f (stale) Mirror 24: 0x53a1bc9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab28: skipped 000546 1786721027.468484 write 0x00f810 thru 0x01d6e4 (0xded5 bytes) 000547 1786721027.482443 mapwrite 0x00f806 thru 0x01719b (0x7996 bytes) 000549 1786721027.585635 read 0x017684 thru 0x01e4b1 (0x6e2e bytes) 000550 1786721027.625004 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ab28): Mirror 11: 0x53a1bc9f (stale) Mirror 16: 0x53a1bc9f (stale) Mirror 18: 0x53a1bc9f (stale) Mirror 19: 0x53a1bc9f (stale) Mirror 20: 0x53a1bc9f (stale) Mirror 21: 0x53a1bc9f (stale) Mirror 22: 0x53a1bc9f (stale) Mirror 23: 0x53a1bc9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab28): Mirror 11: 0x53a1bc9f (stale) Mirror 16: 0x53a1bc9f (stale) Mirror 18: 0x53a1bc9f (stale) Mirror 19: 0x53a1bc9f (stale) Mirror 20: 0x53a1bc9f (stale) Mirror 21: 0x53a1bc9f (stale) Mirror 22: 0x53a1bc9f (stale) Mirror 23: 0x53a1bc9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab28: skipped 000552 1786721028.648430 read 0x01528e thru 0x01b14a (0x5ebd bytes) 000553 1786721028.664216 read 0x0099a1 thru 0x00dbe3 (0x4243 bytes) 000558 1786721028.696418 read 0x005c1e thru 0x0157d6 (0xfbb9 bytes) 000560 1786721028.703082 write 0x024a3d thru 0x025052 (0x616 bytes) 000561 1786721028.704315 read 0x00e2ae thru 0x01196a (0x36bd bytes) 000562 1786721028.706457 mapread 0x004a74 thru 0x008f94 (0x4521 bytes) 000563 1786721028.716399 trunc from 0x02ab28 to 0x0147f3 (0x16334 bytes) 000564 1786721028.763092 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x147f4): Mirror 11: 0x7d3b3115 (stale) Mirror 16: 0x7d3b3115 (stale) Mirror 19: 0x7d3b3115 (stale) Mirror 20: 0x7d3b3115 (stale) Mirror 21: 0x7d3b3115 (stale) Mirror 22: 0x7d3b3115 (stale) Mirror 23: 0x7d3b3115 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x147f4): Mirror 11: 0x7d3b3115 (stale) Mirror 16: 0x7d3b3115 (stale) Mirror 19: 0x7d3b3115 (stale) Mirror 20: 0x7d3b3115 (stale) Mirror 21: 0x7d3b3115 (stale) Mirror 22: 0x7d3b3115 (stale) Mirror 23: 0x7d3b3115 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x147f4: skipped 000565 1786721029.563077 trunc from 0x0147f4 to 0x0102b2 (0x4541 bytes) 000566 1786721029.584643 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x102b3): Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x102b3): Mirror 11: 0xc21b53b2 (stale) Mirror 16: 0xc21b53b2 (stale) Mirror 19: 0xc21b53b2 (stale) Mirror 20: 0xc21b53b2 (stale) Mirror 21: 0xc21b53b2 (stale) Mirror 22: 0xc21b53b2 (stale) Mirror 23: 0xc21b53b2 (stale) CRC-32 checksum value for chunk [0, 0x102b3): Mirror 1: 0x7ba3a0c7 Mirror 24: 0x7ba3a0c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x102b3: skipped 000569 1786721031.248026 mapwrite 0x009f6d thru 0x00c9d4 (0x2a68 bytes) 000570 1786721031.482296 mapwrite 0x0384be thru 0x03c1ac (0x3cef bytes) 000571 1786721031.631177 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1ad): Mirror 16: 0xdf791279 (stale) Mirror 19: 0xdf791279 (stale) Mirror 20: 0xdf791279 (stale) Mirror 21: 0xdf791279 (stale) Mirror 22: 0xdf791279 (stale) Mirror 23: 0xdf791279 (stale) Mirror 24: 0x59870ff2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1ad): Mirror 16: 0xdf791279 (stale) Mirror 19: 0xdf791279 (stale) Mirror 20: 0xdf791279 (stale) Mirror 21: 0xdf791279 (stale) Mirror 22: 0xdf791279 (stale) Mirror 23: 0xdf791279 (stale) Mirror 24: 0x59870ff2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1ad: skipped 000572 1786721032.532366 mapwrite 0x008f0f thru 0x00f6a0 (0x6792 bytes) 000573 1786721032.626073 trunc from 0x03c1ad to 0x025cb3 (0x164f9 bytes) 000574 1786721032.666388 mapwrite 0x008a35 thru 0x014b37 (0xc103 bytes) 000576 1786721032.805705 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: 0x5691a14b (stale) Mirror 19: 0x5691a14b (stale) Mirror 20: 0x5691a14b (stale) Mirror 21: 0x5691a14b (stale) Mirror 22: 0x5691a14b (stale) Mirror 23: 0x5691a14b (stale) Mirror 24: 0xe81a6d74 (stale) CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 1: 0x67064b74 Mirror 25: 0x67064b74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb4: skipped 000577 1786721034.431154 mapread 0x015cbc thru 0x0248ec (0xec31 bytes) 000578 1786721034.527987 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: 0x5691a14b (stale) Mirror 19: 0x5691a14b (stale) Mirror 20: 0x5691a14b (stale) Mirror 21: 0x5691a14b (stale) Mirror 22: 0x5691a14b (stale) Mirror 23: 0x5691a14b (stale) Mirror 24: 0xe81a6d74 (stale) CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 1: 0x67064b74 Mirror 25: 0x67064b74 Mirror 26: 0x67064b74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb4: skipped 000579 1786721035.997275 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 16: 0x5691a14b (stale) Mirror 19: 0x5691a14b (stale) Mirror 20: 0x5691a14b (stale) Mirror 21: 0x5691a14b (stale) Mirror 22: 0x5691a14b (stale) Mirror 23: 0x5691a14b (stale) Mirror 24: 0xe81a6d74 (stale) CRC-32 checksum value for chunk [0, 0x25cb4): Mirror 1: 0x67064b74 Mirror 25: 0x67064b74 Mirror 26: 0x67064b74 Mirror 27: 0x67064b74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb4: skipped 000580 1786721037.171665 write 0x03eed4 thru 0x03ffff (0x112c bytes) 000581 1786721037.261861 mapwrite 0x017029 thru 0x0215a7 (0xa57f bytes) 000583 1786721037.347146 write 0x01b35f thru 0x0226a7 (0x7349 bytes) 000585 1786721037.355983 mapwrite 0x01054c thru 0x017a0a (0x74bf bytes) 000586 1786721037.462963 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xbd63d16 (stale) Mirror 19: 0xbd63d16 (stale) Mirror 20: 0xbd63d16 (stale) Mirror 21: 0xbd63d16 (stale) Mirror 22: 0xbd63d16 (stale) Mirror 23: 0xbd63d16 (stale) Mirror 24: 0xf9d5ce0c (stale) Mirror 25: 0xf090ee21 (stale) Mirror 26: 0xf090ee21 (stale) Mirror 27: 0xf090ee21 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa8f8d19 Mirror 28: 0xfa8f8d19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1786721038.960391 trunc from 0x040000 to 0x01bab1 (0x2454e bytes) 000589 1786721039.048127 trunc from 0x01bab2 to 0x00e8c9 (0xd1e8 bytes) 000590 1786721039.073175 read 0x006622 thru 0x00e8c9 (0x82a8 bytes) 000594 1786721039.096273 mapwrite 0x00ef77 thru 0x01a6e2 (0xb76c bytes) 000597 1786721039.264418 trunc from 0x01a6e3 to 0x00a32c (0x103b6 bytes) 000600 1786721039.278720 mapwrite 0x00a5b6 thru 0x010361 (0x5dac bytes) 000601 1786721039.415441 mapwrite 0x025030 thru 0x02cdb5 (0x7d86 bytes) 000604 1786721039.544850 trunc from 0x02cdb6 to 0x03c183 (0xf3ce bytes) 000605 1786721039.564066 trunc from 0x03c184 to 0x01a3ba (0x21dc9 bytes) 000607 1786721039.589887 read 0x00a508 thru 0x00fc6e (0x5767 bytes) 000608 1786721039.591606 write 0x028596 thru 0x034086 (0xbaf1 bytes) 000610 1786721039.598359 write 0x01ff8b thru 0x022ef6 (0x2f6c bytes) 000612 1786721039.620650 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34087): Mirror 16: 0xb86332ed (stale) Mirror 19: 0xb86332ed (stale) Mirror 21: 0xb86332ed (stale) Mirror 22: 0xb86332ed (stale) Mirror 23: 0xb86332ed (stale) Mirror 24: 0xb443ea56 (stale) Mirror 25: 0x2639d277 (stale) Mirror 26: 0x2639d277 (stale) Mirror 27: 0x2639d277 (stale) Mirror 28: 0xbb7c7392 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34087): Mirror 16: 0xb86332ed (stale) Mirror 19: 0xb86332ed (stale) Mirror 21: 0xb86332ed (stale) Mirror 22: 0xb86332ed (stale) Mirror 23: 0xb86332ed (stale) Mirror 24: 0xb443ea56 (stale) Mirror 25: 0x2639d277 (stale) Mirror 26: 0x2639d277 (stale) Mirror 27: 0x2639d277 (stale) Mirror 28: 0xbb7c7392 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34087: skipped 000613 1786721040.329514 read 0x002cb5 thru 0x00be1a (0x9166 bytes) 000615 1786721040.347822 read 0x00ad12 thru 0x00e7d4 (0x3ac3 bytes) 000618 1786721040.366447 mapread 0x017412 thru 0x01e736 (0x7325 bytes) 000619 1786721040.368096 read 0x007e4d thru 0x009ed1 (0x2085 bytes) 000620 1786721040.369598 read 0x0175ff thru 0x020218 (0x8c1a bytes) 000621 1786721040.370803 write 0x00f29e thru 0x01d429 (0xe18c bytes) 000623 1786721040.374456 mapread 0x001aa8 thru 0x00f36d (0xd8c6 bytes) 000624 1786721040.376983 mapread 0x019573 thru 0x01d515 (0x3fa3 bytes) 000625 1786721040.381278 mapread 0x0108c0 thru 0x01b4e8 (0xac29 bytes) 000627 1786721040.384011 mapread 0x00f331 thru 0x01108f (0x1d5f bytes) 000629 1786721040.386081 mapread 0x02e4ba thru 0x030b49 (0x2690 bytes) 000630 1786721040.387946 read 0x014e8e thru 0x0177c1 (0x2934 bytes) 000631 1786721040.389767 trunc from 0x034087 to 0x0175ad (0x1cad9 bytes) 000632 1786721040.407731 write 0x009dcc thru 0x00a9dd (0xc12 bytes) 000633 1786721040.414974 MIRROR_RESYNC 11 mirrors 000634 1786721044.014954 mapwrite 0x03ded5 thru 0x03ffff (0x212b bytes) 000637 1786721044.340421 write 0x02a21b thru 0x02b783 (0x1569 bytes) 000638 1786721044.352559 read 0x00246c thru 0x00909a (0x6c2f bytes) 000641 1786721044.367402 read 0x0246f0 thru 0x027e6a (0x377b bytes) 000642 1786721044.413996 mapread 0x007055 thru 0x01392b (0xc8d7 bytes) 000643 1786721044.420402 mapread 0x01fbf4 thru 0x02cb01 (0xcf0e bytes) 000644 1786721044.426911 mapread 0x016128 thru 0x020035 (0x9f0e bytes) 000645 1786721044.432303 mapread 0x017722 thru 0x022b22 (0xb401 bytes) 000646 1786721044.433880 read 0x008e6b thru 0x00e7a6 (0x593c bytes) 000647 1786721044.436763 trunc from 0x040000 to 0x03bf81 (0x407e bytes) 000648 1786721044.510539 read 0x006862 thru 0x01142b (0xabca bytes) 000651 1786721044.520367 read 0x021ce5 thru 0x029aa9 (0x7dc5 bytes) 000652 1786721044.524343 trunc from 0x03bf82 to 0x003ca4 (0x382dd bytes) 000653 1786721044.567130 mapread 0x00339b thru 0x003ca4 (0x90a bytes) 000655 1786721044.569723 mapread 0x00073b thru 0x003ca4 (0x356a bytes) 000656 1786721044.579241 mapwrite 0x017f3d thru 0x027e5b (0xff1f bytes) 000661 1786721044.691480 read 0x01a658 thru 0x025549 (0xaef2 bytes) 000662 1786721044.692509 trunc from 0x027e5c to 0x01315d (0x14cfe bytes) 000665 1786721044.729539 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1315e): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1315e): Mirror 16: 0x80d585e5 (stale) Mirror 19: 0x80d585e5 (stale) Mirror 21: 0x80d585e5 (stale) Mirror 22: 0x80d585e5 (stale) Mirror 23: 0x80d585e5 (stale) Mirror 24: 0x80d585e5 (stale) Mirror 25: 0x80d585e5 (stale) Mirror 26: 0x80d585e5 (stale) Mirror 27: 0x80d585e5 (stale) Mirror 28: 0x80d585e5 (stale) CRC-32 checksum value for chunk [0, 0x1315e): Mirror 1: 0xf93778ae Mirror 29: 0xf93778ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1315e: skipped 000670 1786721046.951507 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1315e): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1315e): Mirror 16: 0x80d585e5 (stale) Mirror 19: 0x80d585e5 (stale) Mirror 21: 0x80d585e5 (stale) Mirror 22: 0x80d585e5 (stale) Mirror 23: 0x80d585e5 (stale) Mirror 24: 0x80d585e5 (stale) Mirror 25: 0x80d585e5 (stale) Mirror 26: 0x80d585e5 (stale) Mirror 27: 0x80d585e5 (stale) Mirror 28: 0x80d585e5 (stale) CRC-32 checksum value for chunk [0, 0x1315e): Mirror 1: 0xf93778ae Mirror 29: 0xf93778ae Mirror 30: 0xf93778ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1315e: skipped 000671 1786721048.532887 mapread 0x0109fb thru 0x01315d (0x2763 bytes) 000674 1786721048.611839 write 0x028b0d thru 0x02f321 (0x6815 bytes) 000675 1786721048.765522 trunc from 0x02f322 to 0x01a9b9 (0x14968 bytes) 000676 1786721048.839011 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a9ba): Mirror 16: 0xac15dee9 (stale) Mirror 19: 0xac15dee9 (stale) Mirror 21: 0xac15dee9 (stale) Mirror 22: 0xac15dee9 (stale) Mirror 23: 0xac15dee9 (stale) Mirror 24: 0xac15dee9 (stale) Mirror 25: 0xac15dee9 (stale) Mirror 26: 0xac15dee9 (stale) Mirror 27: 0xac15dee9 (stale) Mirror 28: 0xac15dee9 (stale) Mirror 30: 0x93af983b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a9ba): Mirror 16: 0xac15dee9 (stale) Mirror 19: 0xac15dee9 (stale) Mirror 21: 0xac15dee9 (stale) Mirror 22: 0xac15dee9 (stale) Mirror 23: 0xac15dee9 (stale) Mirror 24: 0xac15dee9 (stale) Mirror 25: 0xac15dee9 (stale) Mirror 26: 0xac15dee9 (stale) Mirror 27: 0xac15dee9 (stale) Mirror 28: 0xac15dee9 (stale) Mirror 30: 0x93af983b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a9ba: skipped 000677 1786721049.604311 read 0x017e34 thru 0x01a9b9 (0x2b86 bytes) 000678 1786721049.611601 read 0x00b1f2 thru 0x010764 (0x5573 bytes) 000679 1786721049.631829 read 0x00ed8c thru 0x0109c0 (0x1c35 bytes) 000680 1786721049.632551 mapwrite 0x01d730 thru 0x02a157 (0xca28 bytes) 000683 1786721049.814279 read 0x00a7a8 thru 0x00e90e (0x4167 bytes) 000685 1786721049.819045 mapread 0x006147 thru 0x012b40 (0xc9fa bytes) 000686 1786721049.820905 mapwrite 0x00565b thru 0x00bfdb (0x6981 bytes) 000687 1786721049.948280 mapwrite 0x00f8d1 thru 0x01278b (0x2ebb bytes) 000688 1786721050.053555 write 0x03a7c8 thru 0x03ea50 (0x4289 bytes) 000689 1786721050.061044 mapwrite 0x011b95 thru 0x016584 (0x49f0 bytes) 000693 1786721050.167051 write 0x03cf34 thru 0x03f4b9 (0x2586 bytes) 000694 1786721050.169588 mapwrite 0x024f30 thru 0x02d1e2 (0x82b3 bytes) 000699 1786721050.261767 MIRROR_RESYNC 12 mirrors 000700 1786721057.292570 write 0x031a5f thru 0x03ffff (0xe5a1 bytes) 000701 1786721057.388189 trunc from 0x040000 to 0x03e43d (0x1bc2 bytes) 000702 1786721057.423942 trunc from 0x03e43e to 0x01dd1a (0x20723 bytes) 000703 1786721057.454418 MIRROR_RESYNC 12 mirrors 000705 1786721062.807432 trunc from 0x01dd1b to 0x03cab6 (0x1ed9c bytes) 000706 1786721062.934325 mapwrite 0x03348c thru 0x035f01 (0x2a76 bytes) 000707 1786721063.029258 mapread 0x02e7de thru 0x036764 (0x7f87 bytes) 000713 1786721063.039751 mapwrite 0x03301c thru 0x03603e (0x3023 bytes) 000714 1786721063.100491 mapread 0x00440f thru 0x01183b (0xd42d bytes) 000716 1786721063.122489 write 0x01ba6f thru 0x0238f8 (0x7e8a bytes) 000717 1786721063.139364 mapread 0x01be6b thru 0x01fd91 (0x3f27 bytes) 000718 1786721063.146277 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3cab7): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cab7): Mirror 16: 0x18e297dc (stale) Mirror 19: 0x18e297dc (stale) Mirror 21: 0x18e297dc (stale) Mirror 22: 0x18e297dc (stale) Mirror 23: 0x18e297dc (stale) Mirror 24: 0x18e297dc (stale) Mirror 25: 0x18e297dc (stale) Mirror 26: 0x18e297dc (stale) Mirror 27: 0x18e297dc (stale) Mirror 28: 0x18e297dc (stale) Mirror 30: 0x18e297dc (stale) CRC-32 checksum value for chunk [0, 0x3cab7): Mirror 1: 0xc2e08967 Mirror 31: 0xc2e08967 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cab7: skipped 000719 1786721064.957454 read 0x02afa2 thru 0x033fc7 (0x9026 bytes) 000721 1786721064.993068 write 0x02e4b5 thru 0x0353d6 (0x6f22 bytes) 000722 1786721065.191353 MIRROR_RESYNC 13 mirrors 000724 1786721072.873365 read 0x01effd thru 0x02c2c3 (0xd2c7 bytes) 000725 1786721072.898333 write 0x01f7e9 thru 0x021828 (0x2040 bytes) 000728 1786721073.082913 mapread 0x031d43 thru 0x039d1e (0x7fdc bytes) 000729 1786721073.115699 write 0x02f270 thru 0x032f45 (0x3cd6 bytes) 000732 1786721073.128440 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cab7): Mirror 19: 0xd0be8719 (stale) Mirror 21: 0xd0be8719 (stale) Mirror 22: 0xd0be8719 (stale) Mirror 23: 0xd0be8719 (stale) Mirror 24: 0xd0be8719 (stale) Mirror 25: 0xd0be8719 (stale) Mirror 26: 0xd0be8719 (stale) Mirror 27: 0xd0be8719 (stale) Mirror 28: 0xd0be8719 (stale) Mirror 30: 0xd0be8719 (stale) Mirror 31: 0xd0be8719 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cab7): Mirror 19: 0xd0be8719 (stale) Mirror 21: 0xd0be8719 (stale) Mirror 22: 0xd0be8719 (stale) Mirror 23: 0xd0be8719 (stale) Mirror 24: 0xd0be8719 (stale) Mirror 25: 0xd0be8719 (stale) Mirror 26: 0xd0be8719 (stale) Mirror 27: 0xd0be8719 (stale) Mirror 28: 0xd0be8719 (stale) Mirror 30: 0xd0be8719 (stale) Mirror 31: 0xd0be8719 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cab7: skipped 000733 1786721074.112521 trunc from 0x03cab7 to 0x02cf4b (0xfb6b bytes) 000734 1786721074.196777 MIRROR_RESYNC 12 mirrors 000735 1786721080.069048 write 0x035a84 thru 0x03e55a (0x8ad7 bytes) 000736 1786721080.310522 MIRROR_RESYNC 12 mirrors 000738 1786721089.705955 mapwrite 0x010eb3 thru 0x01e85e (0xd9ac bytes) 000739 1786721090.175289 trunc from 0x03e55b to 0x02de21 (0x10739 bytes) 000740 1786721090.218555 write 0x016e48 thru 0x0246cd (0xd886 bytes) 000742 1786721090.243716 write 0x02c484 thru 0x034968 (0x84e5 bytes) 000743 1786721090.254739 write 0x01f09e thru 0x02913d (0xa0a0 bytes) 000744 1786721090.277499 write 0x03fbbf thru 0x03ffff (0x441 bytes) 000748 1786721090.288502 trunc from 0x040000 to 0x02af16 (0x150e9 bytes) 000752 1786721090.339371 write 0x0189a4 thru 0x0246ef (0xbd4c bytes) 000753 1786721090.342327 mapread 0x002579 thru 0x008f24 (0x69ac bytes) 000754 1786721090.365213 trunc from 0x02af17 to 0x035486 (0xa570 bytes) 000758 1786721090.465272 write 0x01235d thru 0x012f07 (0xbab bytes) 000759 1786721090.468502 write 0x01e9df thru 0x023aeb (0x510d bytes) 000760 1786721090.471420 trunc from 0x035487 to 0x03cb1c (0x7696 bytes) 000765 1786721090.516430 write 0x03c97d thru 0x03ffff (0x3683 bytes) 000766 1786721090.538769 read 0x02a79a thru 0x02d52a (0x2d91 bytes) 000767 1786721090.549752 write 0x03a7bd thru 0x03ffff (0x5843 bytes) 000768 1786721090.593167 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xcb45f929 (stale) Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 28: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xcb45f929 (stale) Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 28: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000769 1786721091.918079 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xcb45f929 (stale) Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xcb45f929 (stale) Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1786721093.146669 trunc from 0x040000 to 0x001291 (0x3ed6e bytes) 000773 1786721093.167656 read 0x000fb8 thru 0x001291 (0x2da bytes) 000776 1786721093.177213 write 0x03c5c4 thru 0x03ffff (0x3a3c bytes) 000777 1786721093.179876 mapread 0x00c32f thru 0x015210 (0x8ee2 bytes) 000778 1786721093.199011 mapread 0x03b131 thru 0x03ffff (0x4ecf bytes) 000779 1786721093.213041 read 0x0086ec thru 0x00de86 (0x579b bytes) 000780 1786721093.214727 trunc from 0x040000 to 0x003a98 (0x3c567 bytes) 000781 1786721093.247329 mapwrite 0x006121 thru 0x009bf0 (0x3ad0 bytes) 000782 1786721093.371254 trunc from 0x009bf1 to 0x037a56 (0x2de66 bytes) 000783 1786721093.408252 trunc from 0x037a57 to 0x037dd1 (0x37b bytes) 000784 1786721093.456834 mapwrite 0x000c17 thru 0x005251 (0x463b bytes) 000789 1786721093.527372 write 0x03e159 thru 0x03ffff (0x1ea7 bytes) 000791 1786721093.536666 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000792 1786721094.529430 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xcb45f929 (stale) Mirror 22: 0xcb45f929 (stale) Mirror 23: 0xcb45f929 (stale) Mirror 24: 0xcb45f929 (stale) Mirror 25: 0xcb45f929 (stale) Mirror 26: 0xcb45f929 (stale) Mirror 30: 0xcb45f929 (stale) Mirror 31: 0xcb45f929 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ab9c050 Mirror 32: 0x7ab9c050 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000793 1786721096.416383 read 0x0308a4 thru 0x032d18 (0x2475 bytes) 000794 1786721096.437422 read 0x027dc1 thru 0x034534 (0xc774 bytes) 000795 1786721096.465570 read 0x001557 thru 0x00a1c6 (0x8c70 bytes) 000796 1786721096.493240 MIRROR_RESYNC 10 mirrors 000797 1786721100.715203 mapread 0x01c3cc thru 0x025f95 (0x9bca bytes) 000798 1786721100.749403 write 0x00c6d8 thru 0x0141f8 (0x7b21 bytes) 000800 1786721100.858293 mapwrite 0x02923e thru 0x032ae5 (0x98a8 bytes) 000802 1786721100.970374 read 0x020b94 thru 0x025b42 (0x4faf bytes) 000803 1786721100.981831 trunc from 0x040000 to 0x018dfb (0x27204 bytes) 000804 1786721101.026611 MIRROR_RESYNC 10 mirrors 000805 1786721104.428659 read 0x00678e thru 0x009201 (0x2a74 bytes) 000806 1786721104.440469 read 0x016782 thru 0x018dfb (0x267a bytes) 000810 1786721104.453161 mapread 0x0029ad thru 0x00f37d (0xc9d1 bytes) 000811 1786721104.454546 mapread 0x003f58 thru 0x0055c9 (0x1672 bytes) 000814 1786721104.456246 mapread 0x00d10f thru 0x011d0b (0x4bfd bytes) 000815 1786721104.457219 write 0x03fec6 thru 0x03ffff (0x13a bytes) 000818 1786721104.526499 mapwrite 0x00a9b6 thru 0x00c70b (0x1d56 bytes) 000819 1786721104.685284 read 0x02f6f1 thru 0x03e7c4 (0xf0d4 bytes) 000820 1786721104.687363 mapwrite 0x03787b thru 0x03ffff (0x8785 bytes) 000821 1786721104.791247 mapread 0x03bafd thru 0x03cc1a (0x111e bytes) 000822 1786721104.794690 read 0x007ca2 thru 0x00909a (0x13f9 bytes) 000827 1786721104.828533 write 0x03785f thru 0x03ffff (0x87a1 bytes) 000829 1786721104.832683 mapwrite 0x000919 thru 0x00815c (0x7844 bytes) 000831 1786721104.912884 mapread 0x0184e9 thru 0x018dc5 (0x8dd bytes) 000832 1786721104.915241 trunc from 0x040000 to 0x0309ac (0xf653 bytes) 000834 1786721104.985940 write 0x00c4ee thru 0x016fce (0xaae1 bytes) 000837 1786721104.997905 mapwrite 0x03f7a6 thru 0x03ffff (0x85a bytes) 000839 1786721105.100714 mapread 0x039af9 thru 0x03ffff (0x6507 bytes) 000842 1786721105.149159 mapread 0x03dde6 thru 0x03ffff (0x221a bytes) 000844 1786721105.160880 write 0x0204bb thru 0x0226c3 (0x2209 bytes) 000847 1786721105.165390 write 0x01dd6a thru 0x0204dd (0x2774 bytes) 000848 1786721105.167179 trunc from 0x040000 to 0x026c77 (0x19388 bytes) 000849 1786721105.208675 read 0x00b633 thru 0x01019b (0x4b69 bytes) 000850 1786721105.217236 write 0x02eff4 thru 0x034144 (0x5151 bytes) 000852 1786721105.235469 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34145): Mirror 21: 0xee7e1591 (stale) Mirror 22: 0xee7e1591 (stale) Mirror 23: 0xee7e1591 (stale) Mirror 24: 0xee7e1591 (stale) Mirror 25: 0xee7e1591 (stale) Mirror 26: 0xee7e1591 (stale) Mirror 30: 0xee7e1591 (stale) Mirror 31: 0xee7e1591 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34145): Mirror 21: 0xee7e1591 (stale) Mirror 22: 0xee7e1591 (stale) Mirror 23: 0xee7e1591 (stale) Mirror 24: 0xee7e1591 (stale) Mirror 25: 0xee7e1591 (stale) Mirror 26: 0xee7e1591 (stale) Mirror 30: 0xee7e1591 (stale) Mirror 31: 0xee7e1591 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34145: skipped 000857 1786721105.916157 read 0x023242 thru 0x028074 (0x4e33 bytes) 000858 1786721105.927207 mapwrite 0x029f6b thru 0x039cb5 (0xfd4b bytes) 000859 1786721106.154165 read 0x014b9f thru 0x01cd4d (0x81af bytes) 000860 1786721106.186281 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39cb6): Mirror 21: 0xa578df44 (stale) Mirror 22: 0xa578df44 (stale) Mirror 23: 0xa578df44 (stale) Mirror 25: 0xa578df44 (stale) Mirror 26: 0xa578df44 (stale) Mirror 30: 0xa578df44 (stale) Mirror 31: 0xa578df44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cb6): Mirror 21: 0xa578df44 (stale) Mirror 22: 0xa578df44 (stale) Mirror 23: 0xa578df44 (stale) Mirror 25: 0xa578df44 (stale) Mirror 26: 0xa578df44 (stale) Mirror 30: 0xa578df44 (stale) Mirror 31: 0xa578df44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cb6: skipped 000861 1786721107.237453 read 0x0260b9 thru 0x02b98b (0x58d3 bytes) 000863 1786721107.252376 mapread 0x00a2d5 thru 0x011c4d (0x7979 bytes) 000865 1786721107.321226 write 0x029736 thru 0x030c24 (0x74ef bytes) 000868 1786721107.355446 read 0x00e55a thru 0x0113d0 (0x2e77 bytes) 000870 1786721107.362506 mapread 0x038449 thru 0x039cb5 (0x186d bytes) 000871 1786721107.395587 write 0x0005b8 thru 0x00d18f (0xcbd8 bytes) 000872 1786721107.399034 trunc from 0x039cb6 to 0x034acf (0x51e6 bytes) 000874 1786721107.453731 mapwrite 0x020fdc thru 0x02bff6 (0xb01b bytes) 000875 1786721107.600481 read 0x01e6e5 thru 0x029721 (0xb03d bytes) 000876 1786721107.608998 trunc from 0x034ad0 to 0x000ffc (0x33ad3 bytes) 000880 1786721107.672524 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xffd): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xffd): Mirror 21: 0x1d58a321 (stale) Mirror 22: 0x1d58a321 (stale) Mirror 23: 0x1d58a321 (stale) Mirror 25: 0x1d58a321 (stale) Mirror 26: 0x1d58a321 (stale) Mirror 30: 0x1d58a321 (stale) Mirror 31: 0x1d58a321 (stale) CRC-32 checksum value for chunk [0, 0xffd): Mirror 1: 0x6e8721c1 Mirror 32: 0x6e8721c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xffd: skipped 000883 1786721109.174321 trunc from 0x000ffd to 0x03bd9b (0x3ad9f bytes) 000884 1786721109.333608 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: 0x6523884c (stale) Mirror 22: 0x6523884c (stale) Mirror 23: 0x6523884c (stale) Mirror 25: 0x6523884c (stale) Mirror 26: 0x6523884c (stale) Mirror 30: 0x6523884c (stale) Mirror 31: 0x6523884c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: 0x6523884c (stale) Mirror 22: 0x6523884c (stale) Mirror 23: 0x6523884c (stale) Mirror 25: 0x6523884c (stale) Mirror 26: 0x6523884c (stale) Mirror 30: 0x6523884c (stale) Mirror 31: 0x6523884c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd9c: skipped 000887 1786721110.380085 read 0x0224eb thru 0x02da23 (0xb539 bytes) 000888 1786721110.393621 read 0x022c19 thru 0x02b362 (0x874a bytes) 000889 1786721110.395116 read 0x00e673 thru 0x00f00a (0x998 bytes) 000891 1786721110.410212 mapread 0x017424 thru 0x018576 (0x1153 bytes) 000892 1786721110.412063 write 0x01cc5d thru 0x01f026 (0x23ca bytes) 000893 1786721110.414203 mapread 0x00eea7 thru 0x01a653 (0xb7ad bytes) 000894 1786721110.420133 read 0x01262c thru 0x01589a (0x326f bytes) 000895 1786721110.422251 mapread 0x0073c9 thru 0x00a966 (0x359e bytes) 000896 1786721110.424357 write 0x004367 thru 0x005756 (0x13f0 bytes) 000897 1786721110.426868 mapwrite 0x0170fd thru 0x01a3ac (0x32b0 bytes) 000899 1786721110.523633 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: 0x6523884c (stale) Mirror 22: 0x6523884c (stale) Mirror 23: 0x6523884c (stale) Mirror 25: 0x6523884c (stale) Mirror 26: 0x6523884c (stale) Mirror 30: 0x6523884c (stale) Mirror 31: 0x6523884c (stale) CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 1: 0xa0ed16f Mirror 32: 0xa0ed16f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd9c: skipped 000900 1786721112.581306 mapwrite 0x012d31 thru 0x01f8ca (0xcb9a bytes) 000901 1786721112.974844 read 0x036a1c thru 0x03bd9b (0x5380 bytes) 000902 1786721113.005378 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: 0x6523884c (stale) Mirror 22: 0x6523884c (stale) Mirror 23: 0x6523884c (stale) Mirror 25: 0x6523884c (stale) Mirror 30: 0x6523884c (stale) Mirror 31: 0x6523884c (stale) Mirror 32: 0xa0ed16f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd9c): Mirror 21: 0x6523884c (stale) Mirror 22: 0x6523884c (stale) Mirror 23: 0x6523884c (stale) Mirror 25: 0x6523884c (stale) Mirror 30: 0x6523884c (stale) Mirror 31: 0x6523884c (stale) Mirror 32: 0xa0ed16f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd9c: skipped 000903 1786721113.828589 mapwrite 0x0232e8 thru 0x0299e2 (0x66fb bytes) 000904 1786721113.952334 trunc from 0x03bd9c to 0x01113f (0x2ac5c bytes) 000908 1786721113.996040 write 0x0087e1 thru 0x012537 (0x9d57 bytes) 000910 1786721114.016036 mapwrite 0x01649f thru 0x024dbe (0xe920 bytes) 000911 1786721114.233104 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24dbf): Mirror 21: 0xdc5e1ad (stale) Mirror 22: 0xdc5e1ad (stale) Mirror 23: 0xdc5e1ad (stale) Mirror 25: 0xdc5e1ad (stale) Mirror 30: 0xdc5e1ad (stale) Mirror 31: 0xdc5e1ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24dbf): Mirror 21: 0xdc5e1ad (stale) Mirror 22: 0xdc5e1ad (stale) Mirror 23: 0xdc5e1ad (stale) Mirror 25: 0xdc5e1ad (stale) Mirror 30: 0xdc5e1ad (stale) Mirror 31: 0xdc5e1ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24dbf: skipped 000914 1786721115.151472 mapread 0x00aa65 thru 0x012ccb (0x8267 bytes) 000915 1786721115.176625 mapread 0x00a6d0 thru 0x00ca36 (0x2367 bytes) 000916 1786721115.184383 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24dbf): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24dbf): Mirror 21: 0xdc5e1ad (stale) Mirror 22: 0xdc5e1ad (stale) Mirror 23: 0xdc5e1ad (stale) Mirror 25: 0xdc5e1ad (stale) Mirror 30: 0xdc5e1ad (stale) Mirror 31: 0xdc5e1ad (stale) CRC-32 checksum value for chunk [0, 0x24dbf): Mirror 1: 0xf992b1e5 Mirror 32: 0xf992b1e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24dbf: skipped 000917 1786721117.611696 write 0x0336e5 thru 0x03474d (0x1069 bytes) 000918 1786721117.823521 mapwrite 0x032f5d thru 0x0338a6 (0x94a bytes) 000920 1786721117.990741 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3474e): Mirror 21: 0xef12d40d (stale) Mirror 23: 0xef12d40d (stale) Mirror 25: 0xef12d40d (stale) Mirror 30: 0xef12d40d (stale) Mirror 31: 0xef12d40d (stale) Mirror 32: 0x45318077 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3474e): Mirror 21: 0xef12d40d (stale) Mirror 23: 0xef12d40d (stale) Mirror 25: 0xef12d40d (stale) Mirror 30: 0xef12d40d (stale) Mirror 31: 0xef12d40d (stale) Mirror 32: 0x45318077 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3474e: skipped 000923 1786721118.775523 mapwrite 0x00a78b thru 0x016ea1 (0xc717 bytes) 000924 1786721118.952972 mapwrite 0x015142 thru 0x01bc9c (0x6b5b bytes) 000925 1786721119.024468 write 0x036c40 thru 0x03e544 (0x7905 bytes) 000926 1786721119.035980 read 0x016d22 thru 0x021d2f (0xb00e bytes) 000928 1786721119.062037 mapwrite 0x02edb2 thru 0x03e7bb (0xfa0a bytes) 000930 1786721119.240516 MIRROR_RESYNC 7 mirrors 000932 1786721122.996970 write 0x01d491 thru 0x01e84d (0x13bd bytes) 000934 1786721123.141920 write 0x003db8 thru 0x00eefb (0xb144 bytes) 000936 1786721123.173367 trunc from 0x03e7bc to 0x02d8a6 (0x10f15 bytes) 000937 1786721123.212881 mapread 0x021703 thru 0x02d8a6 (0xc1a4 bytes) 000938 1786721123.281713 mapread 0x024255 thru 0x02afb7 (0x6d63 bytes) 000941 1786721123.292633 mapwrite 0x036f2c thru 0x03ffff (0x90d4 bytes) 000942 1786721123.499171 mapwrite 0x039419 thru 0x0394e9 (0xd1 bytes) 000943 1786721123.615315 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x1ffbb80b (stale) Mirror 23: 0x1ffbb80b (stale) Mirror 25: 0x1ffbb80b (stale) Mirror 30: 0x1ffbb80b (stale) Mirror 31: 0x1ffbb80b (stale) Mirror 32: 0x1ffbb80b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b433b28 Mirror 33: 0x9b433b28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1786721125.577347 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x1ffbb80b (stale) Mirror 23: 0x1ffbb80b (stale) Mirror 25: 0x1ffbb80b (stale) Mirror 30: 0x1ffbb80b (stale) Mirror 31: 0x1ffbb80b (stale) Mirror 32: 0x1ffbb80b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b433b28 Mirror 33: 0x9b433b28 Mirror 34: 0x9b433b28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1786721127.239913 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x1ffbb80b (stale) Mirror 23: 0x1ffbb80b (stale) Mirror 25: 0x1ffbb80b (stale) Mirror 30: 0x1ffbb80b (stale) Mirror 31: 0x1ffbb80b (stale) Mirror 32: 0x1ffbb80b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b433b28 Mirror 33: 0x9b433b28 Mirror 34: 0x9b433b28 Mirror 35: 0x9b433b28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000946 1786721128.765428 trunc from 0x040000 to 0x03bfab (0x4054 bytes) 000948 1786721128.935506 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfac): Mirror 21: 0x2b91d6ba (stale) Mirror 23: 0x2b91d6ba (stale) Mirror 25: 0x2b91d6ba (stale) Mirror 30: 0x2b91d6ba (stale) Mirror 31: 0x2b91d6ba (stale) Mirror 32: 0x2b91d6ba (stale) Mirror 33: 0x70b3141f (stale) Mirror 34: 0x70b3141f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfac): Mirror 21: 0x2b91d6ba (stale) Mirror 23: 0x2b91d6ba (stale) Mirror 25: 0x2b91d6ba (stale) Mirror 30: 0x2b91d6ba (stale) Mirror 31: 0x2b91d6ba (stale) Mirror 32: 0x2b91d6ba (stale) Mirror 33: 0x70b3141f (stale) Mirror 34: 0x70b3141f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfac: skipped 000949 1786721129.821171 mapwrite 0x0216ee thru 0x029a67 (0x837a bytes) 000950 1786721130.021992 mapread 0x01b289 thru 0x025bb0 (0xa928 bytes) 000951 1786721130.035603 mapread 0x022d90 thru 0x02f97e (0xcbef bytes) 000953 1786721130.058543 mapread 0x01f9d2 thru 0x02e7fb (0xee2a bytes) 000954 1786721130.065190 trunc from 0x03bfac to 0x017d67 (0x24244 bytes) 000957 1786721130.119122 mapread 0x002c92 thru 0x00b596 (0x8905 bytes) 000958 1786721130.133087 trunc from 0x017d68 to 0x03d1ee (0x25487 bytes) 000960 1786721130.188640 mapwrite 0x006da5 thru 0x0125ab (0xb807 bytes) 000961 1786721130.312438 read 0x00732a thru 0x0160f0 (0xedc7 bytes) 000962 1786721130.316194 trunc from 0x03d1ef to 0x0230eb (0x1a103 bytes) 000963 1786721130.349224 trunc from 0x0230ec to 0x021bc1 (0x152a bytes) 000964 1786721130.381089 write 0x02b030 thru 0x03a04e (0xf01f bytes) 000965 1786721130.384420 mapread 0x0027e8 thru 0x002b4f (0x368 bytes) 000966 1786721130.386689 mapread 0x01ff58 thru 0x025822 (0x58cb bytes) 000968 1786721130.402901 read 0x02c329 thru 0x02cf23 (0xbfb bytes) 000969 1786721130.412182 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 21: 0x4894628f (stale) Mirror 23: 0x4894628f (stale) Mirror 25: 0x4894628f (stale) Mirror 30: 0x4894628f (stale) Mirror 31: 0x4894628f (stale) Mirror 32: 0x4894628f (stale) Mirror 33: 0x294cfe2b (stale) Mirror 34: 0x294cfe2b (stale) CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 1: 0xabd9515e Mirror 35: 0xabd9515e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a04f: skipped 000970 1786721132.553588 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 21: 0x4894628f (stale) Mirror 23: 0x4894628f (stale) Mirror 25: 0x4894628f (stale) Mirror 30: 0x4894628f (stale) Mirror 31: 0x4894628f (stale) Mirror 32: 0x4894628f (stale) Mirror 33: 0x294cfe2b (stale) Mirror 34: 0x294cfe2b (stale) CRC-32 checksum value for chunk [0, 0x3a04f): Mirror 1: 0xabd9515e Mirror 35: 0xabd9515e Mirror 36: 0xabd9515e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a04f: skipped 000972 1786721134.067231 trunc from 0x03a04f to 0x007da9 (0x322a5 bytes) 000973 1786721134.244225 trunc from 0x007daa to 0x03de4c (0x360a3 bytes) 000974 1786721134.295218 trunc from 0x03de4d to 0x0189da (0x25472 bytes) 000975 1786721134.344029 mapwrite 0x00fc57 thru 0x012552 (0x28fc bytes) 000976 1786721134.507358 mapwrite 0x01365e thru 0x01617f (0x2b22 bytes) 000978 1786721134.590015 read 0x006d5c thru 0x0090d9 (0x237e bytes) 000981 1786721134.604763 mapread 0x008e46 thru 0x009b64 (0xd1f bytes) 000983 1786721134.620435 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 25: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 35: 0x8e94649c (stale) Mirror 36: 0x8e94649c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 25: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 35: 0x8e94649c (stale) Mirror 36: 0x8e94649c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x189db: skipped 000984 1786721135.440959 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 25: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 36: 0x8e94649c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 25: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 36: 0x8e94649c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x189db: skipped 000986 1786721136.049389 mapread 0x015ef1 thru 0x0189da (0x2aea bytes) 000989 1786721136.090099 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 36: 0x8e94649c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x189db): Mirror 21: 0xcc6fdf5e (stale) Mirror 23: 0xcc6fdf5e (stale) Mirror 30: 0xcc6fdf5e (stale) Mirror 31: 0xcc6fdf5e (stale) Mirror 33: 0x3333b287 (stale) Mirror 34: 0x3333b287 (stale) Mirror 36: 0x8e94649c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x189db: skipped 000990 1786721136.842821 read 0x0007b6 thru 0x00194e (0x1199 bytes) 000991 1786721136.854617 mapwrite 0x00c3b5 thru 0x015dbd (0x9a09 bytes) 000994 1786721137.001897 mapwrite 0x019878 thru 0x026ece (0xd657 bytes) 000995 1786721137.157150 write 0x03ec63 thru 0x03ffff (0x139d bytes) 000996 1786721137.158337 read 0x00ca3c thru 0x00f645 (0x2c0a bytes) 000997 1786721137.160978 mapwrite 0x006256 thru 0x00b0e9 (0x4e94 bytes) 000998 1786721137.255698 mapwrite 0x01b4d0 thru 0x01ed82 (0x38b3 bytes) 001000 1786721137.326152 mapread 0x0019a0 thru 0x00525e (0x38bf bytes) All operations completed A-OK! PASS 70a (232s) == sanity-flr test 71: check mirror extend parent fid ==== 11:25:43 (1786721143) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 11:25:47 (1786721147) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.347462 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286166 s, 3.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:25:56 (1786721156) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4773152 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3271872 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3737888 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7355872 .. Extending file size to 8800992 .. Extending file size to 7923072 .. Extending file size to 8173632 .. Extending file size to 8321088 .. Extending file size to 9063968 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8476384 .. Extending file size to 9003552 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57462 57464 57465 57466 57467 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:27:24 (1786721244) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5140640 .. Extending file size to 4552480 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6943936 .. Extending file size to 7327008 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6518400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7863968 .. failed Extending file size to 8143872 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8361568 .. Extending file size to 8638272 .. Extending file size to 8565920 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8913664 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8746400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58994 58995 58997 58998 58999 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:28:46 (1786721326) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175739 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.357492 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xaf0d552b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:29:10 (1786721350) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124172 s, 16.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 11:29:18 (1786721358) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:29:26 (1786721366) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.333941 s, 6.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:29:36 (1786721376) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.262591 s, 8.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:29:44 (1786721384) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267501 s, 7.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:30:19 (1786721419) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160957 s, 6.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } PASS 204d (67s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:31:26 (1786721486) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.435992 s, 4.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:32:33 (1786721553) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. Creating new pool test_204f oleg420-server: Pool lustre.test_204f created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299557 s, 7.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg420-server: Pool lustre.test_204f destroyed PASS 204f (89s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:34:02 (1786721642) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:34:10 (1786721650) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:34:19 (1786721659) oleg420-server: Pool lustre.test_206 created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg420-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:34:45 (1786721685) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828608 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721909 s, 14.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4fcd26c0 Mirror 3: 0x4fcd26c0 PASS 207 (16s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:35:01 (1786721701) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:35:06 (1786721706) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:35:11 (1786721711) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.170047 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1190000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1820000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1820000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1820000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1210000 Read this (4 bytes): BBBB 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:35:51 (1786721751) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0427368 s, 192 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:35:57 (1786721757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759564 s, 13.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x212 lmm_fid: [0x200000404:0x212:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x212 lmm_fid: [0x200000404:0x212:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834450 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 11:35:59.000000000 -0400 Modify: 2026-08-14 11:35:59.000000000 -0400 Change: 2026-08-14 11:36:00.000000000 -0400 Birth: 2026-08-14 11:36:00.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:36:06 (1786721766) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:36:10 (1786721770) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.879815 s, 46.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0891057 s, 46.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 11:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x214:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 11:36 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:36:19 (1786721779) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:36:28 (1786721788) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2011 sec =========== 11:36:38 (1786721798) === sanity-flr: start cleanup 11:36:40 (1786721800) === === sanity-flr: finish cleanup 11:36:47 (1786721807) ===