-----============= acceptance-small: sanity-flr ============----- Tue Aug 18 01:49:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg135-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:49:44 (1787032184) === oleg135-client.virtnet: executing check_config_client /mnt/lustre oleg135-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg135-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff937cc60b7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff937cc60b7800.idle_timeout=debug disable quota as required oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:50:22 (1787032222) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7604] == sanity-flr test 0a: lfs mirror create with -N option == 01:50:24 (1787032224) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:50:44 (1787032244) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:50:48 (1787032248) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:50:51 (1787032251) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:51:19 (1787032279) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:51:24 (1787032284) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:51:27 (1787032287) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff937cc60b7800.stats=clear osc.lustre-OST0001-osc-ffff937cc60b7800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:51:36 (1787032296) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:51:49 (1787032309) PASS 0j (13s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:52:02 (1787032322) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (6s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:52:08 (1787032328) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:52:18 (1787032338) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:52:28 (1787032348) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:52:36 (1787032356) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:52:40 (1787032360) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:52:49 (1787032369) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:53:00 (1787032380) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:53:07 (1787032387) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.89775 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190803 s, 5.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:53:19 (1787032399) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.488275 s, 27.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:53:29 (1787032409) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:53:40 (1787032420) Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:54:54 (1787032494) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.458817 s, 18.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:55:32 (1787032532) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:56:43 (1787032603) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.96077 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 18 01:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.268s user 0m0.003s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.256s user 0m0.000s sys 0m0.198s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.343s user 0m0.002s sys 0m0.200s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937cc60b7800.ost_server_uuid in FULL state after 0 sec PASS 33b (87s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:58:10 (1787032690) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:58:14 (1787032694) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:58:17 (1787032697) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:58:21 (1787032701) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 01:58:29 (1787032709) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.649097 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:58:42 (1787032722) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576379 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029472 s, 35.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:58:50 (1787032730) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:58:59 (1787032739) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00304981 s, 336 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00178304 s, 574 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000807551 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125829 s, 814 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 01:59:09 (1787032749) create test files with size 15760744 .. f2f46938ff2584f60cf93ec5bb59c589 - 786ba844ee4a9c25bc5cd7b6daa84a63 - e4570d47f6bc1cc8866553b2466270d7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 01:59:36 (1787032776) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:59:48 (1787032788) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:59:55 (1787032795) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147541 s, 7.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232385 s, 45.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 02:00:04 (1787032804) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161626 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.484539 s, 8.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195476 s, 10.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 02:00:21 (1787032821) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:00:25 (1787032825) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:00:28 (1787032828) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463911 s, 22.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:00:38 (1787032838) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:00:42 (1787032842) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:00:53 (1787032853) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.27973 s, 37.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:01:06 (1787032866) llite.lustre-ffff937cc9e55800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295401 s, 35.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff937ce0ee6000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:01:19 (1787032879) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:02:01 (1787032921) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:02:40 (1787032960) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:02:44 (1787032964) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299363 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.203907 s, 15.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x526278f2 (nosync,stale) Mirror 2: 0x526278f2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 473b53568d1606be9d210badfc6d2313 - mirror 1 checksum: 0aa7a9c4aa9b0de979ad8f45d2185b9a - mirror 2 checksum: 473b53568d1606be9d210badfc6d2313 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T02:02:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T02:02:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 473b53568d1606be9d210badfc6d2313 - mirror 1 checksum: 0aa7a9c4aa9b0de979ad8f45d2185b9a - mirror 2 checksum: 0aa7a9c4aa9b0de979ad8f45d2185b9a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T02:02:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T02:02:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:02:58 (1787032978) Failing mds1 on oleg135-server Stopping /mnt/lustre-mds1 (opts:) on oleg135-server 02:03:10 (1787032990) shut down facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server Failover mds1 to oleg135-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 02:03:43 (1787033023) targets are mounted 02:03:43 (1787033023) facet_failover done oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03329 s, 31.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:04:01 (1787033041) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:04:04 (1787033044) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:04:08 (1787033048) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00496006 s, 3.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:04:17 (1787033057) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:04:20 (1787033060) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:04:24 (1787033064) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:04:49 (1787033089) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:05:17 (1787033117) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:05:40 (1787033140) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:06:06 (1787033166) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 02:06:13 (1787033173) mirror create pid 53942 mirror split pid 53943 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53942 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53943 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:07:21 (1787033241) Seed set to 3243 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x21d45 000004 1787033243.911573 trunc from 00000000 to 0x021d44 (0x21d45 bytes) 000005 1787033243.932949 mapread 0x016b30 thru 0x01bc03 (0x50d4 bytes) 000008 1787033243.959675 write 0x02baca thru 0x035363 (0x989a bytes) 000009 1787033243.962039 mapread 0x02d647 thru 0x035363 (0x7d1d bytes) truncating to largest ever: 0x3e36e 000010 1787033243.965530 trunc from 0x035364 to 0x03e36d (0x900a bytes) 000011 1787033243.988834 mapwrite 0x00444a thru 0x010c19 (0xc7d0 bytes) 000012 1787033244.122300 mapwrite 0x00d974 thru 0x01cfa1 (0xf62e bytes) 000015 1787033244.170735 write 0x031666 thru 0x03669d (0x5038 bytes) 000020 1787033244.179856 read 0x038629 thru 0x03e36d (0x5d45 bytes) 000023 1787033244.191993 mapread 0x001761 thru 0x0031d9 (0x1a79 bytes) 000029 1787033244.199570 write 0x021d62 thru 0x026edb (0x517a bytes) 000034 1787033244.213870 write 0x0258e7 thru 0x03180f (0xbf29 bytes) 000035 1787033244.218005 mapread 0x03b65f thru 0x03e36d (0x2d0f bytes) 000036 1787033244.219897 write 0x0302a0 thru 0x039d83 (0x9ae4 bytes) 000039 1787033244.226366 mapwrite 0x019bf2 thru 0x01c97e (0x2d8d bytes) 000040 1787033244.276936 read 0x03cbe3 thru 0x03e36d (0x178b bytes) 000043 1787033244.280364 mapread 0x02c2ae thru 0x034953 (0x86a6 bytes) 000045 1787033244.284309 mapread 0x016988 thru 0x0219f0 (0xb069 bytes) 000049 1787033244.289952 mapwrite 0x0289b8 thru 0x0297ec (0xe35 bytes) 000050 1787033244.333325 read 0x013d3a thru 0x015eb3 (0x217a bytes) 000051 1787033244.333927 read 0x001f99 thru 0x002ca9 (0xd11 bytes) 000052 1787033244.334512 mapwrite 0x028d4f thru 0x035b2e (0xcde0 bytes) 000053 1787033244.394899 read 0x031a76 thru 0x03d891 (0xbe1c bytes) 000055 1787033244.397869 mapwrite 0x0301af thru 0x03aa61 (0xa8b3 bytes) 000058 1787033244.455949 mapwrite 0x00a802 thru 0x0177e8 (0xcfe7 bytes) 000059 1787033244.524838 trunc from 0x03e36e to 0x011e30 (0x2c53d bytes) 000061 1787033244.552440 MIRROR_EXTEND to 2 mirrors 000064 1787033245.084076 read 0x004860 thru 0x005d01 (0x14a2 bytes) 000066 1787033245.099825 MIRROR_EXTEND to 3 mirrors 000068 1787033245.729854 mapread 0x007d2a thru 0x011e30 (0xa107 bytes) 000069 1787033245.753005 mapwrite 0x00ff6d thru 0x010c0e (0xca2 bytes) 000072 1787033245.893193 write 0x0289b9 thru 0x02ea26 (0x606e bytes) 000074 1787033245.895595 write 0x002032 thru 0x00432f (0x22fe bytes) 000076 1787033245.908116 write 0x03974a thru 0x03ffff (0x68b6 bytes) 000077 1787033245.910195 read 0x019032 thru 0x01de33 (0x4e02 bytes) 000078 1787033245.918050 mapread 0x00e46a thru 0x017c11 (0x97a8 bytes) 000079 1787033245.940893 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2b7d3370 (stale) Mirror 3: 0x2b7d3370 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43b4a12f Mirror 4: 0x43b4a12f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000082 1787033246.798453 mapwrite 0x02d05b thru 0x0360da (0x9080 bytes) 000083 1787033246.983201 write 0x032d9f thru 0x03fe39 (0xd09b bytes) 000085 1787033246.998842 mapread 0x037a6f thru 0x03cc19 (0x51ab bytes) 000086 1787033247.000066 mapread 0x01a51d thru 0x01eb96 (0x467a bytes) 000087 1787033247.014816 mapread 0x00cc4b thru 0x00e7fa (0x1bb0 bytes) 000088 1787033247.030411 trunc from 0x040000 to 0x00e22f (0x31dd0 bytes) 000089 1787033247.059324 read 0x0031d0 thru 0x00423a (0x106b bytes) 000091 1787033247.069283 trunc from 0x00e230 to 0x024073 (0x15e44 bytes) 000092 1787033247.104424 MIRROR_RESYNC 4 mirrors 000094 1787033247.971432 trunc from 0x024074 to 0x021a85 (0x25ee bytes) 000095 1787033248.064635 mapread 0x002db7 thru 0x011e99 (0xf0e3 bytes) 000096 1787033248.093854 read 0x014345 thru 0x01d0c7 (0x8d83 bytes) 000097 1787033248.103367 mapread 0x0007b4 thru 0x00522d (0x4a7a bytes) 000098 1787033248.109506 write 0x02a28c thru 0x0341cf (0x9f44 bytes) 000101 1787033248.113623 write 0x023e46 thru 0x02890d (0x4ac8 bytes) 000102 1787033248.128303 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x341d0): Mirror 2: 0xc7b3d344 (stale) Mirror 4: 0xc7b3d344 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341d0): Mirror 2: 0xc7b3d344 (stale) Mirror 4: 0xc7b3d344 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341d0: skipped 000103 1787033248.699811 mapread 0x0311be thru 0x0341cf (0x3012 bytes) 000104 1787033248.738670 mapread 0x025836 thru 0x0281ef (0x29ba bytes) 000106 1787033248.763917 mapwrite 0x0326ec thru 0x03e190 (0xbaa5 bytes) 000107 1787033248.871409 trunc from 0x03e191 to 0x00bfd6 (0x321ba bytes) 000109 1787033248.898850 mapread 0x003514 thru 0x0082ee (0x4ddb bytes) 000110 1787033248.913745 trunc from 0x00bfd7 to 0x002849 (0x978d bytes) 000111 1787033248.931042 write 0x032bbe thru 0x037a2f (0x4e72 bytes) 000114 1787033248.935655 write 0x01012a thru 0x011247 (0x111e bytes) 000115 1787033248.953633 MIRROR_RESYNC 3 mirrors 000117 1787033249.798564 trunc from 0x037a30 to 0x01f683 (0x183ac bytes) 000118 1787033249.882282 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f684): Mirror 4: 0x565d80a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f684): Mirror 4: 0x565d80a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f684: skipped 000119 1787033250.186644 read 0x01d661 thru 0x01f683 (0x2023 bytes) 000120 1787033250.196644 mapwrite 0x02b80e thru 0x02f13d (0x3930 bytes) 000121 1787033250.293375 read 0x02e0e1 thru 0x02f13d (0x105d bytes) 000122 1787033250.297973 write 0x018b28 thru 0x01c054 (0x352d bytes) 000124 1787033250.333582 mapwrite 0x0111be thru 0x020dd7 (0xfc1a bytes) 000126 1787033250.442835 write 0x0195d6 thru 0x019636 (0x61 bytes) 000128 1787033250.449452 write 0x00c67b thru 0x019669 (0xcfef bytes) 000130 1787033250.451069 mapwrite 0x02b6d1 thru 0x035655 (0x9f85 bytes) 000133 1787033250.658915 mapwrite 0x00a2fd thru 0x00dde9 (0x3aed bytes) 000134 1787033250.787785 trunc from 0x035656 to 0x00ea3e (0x26c17 bytes) 000136 1787033250.822457 read 0x005e05 thru 0x00bcf4 (0x5ef0 bytes) 000137 1787033250.826371 read 0x00be71 thru 0x00ea3e (0x2bce bytes) 000138 1787033250.829183 trunc from 0x00ea3f to 0x034ae4 (0x260a6 bytes) 000139 1787033250.873765 trunc from 0x034ae5 to 0x031d83 (0x2d61 bytes) 000140 1787033250.893219 mapread 0x02a62c thru 0x031d83 (0x7758 bytes) 000143 1787033250.926053 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31d84): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d84): Mirror 4: 0x6d54b984 (stale) CRC-32 checksum value for chunk [0, 0x31d84): Mirror 1: 0xe83290be Mirror 5: 0xe83290be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d84: skipped 000144 1787033251.996700 mapread 0x015f49 thru 0x01c1bf (0x6277 bytes) 000145 1787033252.031670 mapwrite 0x006c44 thru 0x00ca29 (0x5de6 bytes) 000147 1787033252.201939 mapwrite 0x034f15 thru 0x036728 (0x1814 bytes) 000150 1787033252.289447 read 0x0241ac thru 0x02c12f (0x7f84 bytes) 000151 1787033252.309348 mapwrite 0x01b641 thru 0x023815 (0x81d5 bytes) 000152 1787033252.376103 mapwrite 0x027903 thru 0x031306 (0x9a04 bytes) 000153 1787033252.418225 trunc from 0x036729 to 0x017a1a (0x1ed0e bytes) 000154 1787033252.437277 mapwrite 0x023343 thru 0x02bfc4 (0x8c82 bytes) 000155 1787033252.619934 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfc5): Mirror 5: 0x24e44ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfc5): Mirror 5: 0x24e44ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfc5: skipped 000156 1787033253.039617 mapwrite 0x00f9d0 thru 0x0189e2 (0x9013 bytes) 000157 1787033253.150539 trunc from 0x02bfc5 to 0x01ef68 (0xd05c bytes) 000159 1787033253.175284 mapread 0x0127c4 thru 0x01e91a (0xc157 bytes) 000160 1787033253.185702 read 0x013769 thru 0x018380 (0x4c18 bytes) 000161 1787033253.186652 write 0x029663 thru 0x033393 (0x9d31 bytes) 000162 1787033253.189591 mapread 0x00c8f3 thru 0x011468 (0x4b76 bytes) 000163 1787033253.190987 read 0x02b99c thru 0x033393 (0x79f8 bytes) 000164 1787033253.195632 mapwrite 0x02d74c thru 0x035ee7 (0x879c bytes) 000165 1787033253.339862 trunc from 0x035ee8 to 0x00e99a (0x2754d bytes) 000168 1787033253.364228 trunc from 0x00e99b to 0x01e17a (0xf7e0 bytes) 000172 1787033253.391015 write 0x021db3 thru 0x02bf8a (0xa1d8 bytes) 000174 1787033253.398052 MIRROR_RESYNC 2 mirrors 000176 1787033254.410926 read 0x02b19b thru 0x02bf8a (0xdf0 bytes) 000177 1787033254.417946 trunc from 0x02bf8b to 0x0302df (0x4355 bytes) 000179 1787033254.503058 MIRROR_RESYNC 2 mirrors 000184 1787033255.296286 mapread 0x02a0be thru 0x0302df (0x6222 bytes) 000187 1787033255.324310 write 0x0078ed thru 0x00adcd (0x34e1 bytes) 000188 1787033255.403282 read 0x02b420 thru 0x0302df (0x4ec0 bytes) 000189 1787033255.416310 mapread 0x00ddb1 thru 0x01a8ec (0xcb3c bytes) 000192 1787033255.453101 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x302e0): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302e0): Mirror 5: 0x5d172a6d (stale) CRC-32 checksum value for chunk [0, 0x302e0): Mirror 1: 0xfdde0959 Mirror 6: 0xfdde0959 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302e0: skipped 000193 1787033256.543330 write 0x0243df thru 0x03249a (0xe0bc bytes) 000194 1787033256.793806 read 0x021ef7 thru 0x026dc9 (0x4ed3 bytes) 000195 1787033256.809751 trunc from 0x03249b to 0x009bc5 (0x288d5 bytes) 000196 1787033256.936747 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x9bc6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9bc6): Mirror 5: 0x80e14477 (stale) Mirror 6: 0x3f9f0f7a (stale) CRC-32 checksum value for chunk [0, 0x9bc6): Mirror 1: 0x3f9f0f7a Mirror 7: 0x3f9f0f7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9bc6: skipped 000202 1787033258.574648 mapread 0x0002f0 thru 0x009bc5 (0x98d6 bytes) 000203 1787033258.622403 mapread 0x001d45 thru 0x00688a (0x4b46 bytes) 000205 1787033258.750962 read 0x003b24 thru 0x009bc5 (0x60a2 bytes) 000207 1787033258.777717 write 0x00b0a0 thru 0x019775 (0xe6d6 bytes) 000208 1787033258.926914 trunc from 0x019776 to 0x015c9e (0x3ad7 bytes) 000209 1787033258.961968 mapwrite 0x00e76e thru 0x01de9e (0xf731 bytes) 000211 1787033259.082680 mapwrite 0x02b109 thru 0x036b61 (0xba59 bytes) 000212 1787033259.195964 trunc from 0x036b62 to 0x01a3e7 (0x1c77a bytes) 000214 1787033259.241836 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a3e8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3e8): Mirror 5: 0x2b5cc9d7 (stale) Mirror 6: 0xa093111d (stale) Mirror 7: 0x2be1e912 (stale) CRC-32 checksum value for chunk [0, 0x1a3e8): Mirror 1: 0xcf980573 Mirror 8: 0xcf980573 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3e8: skipped 000216 1787033260.608870 MIRROR_RESYNC 5 mirrors 000218 1787033261.661805 trunc from 0x01a3e8 to 0x0016b7 (0x18d30 bytes) 000219 1787033261.797927 write 0x000d77 thru 0x009538 (0x87c2 bytes) 000223 1787033261.830739 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x9539): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9539): Mirror 5: 0xa6d26c57 (stale) Mirror 6: 0xa6d26c57 (stale) Mirror 7: 0xa6d26c57 (stale) Mirror 8: 0xa6d26c57 (stale) CRC-32 checksum value for chunk [0, 0x9539): Mirror 1: 0x27a0d19b Mirror 9: 0x27a0d19b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9539: skipped 000224 1787033263.398278 mapread 0x00563c thru 0x009538 (0x3efd bytes) 000227 1787033263.464958 mapread 0x009264 thru 0x009538 (0x2d5 bytes) 000230 1787033263.562847 mapwrite 0x031a86 thru 0x038770 (0x6ceb bytes) 000233 1787033263.880731 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38771): Mirror 5: 0xfe2e7d7b (stale) Mirror 6: 0xfe2e7d7b (stale) Mirror 7: 0xfe2e7d7b (stale) Mirror 8: 0xfe2e7d7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38771): Mirror 5: 0xfe2e7d7b (stale) Mirror 6: 0xfe2e7d7b (stale) Mirror 7: 0xfe2e7d7b (stale) Mirror 8: 0xfe2e7d7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38771: skipped 000234 1787033264.467595 mapread 0x023f98 thru 0x025fa8 (0x2011 bytes) 000238 1787033264.510920 write 0x002104 thru 0x010c1c (0xeb19 bytes) 000239 1787033264.593549 mapread 0x01b496 thru 0x029b8b (0xe6f6 bytes) 000240 1787033264.596575 read 0x020870 thru 0x029862 (0x8ff3 bytes) 000242 1787033264.620609 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38771): Mirror 6: 0xfe2e7d7b (stale) Mirror 7: 0xfe2e7d7b (stale) Mirror 8: 0xfe2e7d7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38771): Mirror 6: 0xfe2e7d7b (stale) Mirror 7: 0xfe2e7d7b (stale) Mirror 8: 0xfe2e7d7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38771: skipped 000243 1787033265.276549 mapwrite 0x027721 thru 0x02cce9 (0x55c9 bytes) 000248 1787033265.396808 mapread 0x0052d1 thru 0x00910b (0x3e3b bytes) 000249 1787033265.483757 mapwrite 0x01bd01 thru 0x01e17c (0x247c bytes) 000254 1787033265.585865 mapread 0x00c642 thru 0x01b748 (0xf107 bytes) 000255 1787033265.597799 read 0x004f3c thru 0x00badb (0x6ba0 bytes) 000256 1787033265.603205 trunc from 0x038771 to 0x01e74c (0x1a024 bytes) 000257 1787033265.646632 read 0x00ab14 thru 0x0115ad (0x6a9a bytes) 000258 1787033265.650798 read 0x004a19 thru 0x0107a9 (0xbd91 bytes) 000261 1787033265.654357 read 0x01e0f0 thru 0x01e74c (0x65d bytes) 000264 1787033265.669075 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e74d): Mirror 7: 0x5329e16c (stale) Mirror 8: 0x5329e16c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e74d): Mirror 7: 0x5329e16c (stale) Mirror 8: 0x5329e16c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e74d: skipped 000268 1787033266.145337 mapread 0x002577 thru 0x009e9e (0x7928 bytes) 000269 1787033266.185443 mapread 0x014cec thru 0x01e74c (0x9a61 bytes) 000270 1787033266.212742 write 0x0372d6 thru 0x03ffff (0x8d2a bytes) 000271 1787033266.215568 write 0x032c8c thru 0x0363f6 (0x376b bytes) 000272 1787033266.242713 read 0x03d11b thru 0x03ffff (0x2ee5 bytes) 000274 1787033266.254889 MIRROR_RESYNC 3 mirrors 000275 1787033267.085536 mapread 0x01fe6d thru 0x02dc50 (0xdde4 bytes) 000278 1787033267.132350 MIRROR_RESYNC 3 mirrors 000280 1787033267.857026 mapread 0x025ae1 thru 0x02f3c8 (0x98e8 bytes) 000281 1787033267.858007 trunc from 0x040000 to 0x011302 (0x2ecfd bytes) 000286 1787033267.952530 mapread 0x000ae1 thru 0x003a3e (0x2f5e bytes) 000290 1787033267.972035 write 0x001113 thru 0x00c929 (0xb817 bytes) 000291 1787033267.974568 trunc from 0x011303 to 0x0366c9 (0x253c7 bytes) 000293 1787033268.000298 mapread 0x02d01d thru 0x0366c9 (0x96ad bytes) 000296 1787033268.022735 write 0x03bd85 thru 0x03e919 (0x2b95 bytes) 000298 1787033268.036925 MIRROR_RESYNC 3 mirrors 000300 1787033269.482405 mapwrite 0x0075ac thru 0x00976e (0x21c3 bytes) 000301 1787033269.739481 mapwrite 0x0327fb thru 0x03f2dc (0xcae2 bytes) 000304 1787033269.884958 mapread 0x03348d thru 0x03432a (0xe9e bytes) 000305 1787033269.888962 mapread 0x01ab39 thru 0x02215b (0x7623 bytes) 000306 1787033269.919160 read 0x03cd97 thru 0x03f2dc (0x2546 bytes) 000310 1787033269.925659 MIRROR_RESYNC 3 mirrors 000312 1787033271.395187 trunc from 0x03f2dd to 0x038e09 (0x64d3 bytes) 000314 1787033271.508369 mapread 0x026eca thru 0x034b94 (0xdccb bytes) 000315 1787033271.557838 write 0x00d060 thru 0x015250 (0x81f1 bytes) 000317 1787033271.572561 write 0x007324 thru 0x00ee1a (0x7af7 bytes) 000318 1787033271.580628 write 0x03622e thru 0x037dd0 (0x1ba3 bytes) 000319 1787033271.583501 mapwrite 0x019678 thru 0x022c4c (0x95d5 bytes) 000321 1787033271.656878 trunc from 0x038e0a to 0x03003d (0x8dcc bytes) 000322 1787033271.687936 mapwrite 0x00ffd0 thru 0x01b5a5 (0xb5d6 bytes) 000323 1787033271.753771 trunc from 0x03003e to 0x036c09 (0x6bcc bytes) 000324 1787033271.774709 read 0x02172c thru 0x02fba2 (0xe477 bytes) 000327 1787033271.778660 mapwrite 0x004576 thru 0x0088b0 (0x433b bytes) 000332 1787033271.853715 trunc from 0x036c0a to 0x038dd9 (0x21d0 bytes) 000333 1787033271.882980 mapwrite 0x004c99 thru 0x006d2c (0x2094 bytes) 000334 1787033271.926744 mapwrite 0x0372e4 thru 0x03c522 (0x523f bytes) 000335 1787033272.045981 mapwrite 0x009fd8 thru 0x0192bc (0xf2e5 bytes) 000337 1787033272.132672 trunc from 0x03c523 to 0x010592 (0x2bf90 bytes) 000338 1787033272.166453 write 0x02870a thru 0x0378bd (0xf1b4 bytes) 000339 1787033272.171094 write 0x00bdff thru 0x00dc2d (0x1e2f bytes) 000341 1787033272.173803 read 0x028113 thru 0x0378bd (0xf7ab bytes) 000342 1787033272.175441 write 0x022aa5 thru 0x0318f1 (0xee4d bytes) 000343 1787033272.186782 trunc from 0x0378be to 0x025bdf (0x11cde bytes) 000344 1787033272.212863 mapread 0x014a19 thru 0x020602 (0xbbea bytes) 000346 1787033272.216711 write 0x027dd8 thru 0x02ccdf (0x4f08 bytes) 000347 1787033272.218518 trunc from 0x02cce0 to 0x00d3e4 (0x1f8fb bytes) 000348 1787033272.251772 read 0x00b937 thru 0x00d3e4 (0x1aae bytes) 000350 1787033272.253695 write 0x0109dd thru 0x013952 (0x2f76 bytes) 000352 1787033272.257013 mapread 0x012d08 thru 0x013952 (0xc4b bytes) 000355 1787033272.261613 mapwrite 0x0089cb thru 0x01400a (0xb640 bytes) 000358 1787033272.439436 read 0x0016b5 thru 0x00d6c4 (0xc010 bytes) 000361 1787033272.448294 write 0x0161af thru 0x01d446 (0x7298 bytes) 000364 1787033272.464332 mapread 0x0018e8 thru 0x00f410 (0xdb29 bytes) 000365 1787033272.470560 trunc from 0x01d447 to 0x01eee5 (0x1a9f bytes) 000367 1787033272.512782 trunc from 0x01eee6 to 0x03a632 (0x1b74d bytes) 000368 1787033272.550717 read 0x01eb04 thru 0x026e6d (0x836a bytes) 000369 1787033272.601449 write 0x02babf thru 0x0321ea (0x672c bytes) 000370 1787033272.605406 write 0x03421e thru 0x0361b5 (0x1f98 bytes) 000371 1787033272.624547 MIRROR_RESYNC 3 mirrors 000373 1787033273.722749 trunc from 0x03a633 to 0x02e86e (0xbdc4 bytes) 000376 1787033273.782330 mapwrite 0x013a03 thru 0x01a41a (0x6a18 bytes) 000377 1787033273.874617 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 7: 0x199bb9c (stale) Mirror 8: 0x199bb9c (stale) CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 1: 0xa45095fb Mirror 9: 0xa45095fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e86f: skipped 000378 1787033274.686908 mapread 0x00fab0 thru 0x013d5a (0x42ab bytes) 000381 1787033274.770728 mapread 0x0228cb thru 0x02b1ec (0x8922 bytes) 000384 1787033274.845710 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 7: 0x199bb9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 7: 0x199bb9c (stale) CRC-32 checksum value for chunk [0, 0x2e86f): Mirror 1: 0xa45095fb Mirror 9: 0xa45095fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e86f: skipped 000385 1787033275.542187 read 0x008347 thru 0x016085 (0xdd3f bytes) 000386 1787033275.575707 write 0x01e4f7 thru 0x029d7b (0xb885 bytes) 000387 1787033275.682681 write 0x00211f thru 0x00a718 (0x85fa bytes) 000388 1787033275.699910 write 0x0312dc thru 0x03c97a (0xb69f bytes) 000389 1787033275.701356 mapread 0x029893 thru 0x0392a4 (0xfa12 bytes) 000390 1787033275.725343 mapwrite 0x006a5f thru 0x00d34a (0x68ec bytes) 000391 1787033275.870801 trunc from 0x03c97b to 0x00bd6f (0x30c0b bytes) 000392 1787033275.904271 mapread 0x004240 thru 0x0084be (0x427f bytes) 000395 1787033275.910231 trunc from 0x00bd70 to 0x0333ac (0x2763d bytes) 000397 1787033275.938573 trunc from 0x0333ad to 0x022fea (0x103c2 bytes) 000400 1787033275.977566 mapwrite 0x020ddf thru 0x025398 (0x45ba bytes) 000402 1787033276.087339 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25399): Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25399): Mirror 7: 0x6405fae5 (stale) Mirror 9: 0x474869c0 (stale) CRC-32 checksum value for chunk [0, 0x25399): Mirror 1: 0x25e7a288 Mirror 10: 0x25e7a288 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25399: skipped 000404 1787033277.409231 mapwrite 0x02cf97 thru 0x03a37c (0xd3e6 bytes) 000406 1787033277.727779 write 0x018f2c thru 0x01bbc1 (0x2c96 bytes) 000409 1787033277.763476 mapread 0x022062 thru 0x0257d5 (0x3774 bytes) 000410 1787033277.782767 read 0x016931 thru 0x01b465 (0x4b35 bytes) 000411 1787033277.810806 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: 0x467c7319 (stale) Mirror 10: 0x7401de3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: 0x467c7319 (stale) Mirror 10: 0x7401de3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37d: skipped 000412 1787033278.221712 mapwrite 0x01a96c thru 0x020775 (0x5e0a bytes) 000413 1787033278.357518 write 0x013582 thru 0x01dd17 (0xa796 bytes) 000414 1787033278.371067 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: 0x467c7319 (stale) Mirror 10: 0x7401de3f (stale) CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 1: 0x91045f8 Mirror 11: 0x91045f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37d: skipped 000415 1787033280.312455 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 7: 0x467c7319 (stale) Mirror 10: 0x7401de3f (stale) CRC-32 checksum value for chunk [0, 0x3a37d): Mirror 1: 0x91045f8 Mirror 11: 0x91045f8 Mirror 12: 0x91045f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a37d: skipped 000417 1787033282.031694 write 0x0329df thru 0x03b7db (0x8dfd bytes) 000418 1787033282.316553 trunc from 0x03b7dc to 0x03a67f (0x115c bytes) 000421 1787033282.380704 MIRROR_RESYNC 5 mirrors 000422 1787033283.919295 MIRROR_EXTEND to 6 mirrors 000423 1787033284.906816 mapwrite 0x03e081 thru 0x03ffff (0x1f7f bytes) 000426 1787033285.112711 trunc from 0x040000 to 0x00e6e1 (0x3191e bytes) 000428 1787033285.138934 read 0x00e260 thru 0x00e6e1 (0x482 bytes) 000430 1787033285.146764 read 0x004367 thru 0x00a5cb (0x6265 bytes) 000431 1787033285.162778 mapwrite 0x00042f thru 0x008ca7 (0x8879 bytes) 000432 1787033285.220657 trunc from 0x00e6e2 to 0x02bf39 (0x1d858 bytes) 000433 1787033285.249955 write 0x0237d5 thru 0x02bacc (0x82f8 bytes) 000434 1787033285.261975 mapwrite 0x026b22 thru 0x028950 (0x1e2f bytes) 000435 1787033285.373187 mapread 0x00a683 thru 0x00e45b (0x3dd9 bytes) 000437 1787033285.378692 mapwrite 0x0364d5 thru 0x03ffff (0x9b2b bytes) 000438 1787033285.522062 read 0x00dd41 thru 0x018bcb (0xae8b bytes) 000439 1787033285.523269 write 0x01bdf0 thru 0x0231ea (0x73fb bytes) 000441 1787033285.533173 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6391c3f9 (stale) Mirror 10: 0x6391c3f9 (stale) Mirror 11: 0x6391c3f9 (stale) Mirror 13: 0x6391c3f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6391c3f9 (stale) Mirror 10: 0x6391c3f9 (stale) Mirror 11: 0x6391c3f9 (stale) Mirror 13: 0x6391c3f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000442 1787033286.210425 write 0x03d367 thru 0x03ffff (0x2c99 bytes) 000443 1787033286.231699 trunc from 0x040000 to 0x027b31 (0x184ce bytes) 000445 1787033286.281659 mapread 0x013124 thru 0x01e61e (0xb4fb bytes) 000446 1787033286.359006 mapread 0x00c1f6 thru 0x01b8f3 (0xf6fe bytes) 000448 1787033286.379265 mapread 0x01e35d thru 0x020499 (0x213d bytes) 000455 1787033286.394053 mapwrite 0x02347a thru 0x02d29e (0x9e25 bytes) 000456 1787033286.490750 write 0x03c873 thru 0x03ffff (0x378d bytes) 000458 1787033286.509085 MIRROR_RESYNC 5 mirrors 000460 1787033287.892452 mapwrite 0x008987 thru 0x00f281 (0x68fb bytes) 000464 1787033288.140563 trunc from 0x040000 to 0x0198ba (0x26745 bytes) 000469 1787033288.192985 mapwrite 0x03fd90 thru 0x03ffff (0x270 bytes) 000470 1787033288.263798 mapwrite 0x02e451 thru 0x02f6d1 (0x1281 bytes) 000472 1787033288.340069 write 0x01a60f thru 0x028980 (0xe372 bytes) 000473 1787033288.351505 read 0x01acbc thru 0x01ebfa (0x3f3f bytes) 000476 1787033288.357105 write 0x03c084 thru 0x03ffff (0x3f7c bytes) 000477 1787033288.361721 mapwrite 0x003cb1 thru 0x010c70 (0xcfc0 bytes) 000478 1787033288.432978 read 0x03f153 thru 0x03ffff (0xead bytes) 000480 1787033288.434172 mapwrite 0x035746 thru 0x03d715 (0x7fd0 bytes) 000484 1787033288.485339 mapwrite 0x035bec thru 0x036b04 (0xf19 bytes) 000486 1787033288.511967 read 0x03d389 thru 0x03ffff (0x2c77 bytes) 000487 1787033288.512870 trunc from 0x040000 to 0x01cb23 (0x234dc bytes) 000488 1787033288.548622 MIRROR_RESYNC 5 mirrors 000490 1787033289.335048 mapwrite 0x00e370 thru 0x015705 (0x7396 bytes) 000494 1787033289.541950 read 0x00fa43 thru 0x01cb23 (0xd0e1 bytes) 000495 1787033289.562012 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cb24): Mirror 7: 0xdbd00a55 (stale) Mirror 11: 0xdbd00a55 (stale) Mirror 13: 0xdbd00a55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb24): Mirror 7: 0xdbd00a55 (stale) Mirror 11: 0xdbd00a55 (stale) Mirror 13: 0xdbd00a55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb24: skipped 000496 1787033290.318684 mapwrite 0x03db0a thru 0x03fb99 (0x2090 bytes) 000497 1787033290.400490 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb9a): Mirror 7: 0xad83824f (stale) Mirror 13: 0xad83824f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb9a): Mirror 7: 0xad83824f (stale) Mirror 13: 0xad83824f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb9a: skipped 000498 1787033290.685093 trunc from 0x03fb9a to 0x00c90e (0x3328b bytes) 000499 1787033290.732753 write 0x03e8f8 thru 0x03ffff (0x1708 bytes) 000500 1787033290.743061 mapread 0x0292fc thru 0x02dbdd (0x48e2 bytes) 000501 1787033290.816899 MIRROR_RESYNC 3 mirrors 000503 1787033291.893911 trunc from 0x040000 to 0x03da14 (0x25eb bytes) 000504 1787033291.980098 write 0x018048 thru 0x0184d2 (0x48b bytes) 000505 1787033292.002332 read 0x0165ca thru 0x01692b (0x362 bytes) 000507 1787033292.020718 write 0x01dca2 thru 0x026eab (0x920a bytes) 000508 1787033292.066847 MIRROR_RESYNC 3 mirrors 000509 1787033293.234734 MIRROR_EXTEND to 4 mirrors 000510 1787033293.903872 write 0x00afed thru 0x012970 (0x7984 bytes) 000511 1787033294.040381 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3da15): Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da15): Mirror 7: 0x7525230f (stale) Mirror 13: 0x7525230f (stale) Mirror 14: 0x7525230f (stale) CRC-32 checksum value for chunk [0, 0x3da15): Mirror 1: 0x6b9518fc Mirror 15: 0x6b9518fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da15: skipped 000514 1787033295.286469 write 0x02d8f2 thru 0x031278 (0x3987 bytes) 000515 1787033295.458024 mapwrite 0x0029a6 thru 0x00eadb (0xc136 bytes) 000516 1787033295.549866 trunc from 0x03da15 to 0x00be4b (0x31bc9 bytes) 000517 1787033295.591098 mapread 0x007064 thru 0x00be4b (0x4de8 bytes) 000520 1787033295.639234 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbe4c): Mirror 13: 0x81126b5a (stale) Mirror 14: 0x81126b5a (stale) Mirror 15: 0x6db249d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe4c): Mirror 13: 0x81126b5a (stale) Mirror 14: 0x81126b5a (stale) Mirror 15: 0x6db249d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe4c: skipped 000522 1787033296.061466 write 0x021bda thru 0x02ca97 (0xaebe bytes) 000523 1787033296.065955 mapread 0x012eef thru 0x013987 (0xa99 bytes) 000524 1787033296.080183 write 0x00ab89 thru 0x012a4f (0x7ec7 bytes) 000525 1787033296.087905 read 0x01e369 thru 0x0259e8 (0x7680 bytes) 000527 1787033296.102419 MIRROR_RESYNC 4 mirrors 000530 1787033297.678478 MIRROR_EXTEND to 5 mirrors 000531 1787033298.810619 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000532 1787033299.154944 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000534 1787033299.666070 mapread 0x016fb1 thru 0x01fd55 (0x8da5 bytes) 000535 1787033299.703830 mapread 0x014aea thru 0x021b24 (0xd03b bytes) 000536 1787033299.713246 mapwrite 0x036029 thru 0x03ee53 (0x8e2b bytes) 000539 1787033300.160237 MIRROR_RESYNC 3 mirrors 000540 1787033301.442652 write 0x000335 thru 0x00ca38 (0xc704 bytes) 000542 1787033301.640684 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee54): Mirror 16: 0xd36cb8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee54): Mirror 16: 0xd36cb8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee54: skipped 000544 1787033302.226273 write 0x03c76c thru 0x03ffff (0x3894 bytes) 000547 1787033302.245687 mapread 0x01fe45 thru 0x02f4ee (0xf6aa bytes) 000548 1787033302.293582 trunc from 0x040000 to 0x02bceb (0x14314 bytes) 000551 1787033302.339791 read 0x0012f3 thru 0x007ed5 (0x6be3 bytes) 000556 1787033302.387915 trunc from 0x02bcec to 0x0016d0 (0x2a61b bytes) 000557 1787033302.416415 mapread 0x00028a thru 0x0016d0 (0x1447 bytes) 000558 1787033302.432256 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000559 1787033302.576036 read 0x000c07 thru 0x0016d0 (0xaca bytes) 000560 1787033302.586561 read 0x000d3c thru 0x0016d0 (0x995 bytes) 000561 1787033302.587320 mapwrite 0x01fb79 thru 0x02e702 (0xeb8a bytes) 000563 1787033302.679324 mapwrite 0x01d6a8 thru 0x022364 (0x4cbd bytes) 000565 1787033302.779471 read 0x0202b6 thru 0x02a1d9 (0x9f24 bytes) 000567 1787033302.785561 mapwrite 0x037d90 thru 0x03a54d (0x27be bytes) 000569 1787033302.923673 write 0x032c75 thru 0x033b2b (0xeb7 bytes) 000570 1787033302.933187 MIRROR_EXTEND to 2 mirrors 000572 1787033303.646811 read 0x019df6 thru 0x019f07 (0x112 bytes) 000574 1787033303.671235 mapwrite 0x006b42 thru 0x0077dc (0xc9b bytes) 000576 1787033303.878155 read 0x017e17 thru 0x019bdb (0x1dc5 bytes) 000577 1787033303.889018 read 0x027762 thru 0x0375ab (0xfe4a bytes) 000578 1787033303.890466 write 0x00469e thru 0x0085fe (0x3f61 bytes) 000579 1787033303.893352 mapwrite 0x022879 thru 0x0244eb (0x1c73 bytes) 000581 1787033303.935600 mapwrite 0x01998a thru 0x021225 (0x789c bytes) 000583 1787033304.053604 read 0x006d57 thru 0x012196 (0xb440 bytes) 000584 1787033304.056814 trunc from 0x03a54e to 0x02a986 (0xfbc7 bytes) 000585 1787033304.118683 mapwrite 0x012ba3 thru 0x01d729 (0xab87 bytes) 000586 1787033304.239099 write 0x001486 thru 0x00d5fa (0xc175 bytes) 000587 1787033304.241973 read 0x029af8 thru 0x02a986 (0xe8f bytes) 000591 1787033304.249116 read 0x0007cd thru 0x007d43 (0x7577 bytes) 000592 1787033304.252011 read 0x018f87 thru 0x02619b (0xd215 bytes) 000593 1787033304.252859 mapwrite 0x006138 thru 0x015d96 (0xfc5f bytes) 000594 1787033304.387297 MIRROR_RESYNC 2 mirrors 000595 1787033305.264703 read 0x01f1ff thru 0x02a986 (0xb788 bytes) 000597 1787033305.278771 mapwrite 0x03063b thru 0x038200 (0x7bc6 bytes) 000598 1787033305.624832 write 0x02839d thru 0x032acd (0xa731 bytes) 000600 1787033305.654685 mapwrite 0x01fb08 thru 0x028111 (0x860a bytes) 000601 1787033305.823767 read 0x03410c thru 0x038200 (0x40f5 bytes) 000602 1787033305.825041 trunc from 0x038201 to 0x0329c5 (0x583b bytes) 000603 1787033305.874918 MIRROR_RESYNC 2 mirrors 000605 1787033306.896475 MIRROR_RESYNC 2 mirrors 000607 1787033307.345925 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000610 1787033307.536292 mapread 0x0323a5 thru 0x0329c5 (0x621 bytes) 000611 1787033307.544044 trunc from 0x0329c6 to 0x0031a4 (0x2f821 bytes) 000612 1787033307.584929 mapread 0x00110b thru 0x0031a4 (0x209a bytes) 000613 1787033307.598705 write 0x00fab0 thru 0x01480c (0x4d5d bytes) 000614 1787033307.605952 MIRROR_EXTEND to 2 mirrors 000617 1787033308.402574 mapwrite 0x02838e thru 0x029711 (0x1384 bytes) 000618 1787033308.556758 write 0x00cd72 thru 0x01788d (0xab1c bytes) 000619 1787033308.569029 write 0x01e8eb thru 0x025d94 (0x74aa bytes) 000620 1787033308.583967 write 0x03cf89 thru 0x03ffff (0x3077 bytes) 000621 1787033308.585921 mapwrite 0x01ba1c thru 0x029a3b (0xe020 bytes) 000622 1787033308.709191 MIRROR_RESYNC 2 mirrors 000624 1787033309.342427 mapread 0x028557 thru 0x029eda (0x1984 bytes) 000627 1787033309.355871 mapwrite 0x019555 thru 0x02157e (0x802a bytes) 000630 1787033309.542891 mapwrite 0x00db24 thru 0x019bb0 (0xc08d bytes) 000633 1787033309.619734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x53d89f0c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8498e21f Mirror 3: 0x8498e21f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000634 1787033310.861797 read 0x026ab6 thru 0x0363e9 (0xf934 bytes) 000635 1787033310.879874 trunc from 0x040000 to 0x03d454 (0x2bab bytes) 000636 1787033310.971997 trunc from 0x03d455 to 0x03a0ab (0x33a9 bytes) 000637 1787033311.003900 mapread 0x0108d4 thru 0x01c757 (0xbe84 bytes) 000640 1787033311.045732 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a0ac): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0ac): Mirror 2: 0xc9f31eb9 (stale) Mirror 3: 0x497a714 (stale) CRC-32 checksum value for chunk [0, 0x3a0ac): Mirror 1: 0x497a714 Mirror 4: 0x497a714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0ac: skipped 000641 1787033311.911014 read 0x01eb36 thru 0x0293b2 (0xa87d bytes) 000643 1787033311.937950 mapwrite 0x02932f thru 0x031fc6 (0x8c98 bytes) 000645 1787033312.122517 mapwrite 0x01fecc thru 0x02c8a3 (0xc9d8 bytes) 000647 1787033312.177030 write 0x0171d8 thru 0x01bb71 (0x499a bytes) 000650 1787033312.186003 trunc from 0x03a0ac to 0x024f3a (0x15171 bytes) 000653 1787033312.212913 read 0x006e10 thru 0x016a5c (0xfc4d bytes) 000656 1787033312.228694 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x24f3b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f3b): Mirror 2: 0xde4faf09 (stale) Mirror 3: 0xd990d276 (stale) Mirror 4: 0xd990d276 (stale) CRC-32 checksum value for chunk [0, 0x24f3b): Mirror 1: 0xdeb57e6f Mirror 5: 0xdeb57e6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f3b: skipped 000662 1787033313.053683 mapwrite 0x019613 thru 0x0215cf (0x7fbd bytes) 000663 1787033313.203628 trunc from 0x024f3b to 0x018269 (0xccd1 bytes) 000665 1787033313.254029 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1826a): Mirror 2: 0x192eb4f0 (stale) Mirror 3: 0xd7a695c9 (stale) Mirror 5: 0x810cf1e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1826a): Mirror 2: 0x192eb4f0 (stale) Mirror 3: 0xd7a695c9 (stale) Mirror 5: 0x810cf1e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1826a: skipped 000666 1787033313.610962 write 0x01434a thru 0x01f3f9 (0xb0b0 bytes) 000667 1787033313.621236 write 0x0288ae thru 0x034cc4 (0xc417 bytes) 000668 1787033313.624457 read 0x032b57 thru 0x034cc4 (0x216e bytes) 000670 1787033313.631751 MIRROR_RESYNC 4 mirrors 000671 1787033314.533975 mapwrite 0x0272d6 thru 0x03409d (0xcdc8 bytes) 000672 1787033314.738066 write 0x02e655 thru 0x03c0da (0xda86 bytes) 000674 1787033314.743930 mapwrite 0x036ce9 thru 0x03ffff (0x9317 bytes) 000677 1787033314.850409 trunc from 0x040000 to 0x03ad38 (0x52c7 bytes) 000678 1787033314.878821 MIRROR_RESYNC 4 mirrors 000679 1787033315.869793 MIRROR_EXTEND to 5 mirrors 000686 1787033316.771708 write 0x002abe thru 0x007f1d (0x5460 bytes) 000687 1787033316.861163 write 0x03dd4d thru 0x03ffff (0x22b3 bytes) 000688 1787033316.862300 write 0x034f71 thru 0x03d625 (0x86b5 bytes) 000689 1787033316.875819 read 0x038b9d thru 0x03ffff (0x7463 bytes) 000690 1787033316.876943 mapread 0x03259c thru 0x033609 (0x106e bytes) 000691 1787033316.890950 trunc from 0x040000 to 0x004f98 (0x3b067 bytes) 000697 1787033316.930206 write 0x006463 thru 0x0126c7 (0xc265 bytes) 000700 1787033316.947636 MIRROR_RESYNC 5 mirrors 000703 1787033318.107069 write 0x03c96a thru 0x03e3d8 (0x1a6f bytes) 000704 1787033318.208056 mapwrite 0x037f3c thru 0x03c553 (0x4618 bytes) 000705 1787033318.327888 write 0x00bd48 thru 0x00df20 (0x21d9 bytes) 000706 1787033318.355580 mapwrite 0x00cb44 thru 0x016251 (0x970e bytes) 000707 1787033318.517766 write 0x0294bb thru 0x02e55c (0x50a2 bytes) 000708 1787033318.529028 mapread 0x01012e thru 0x019d3e (0x9c11 bytes) 000710 1787033318.556912 mapread 0x00be4d thru 0x0143e8 (0x859c bytes) 000712 1787033318.560012 trunc from 0x03e3d9 to 0x010ec9 (0x2d50f bytes) 000713 1787033318.603180 read 0x00df89 thru 0x00e9ec (0xa64 bytes) 000714 1787033318.603907 mapwrite 0x010ffe thru 0x017c7c (0x6c7f bytes) 000715 1787033318.717413 write 0x02dbf3 thru 0x0349d7 (0x6de5 bytes) 000716 1787033318.720917 mapread 0x02552a thru 0x0311dd (0xbcb4 bytes) 000719 1787033318.733829 trunc from 0x0349d8 to 0x0274db (0xd4fc bytes) 000720 1787033318.758501 write 0x02e416 thru 0x0354c4 (0x70af bytes) 000723 1787033318.764210 read 0x02c2d2 thru 0x0354c4 (0x91f3 bytes) 000724 1787033318.776200 MIRROR_RESYNC 5 mirrors 000726 1787033319.718113 read 0x01b027 thru 0x01c164 (0x113e bytes) 000729 1787033319.727488 mapread 0x017805 thru 0x01c413 (0x4c0f bytes) 000731 1787033319.757387 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000732 1787033320.179083 mapwrite 0x015a75 thru 0x01f458 (0x99e4 bytes) 000733 1787033320.397693 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: 0x6d5b977d (stale) Mirror 3: 0x6d5b977d (stale) Mirror 5: 0x6d5b977d (stale) CRC-32 checksum value for chunk [0, 0x354c5): Mirror 1: 0x479c70ac Mirror 6: 0x479c70ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354c5: skipped 000736 1787033321.372668 read 0x0160a9 thru 0x023690 (0xd5e8 bytes) 000737 1787033321.399029 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: 0x6d5b977d (stale) Mirror 3: 0x6d5b977d (stale) Mirror 5: 0x6d5b977d (stale) CRC-32 checksum value for chunk [0, 0x354c5): Mirror 1: 0x479c70ac Mirror 6: 0x479c70ac Mirror 7: 0x479c70ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354c5: skipped 000738 1787033322.281901 write 0x026e35 thru 0x0322cb (0xb497 bytes) 000739 1787033322.394512 mapread 0x026c81 thru 0x033e5f (0xd1df bytes) 000740 1787033322.418687 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354c5): Mirror 2: 0x6d5b977d (stale) Mirror 3: 0x6d5b977d (stale) Mirror 5: 0x6d5b977d (stale) Mirror 6: 0x479c70ac (stale) Mirror 7: 0x479c70ac (stale) CRC-32 checksum value for chunk [0, 0x354c5): Mirror 1: 0xe5d67f79 Mirror 8: 0xe5d67f79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354c5: skipped 000742 1787033323.347441 write 0x032646 thru 0x0350b9 (0x2a74 bytes) 000743 1787033323.447811 read 0x035109 thru 0x0354c4 (0x3bc bytes) 000748 1787033323.452546 mapread 0x0202c0 thru 0x0203e3 (0x124 bytes) 000749 1787033323.457983 write 0x022008 thru 0x028570 (0x6569 bytes) 000751 1787033323.468527 read 0x02ec75 thru 0x0354c4 (0x6850 bytes) 000752 1787033323.477949 mapwrite 0x00c323 thru 0x01297c (0x665a bytes) 000756 1787033323.527907 write 0x03cb6f thru 0x03ffff (0x3491 bytes) 000757 1787033323.529423 trunc from 0x040000 to 0x02d5dd (0x12a22 bytes) 000759 1787033323.582904 mapwrite 0x0095d2 thru 0x00daca (0x44f9 bytes) 000762 1787033323.724498 write 0x0265ca thru 0x02819a (0x1bd1 bytes) 000764 1787033323.729804 trunc from 0x02d5de to 0x039574 (0xbf97 bytes) 000765 1787033323.774545 mapwrite 0x032e20 thru 0x03a9b0 (0x7b91 bytes) 000767 1787033323.876851 trunc from 0x03a9b1 to 0x02c1e6 (0xe7ca bytes) 000769 1787033323.905660 write 0x0383c6 thru 0x03ac04 (0x283f bytes) 000770 1787033323.906819 trunc from 0x03ac05 to 0x032ab8 (0x814c bytes) 000771 1787033323.930866 write 0x01f00d thru 0x02cf83 (0xdf77 bytes) 000772 1787033323.933384 mapread 0x024d9a thru 0x02ba4c (0x6cb3 bytes) 000774 1787033323.941415 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32ab9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ab9): Mirror 2: 0x19955ef8 (stale) Mirror 3: 0x19955ef8 (stale) Mirror 5: 0x19955ef8 (stale) Mirror 6: 0xaa78ca2 (stale) Mirror 7: 0xaa78ca2 (stale) Mirror 8: 0x526e7c14 (stale) CRC-32 checksum value for chunk [0, 0x32ab9): Mirror 1: 0x423ddad Mirror 9: 0x423ddad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ab9: skipped 000778 1787033325.126992 write 0x0157a5 thru 0x01f005 (0x9861 bytes) 000782 1787033325.256829 mapread 0x003205 thru 0x004403 (0x11ff bytes) 000783 1787033325.268630 mapwrite 0x0106a9 thru 0x0113f6 (0xd4e bytes) 000785 1787033325.437479 MIRROR_RESYNC 8 mirrors 000786 1787033327.451696 mapwrite 0x0003a9 thru 0x0031f3 (0x2e4b bytes) 000789 1787033327.853206 mapread 0x025315 thru 0x02d73f (0x842b bytes) 000791 1787033327.882051 trunc from 0x032ab9 to 0x03ba70 (0x8fb8 bytes) 000792 1787033327.937410 write 0x03a05e thru 0x03ffff (0x5fa2 bytes) 000797 1787033327.989902 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc23e78aa (stale) Mirror 3: 0xc23e78aa (stale) Mirror 5: 0xc23e78aa (stale) Mirror 6: 0xc23e78aa (stale) Mirror 7: 0xc23e78aa (stale) Mirror 8: 0xc23e78aa (stale) Mirror 9: 0xc23e78aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf13803ad Mirror 10: 0xf13803ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000798 1787033329.228680 mapread 0x020dc5 thru 0x028c61 (0x7e9d bytes) 000799 1787033329.298817 mapread 0x003b39 thru 0x005f4a (0x2412 bytes) 000800 1787033329.355048 trunc from 0x040000 to 0x011f36 (0x2e0c9 bytes) 000801 1787033329.517339 mapread 0x00e69f thru 0x0109d3 (0x2335 bytes) 000802 1787033329.527953 mapwrite 0x0147dc thru 0x014b81 (0x3a6 bytes) 000804 1787033329.693755 write 0x011031 thru 0x017bff (0x6bcf bytes) 000805 1787033329.705576 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x17c00): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17c00): Mirror 2: 0x673eb409 (stale) Mirror 3: 0x673eb409 (stale) Mirror 5: 0x673eb409 (stale) Mirror 6: 0x673eb409 (stale) Mirror 7: 0x673eb409 (stale) Mirror 8: 0x673eb409 (stale) Mirror 9: 0x673eb409 (stale) Mirror 10: 0xa05c5e24 (stale) CRC-32 checksum value for chunk [0, 0x17c00): Mirror 1: 0xe3fa887b Mirror 11: 0xe3fa887b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17c00: skipped 000807 1787033331.118719 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x17c00): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17c00): Mirror 2: 0x673eb409 (stale) Mirror 3: 0x673eb409 (stale) Mirror 5: 0x673eb409 (stale) Mirror 6: 0x673eb409 (stale) Mirror 7: 0x673eb409 (stale) Mirror 8: 0x673eb409 (stale) Mirror 9: 0x673eb409 (stale) Mirror 10: 0xa05c5e24 (stale) CRC-32 checksum value for chunk [0, 0x17c00): Mirror 1: 0xe3fa887b Mirror 11: 0xe3fa887b Mirror 12: 0xe3fa887b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17c00: skipped 000808 1787033332.130684 mapwrite 0x009ed9 thru 0x01403d (0xa165 bytes) 000811 1787033332.376919 mapwrite 0x02940d thru 0x02b290 (0x1e84 bytes) 000812 1787033332.493289 write 0x001d9d thru 0x00b7a9 (0x9a0d bytes) 000815 1787033332.509284 mapwrite 0x001441 thru 0x006529 (0x50e9 bytes) 000816 1787033332.657793 write 0x013c8d thru 0x0214ea (0xd85e bytes) 000817 1787033332.680665 MIRROR_RESYNC 11 mirrors 000820 1787033334.514243 read 0x003022 thru 0x00a509 (0x74e8 bytes) 000822 1787033334.528093 read 0x028492 thru 0x02b290 (0x2dff bytes) 000823 1787033334.545159 write 0x03aaa3 thru 0x03eeb6 (0x4414 bytes) 000825 1787033334.626701 trunc from 0x03eeb7 to 0x011440 (0x2da76 bytes) 000826 1787033334.650847 mapread 0x00e9b7 thru 0x011440 (0x2a8a bytes) 000827 1787033334.675703 trunc from 0x011441 to 0x02ef9b (0x1db5b bytes) 000829 1787033334.702807 write 0x00e170 thru 0x015d34 (0x7bc5 bytes) 000830 1787033334.714893 mapwrite 0x034303 thru 0x03a107 (0x5e05 bytes) 000834 1787033334.885285 write 0x032791 thru 0x03f2b7 (0xcb27 bytes) 000835 1787033334.898756 MIRROR_RESYNC 11 mirrors 000837 1787033336.568301 trunc from 0x03f2b8 to 0x026df6 (0x184c1 bytes) 000840 1787033336.653341 mapread 0x0198b5 thru 0x0259e6 (0xc132 bytes) 000842 1787033336.683612 read 0x024a54 thru 0x026df6 (0x23a3 bytes) 000843 1787033336.685400 trunc from 0x026df7 to 0x001520 (0x258d6 bytes) 000844 1787033336.704991 read 0x0013dd thru 0x001520 (0x144 bytes) 000845 1787033336.713332 read 0x0001da thru 0x001520 (0x1347 bytes) 000846 1787033336.714320 mapread 0x000830 thru 0x001520 (0xcf1 bytes) 000847 1787033336.721687 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1521): Mirror 2: 0xff02e784 (stale) Mirror 3: 0xff02e784 (stale) Mirror 6: 0xff02e784 (stale) Mirror 7: 0xff02e784 (stale) Mirror 8: 0xff02e784 (stale) Mirror 9: 0xff02e784 (stale) Mirror 10: 0xff02e784 (stale) Mirror 11: 0xff02e784 (stale) Mirror 12: 0xff02e784 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1521): Mirror 2: 0xff02e784 (stale) Mirror 3: 0xff02e784 (stale) Mirror 6: 0xff02e784 (stale) Mirror 7: 0xff02e784 (stale) Mirror 8: 0xff02e784 (stale) Mirror 9: 0xff02e784 (stale) Mirror 10: 0xff02e784 (stale) Mirror 11: 0xff02e784 (stale) Mirror 12: 0xff02e784 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1521: skipped 000849 1787033337.198838 mapread 0x00019b thru 0x001520 (0x1386 bytes) 000850 1787033337.225754 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1521): Mirror 3: 0xff02e784 (stale) Mirror 6: 0xff02e784 (stale) Mirror 7: 0xff02e784 (stale) Mirror 8: 0xff02e784 (stale) Mirror 9: 0xff02e784 (stale) Mirror 10: 0xff02e784 (stale) Mirror 11: 0xff02e784 (stale) Mirror 12: 0xff02e784 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1521): Mirror 3: 0xff02e784 (stale) Mirror 6: 0xff02e784 (stale) Mirror 7: 0xff02e784 (stale) Mirror 8: 0xff02e784 (stale) Mirror 9: 0xff02e784 (stale) Mirror 10: 0xff02e784 (stale) Mirror 11: 0xff02e784 (stale) Mirror 12: 0xff02e784 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1521: skipped 000851 1787033337.600062 mapread 0x0012c0 thru 0x001520 (0x261 bytes) 000854 1787033337.609413 mapread 0x0014e6 thru 0x001520 (0x3b bytes) 000855 1787033337.611372 read 0x000207 thru 0x001520 (0x131a bytes) 000858 1787033337.626661 write 0x0376fd thru 0x03ffff (0x8903 bytes) 000861 1787033337.635729 mapread 0x02146c thru 0x024238 (0x2dcd bytes) 000862 1787033337.672548 write 0x00eecb thru 0x016c45 (0x7d7b bytes) 000866 1787033337.717847 mapwrite 0x027787 thru 0x02b41a (0x3c94 bytes) 000867 1787033337.866784 trunc from 0x040000 to 0x00de72 (0x3218d bytes) 000871 1787033337.918762 write 0x035693 thru 0x03a67f (0x4fed bytes) 000873 1787033337.923735 trunc from 0x03a680 to 0x026b52 (0x13b2d bytes) 000874 1787033337.964249 read 0x00ca22 thru 0x01852e (0xbb0d bytes) 000876 1787033338.003751 trunc from 0x026b53 to 0x022120 (0x4a32 bytes) 000878 1787033338.044084 trunc from 0x022121 to 0x02b65d (0x953d bytes) 000880 1787033338.080229 trunc from 0x02b65e to 0x0240a5 (0x75b8 bytes) 000881 1787033338.106812 read 0x022b3d thru 0x0240a5 (0x1569 bytes) 000882 1787033338.118803 trunc from 0x0240a6 to 0x01aab6 (0x95ef bytes) 000884 1787033338.143263 mapread 0x00a213 thru 0x012154 (0x7f42 bytes) 000886 1787033338.147355 mapread 0x01636f thru 0x0194d2 (0x3164 bytes) 000887 1787033338.152820 MIRROR_RESYNC 9 mirrors 000888 1787033339.738080 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) 000891 1787033340.171950 trunc from 0x01aab7 to 0x020c2c (0x6176 bytes) 000892 1787033340.287761 mapread 0x005c97 thru 0x01191f (0xbc89 bytes) 000894 1787033340.333255 mapwrite 0x016fc1 thru 0x01ed05 (0x7d45 bytes) 000895 1787033340.403911 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x20c2d): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c2d): Mirror 3: 0x9c7b6f79 (stale) Mirror 6: 0x9c7b6f79 (stale) Mirror 7: 0x9c7b6f79 (stale) Mirror 9: 0x9c7b6f79 (stale) Mirror 10: 0x9c7b6f79 (stale) Mirror 11: 0x9c7b6f79 (stale) Mirror 12: 0x9c7b6f79 (stale) CRC-32 checksum value for chunk [0, 0x20c2d): Mirror 1: 0x91e9d13d Mirror 13: 0x91e9d13d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c2d: skipped 000896 1787033341.291252 mapread 0x0078dc thru 0x0089ba (0x10df bytes) 000897 1787033341.320773 MIRROR_RESYNC 9 mirrors 000900 1787033342.888383 MIRROR_RESYNC 9 mirrors 000901 1787033343.364567 trunc from 0x020c2d to 0x00010c (0x20b20 bytes) 000902 1787033343.488512 mapwrite 0x019f19 thru 0x019f63 (0x4b bytes) 000905 1787033343.593962 mapwrite 0x01ddf4 thru 0x02a18a (0xc397 bytes) 000906 1787033343.730436 read 0x00d40a thru 0x016c74 (0x986b bytes) 000907 1787033343.744976 read 0x01d86b thru 0x01e9b1 (0x1147 bytes) 000908 1787033343.746218 mapwrite 0x001339 thru 0x00d55f (0xc227 bytes) 000909 1787033343.822089 write 0x016067 thru 0x0221fc (0xc196 bytes) 000910 1787033343.824516 read 0x01bc82 thru 0x027a61 (0xbde0 bytes) 000912 1787033343.826111 mapread 0x006d95 thru 0x007ed1 (0x113d bytes) 000913 1787033343.828324 write 0x029965 thru 0x033ac0 (0xa15c bytes) 000915 1787033343.834300 mapread 0x0100d5 thru 0x01ce03 (0xcd2f bytes) 000916 1787033343.836202 trunc from 0x033ac1 to 0x00529f (0x2e821 bytes) 000919 1787033343.865480 mapwrite 0x01c651 thru 0x01f01f (0x29cf bytes) 000920 1787033343.949490 mapread 0x01b72f thru 0x01f01f (0x38f1 bytes) 000921 1787033343.951937 write 0x037fec thru 0x03f63d (0x7652 bytes) 000922 1787033343.955411 mapread 0x0390f1 thru 0x03bf44 (0x2e54 bytes) 000923 1787033343.957547 trunc from 0x03f63e to 0x018c17 (0x26a26 bytes) 000924 1787033343.981177 trunc from 0x018c18 to 0x018890 (0x387 bytes) 000925 1787033344.009061 mapread 0x002dc2 thru 0x00c302 (0x9541 bytes) 000927 1787033344.010647 mapwrite 0x01874b thru 0x0213e4 (0x8c9a bytes) 000928 1787033344.133262 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x213e5): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x213e5): Mirror 3: 0x7ef2f427 (stale) Mirror 6: 0x7ef2f427 (stale) Mirror 7: 0x7ef2f427 (stale) Mirror 9: 0x7ef2f427 (stale) Mirror 10: 0x7ef2f427 (stale) Mirror 11: 0x7ef2f427 (stale) Mirror 12: 0x7ef2f427 (stale) Mirror 13: 0x7ef2f427 (stale) CRC-32 checksum value for chunk [0, 0x213e5): Mirror 1: 0x8aba3e92 Mirror 14: 0x8aba3e92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x213e5: skipped 000929 1787033345.124905 mapwrite 0x00503b thru 0x006b94 (0x1b5a bytes) 000930 1787033345.358481 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x213e5): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x213e5): Mirror 3: 0x7ef2f427 (stale) Mirror 6: 0x7ef2f427 (stale) Mirror 7: 0x7ef2f427 (stale) Mirror 9: 0x7ef2f427 (stale) Mirror 10: 0x7ef2f427 (stale) Mirror 11: 0x7ef2f427 (stale) Mirror 12: 0x7ef2f427 (stale) Mirror 13: 0x7ef2f427 (stale) Mirror 14: 0x8aba3e92 (stale) CRC-32 checksum value for chunk [0, 0x213e5): Mirror 1: 0x8ac0944d Mirror 15: 0x8ac0944d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x213e5: skipped 000931 1787033346.462646 write 0x030062 thru 0x032774 (0x2713 bytes) 000932 1787033346.536538 mapread 0x009a8f thru 0x009dd7 (0x349 bytes) 000933 1787033346.543794 trunc from 0x032775 to 0x01c349 (0x1642b bytes) 000934 1787033346.566618 mapread 0x0166da thru 0x019fe6 (0x390d bytes) 000935 1787033346.576653 mapwrite 0x026c32 thru 0x02b285 (0x4654 bytes) 000936 1787033346.622423 mapwrite 0x021a71 thru 0x02c7b9 (0xad49 bytes) 000938 1787033346.717392 mapwrite 0x038816 thru 0x0399d3 (0x11be bytes) 000945 1787033346.796389 mapread 0x03104a thru 0x034fbd (0x3f74 bytes) 000946 1787033346.798997 write 0x01f09d thru 0x02e932 (0xf896 bytes) 000949 1787033346.807472 read 0x01b4e7 thru 0x020261 (0x4d7b bytes) 000950 1787033346.817639 MIRROR_RESYNC 11 mirrors 000951 1787033348.358666 read 0x0188e6 thru 0x01aff8 (0x2713 bytes) 000954 1787033348.365627 trunc from 0x0399d4 to 0x00bd97 (0x2dc3c bytes) 000955 1787033348.457320 write 0x02e7ba thru 0x02ed30 (0x577 bytes) 000956 1787033348.460982 write 0x00a766 thru 0x00e99f (0x423a bytes) 000957 1787033348.490443 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2ed31): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed31): Mirror 3: 0x7dc59a2e (stale) Mirror 6: 0x7dc59a2e (stale) Mirror 7: 0x7dc59a2e (stale) Mirror 9: 0x7dc59a2e (stale) Mirror 10: 0x7dc59a2e (stale) Mirror 11: 0x7dc59a2e (stale) Mirror 12: 0x7dc59a2e (stale) Mirror 13: 0x7dc59a2e (stale) Mirror 14: 0x7dc59a2e (stale) Mirror 15: 0x7dc59a2e (stale) CRC-32 checksum value for chunk [0, 0x2ed31): Mirror 1: 0xd0f65c6a Mirror 16: 0xd0f65c6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed31: skipped 000958 1787033349.955922 trunc from 0x02ed31 to 0x029640 (0x56f0 bytes) 000959 1787033350.139067 write 0x01d6ad thru 0x028691 (0xafe5 bytes) 000960 1787033350.159874 trunc from 0x029641 to 0x01ccc0 (0xc980 bytes) 000963 1787033350.217871 mapread 0x017010 thru 0x01ccc0 (0x5cb1 bytes) 000965 1787033350.262093 read 0x0004e8 thru 0x00c4c7 (0xbfe0 bytes) 000966 1787033350.284726 write 0x0028d9 thru 0x008f89 (0x66b1 bytes) 000968 1787033350.297688 write 0x026539 thru 0x030af6 (0xa5be bytes) 000969 1787033350.304381 mapwrite 0x02e2f2 thru 0x03e036 (0xfd45 bytes) 000971 1787033350.513095 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3e037): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e037): Mirror 3: 0xc5be9023 (stale) Mirror 6: 0xc5be9023 (stale) Mirror 7: 0xc5be9023 (stale) Mirror 9: 0xc5be9023 (stale) Mirror 10: 0xc5be9023 (stale) Mirror 11: 0xc5be9023 (stale) Mirror 12: 0xc5be9023 (stale) Mirror 13: 0xc5be9023 (stale) Mirror 14: 0xc5be9023 (stale) Mirror 15: 0xc5be9023 (stale) Mirror 16: 0xdc6cede (stale) CRC-32 checksum value for chunk [0, 0x3e037): Mirror 1: 0xe00ac437 Mirror 17: 0xe00ac437 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e037: skipped 000972 1787033351.461854 mapread 0x035467 thru 0x03ccd5 (0x786f bytes) 000974 1787033351.511996 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3e037): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e037): Mirror 3: 0xc5be9023 (stale) Mirror 6: 0xc5be9023 (stale) Mirror 7: 0xc5be9023 (stale) Mirror 9: 0xc5be9023 (stale) Mirror 10: 0xc5be9023 (stale) Mirror 11: 0xc5be9023 (stale) Mirror 12: 0xc5be9023 (stale) Mirror 13: 0xc5be9023 (stale) Mirror 14: 0xc5be9023 (stale) Mirror 15: 0xc5be9023 (stale) Mirror 16: 0xdc6cede (stale) CRC-32 checksum value for chunk [0, 0x3e037): Mirror 1: 0xe00ac437 Mirror 17: 0xe00ac437 Mirror 18: 0xe00ac437 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e037: skipped 000979 1787033352.621445 mapread 0x028c32 thru 0x02fd73 (0x7142 bytes) 000980 1787033352.644357 mapwrite 0x039281 thru 0x03ffff (0x6d7f bytes) 000981 1787033352.825408 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb56a7775 (stale) Mirror 6: 0xb56a7775 (stale) Mirror 7: 0xb56a7775 (stale) Mirror 9: 0xb56a7775 (stale) Mirror 10: 0xb56a7775 (stale) Mirror 11: 0xb56a7775 (stale) Mirror 12: 0xb56a7775 (stale) Mirror 13: 0xb56a7775 (stale) Mirror 14: 0xb56a7775 (stale) Mirror 15: 0xb56a7775 (stale) Mirror 16: 0xa814b534 (stale) Mirror 17: 0xfe34614 (stale) Mirror 18: 0xfe34614 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9d7028a5 Mirror 19: 0x9d7028a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1787033353.971507 mapread 0x02505e thru 0x027551 (0x24f4 bytes) 000985 1787033354.033107 mapwrite 0x01ba75 thru 0x01f72f (0x3cbb bytes) 000986 1787033354.267043 write 0x00ca3d thru 0x013805 (0x6dc9 bytes) 000987 1787033354.280323 mapread 0x01692c thru 0x01eec3 (0x8598 bytes) 000988 1787033354.286864 trunc from 0x040000 to 0x01d3ea (0x22c15 bytes) 000989 1787033354.334228 trunc from 0x01d3eb to 0x03630e (0x18f24 bytes) 000994 1787033354.370245 mapread 0x02ea93 thru 0x0352f1 (0x685f bytes) 000995 1787033354.401590 write 0x0167d5 thru 0x025d89 (0xf5b5 bytes) 000997 1787033354.412090 trunc from 0x03630f to 0x012a41 (0x238cd bytes) 000998 1787033354.448471 read 0x00c557 thru 0x00fbaf (0x3659 bytes) 001000 1787033354.451303 write 0x030f28 thru 0x032722 (0x17fb bytes) All operations completed A-OK! PASS 70a (118s) == sanity-flr test 71: check mirror extend parent fid ==== 02:09:20 (1787033360) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 02:09:23 (1787033363) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.380004 s, 5.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129159 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 02:09:33 (1787033373) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8170720 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8473376 .. Extending file size to 8907040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8839968 .. Extending file size to 9063520 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9156128 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57036 57037 57039 57040 57042 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) 192.168.201.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:10:57 (1787033457) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7713600 .. Extending file size to 6814208 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7941824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7898720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8366880 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9112064 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8549344 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8669600 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8863136 .. Waiting 58526 58527 58529 58531 58532 done 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) 192.168.201.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:12:16 (1787033536) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162954 s, 12.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274296 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x863cf0d9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:12:34 (1787033554) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0586413 s, 35.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:12:41 (1787033561) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:12:48 (1787033568) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199413 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:12:54 (1787033574) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.226876 s, 9.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:13:01 (1787033581) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253473 s, 8.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:13:33 (1787033613) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195552 s, 5.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:14:34 (1787033674) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153025 s, 13.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:15:35 (1787033735) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Creating new pool test_204f oleg135-server: Pool lustre.test_204f created Adding targets to pool oleg135-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg135-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21107 s, 9.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg135-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg135-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg135-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:16:47 (1787033807) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:16:54 (1787033814) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:17:00 (1787033820) oleg135-server: Pool lustre.test_206 created Adding targets to pool oleg135-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg135-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg135-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg135-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg135-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:17:22 (1787033842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555948 s, 18.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493538 s, 21.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd841cb26 Mirror 3: 0xd841cb26 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:17:31 (1787033851) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:17:34 (1787033854) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:17:37 (1787033857) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0901648 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2140000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1da0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1da0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1da0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16d0000 Read this (4 bytes): BBBB 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:18:11 (1787033891) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0250691 s, 327 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:18:16 (1787033896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472324 s, 22.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834433 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-18 02:18:18.000000000 -0400 Modify: 2026-08-18 02:18:18.000000000 -0400 Change: 2026-08-18 02:18:19.000000000 -0400 Birth: 2026-08-18 02:18:17.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:18:23 (1787033903) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:18:25 (1787033905) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.591628 s, 69.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0701056 s, 58.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 18 02:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x203:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 18 02:18 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:18:33 (1787033913) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:18:40 (1787033920) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1762 sec =========== 02:18:47 (1787033927) === sanity-flr: start cleanup 02:18:48 (1787033928) === === sanity-flr: finish cleanup 02:18:53 (1787033933) ===