-----============= acceptance-small: sanity-flr ============----- Sat Aug 8 06:46:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg211-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:46:45 (1786186005) === oleg211-client.virtnet: executing check_config_client /mnt/lustre oleg211-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg211-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff914b51a89800.idle_timeout=debug osc.lustre-OST0001-osc-ffff914b51a89800.idle_timeout=debug disable quota as required oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:47:26 (1786186046) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7618] == sanity-flr test 0a: lfs mirror create with -N option == 06:47:29 (1786186049) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:47:48 (1786186068) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:47:52 (1786186072) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:47:56 (1786186076) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:48:18 (1786186098) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:48:21 (1786186101) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:48:24 (1786186104) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff914b51a89800.stats=clear osc.lustre-OST0001-osc-ffff914b51a89800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:48:32 (1786186112) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:48:45 (1786186125) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:48:56 (1786186136) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:49:00 (1786186140) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:49:07 (1786186147) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 06:49:14 (1786186154) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:49:22 (1786186162) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:49:26 (1786186166) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:49:35 (1786186175) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:49:48 (1786186188) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:49:56 (1786186196) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.07316 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281392 s, 3.7 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:50:07 (1786186207) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.346751 s, 33.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:50:16 (1786186216) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:50:27 (1786186227) Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:51:42 (1786186302) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.466399 s, 18.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:52:23 (1786186343) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:53:42 (1786186422) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.39167 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 8 06:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.100s user 0m0.002s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.881s user 0m0.002s sys 0m0.179s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.927s user 0m0.001s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg211-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff914b51a89800.ost_server_uuid in FULL state after 0 sec PASS 33b (82s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:55:04 (1786186504) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:55:07 (1786186507) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:55:11 (1786186511) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:55:15 (1786186515) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 06:55:24 (1786186524) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.5679 s, 22.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:55:37 (1786186537) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477284 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683571 s, 15.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:55:44 (1786186544) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:55:53 (1786186553) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00168823 s, 607 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000801196 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00076967 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000822647 s, 1.2 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:56:04 (1786186564) create test files with size 15749347 .. 5437d82747390c1064e5e4c742cc15f3 - 6d1dff76e5926c74ce4c423b08053f36 - f8fb8fe7a8343b039c70019b2eeb6c9b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 06:56:26 (1786186586) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:56:41 (1786186601) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:56:48 (1786186608) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147032 s, 7.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052199 s, 20.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:56:58 (1786186618) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.099735 s, 21.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.500453 s, 8.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221489 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:57:15 (1786186635) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:57:18 (1786186638) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:57:21 (1786186641) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578706 s, 18.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:57:30 (1786186650) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:57:34 (1786186654) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:57:45 (1786186665) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455861 s, 23.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:57:58 (1786186678) llite.lustre-ffff914b43c5a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.329431 s, 31.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff914b52b7e000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:58:11 (1786186691) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:58:51 (1786186731) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:59:37 (1786186777) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:59:41 (1786186781) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.274516 s, 11.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.204718 s, 15.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x188f8bb2 (nosync,stale) Mirror 2: 0x188f8bb2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b7d32cdf4e94e4afc1d83d431154e072 - mirror 1 checksum: c69bc4d285752e0bb30530c922da62ea - mirror 2 checksum: b7d32cdf4e94e4afc1d83d431154e072 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T06:59:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T06:59:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b7d32cdf4e94e4afc1d83d431154e072 - mirror 1 checksum: c69bc4d285752e0bb30530c922da62ea - mirror 2 checksum: c69bc4d285752e0bb30530c922da62ea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T06:59:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T06:59:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:59:55 (1786186795) Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server 07:00:02 (1786186802) shut down facet: mds1 facet_host: oleg211-server facet_failover_host: oleg211-server Failover mds1 to oleg211-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 07:00:33 (1786186833) targets are mounted 07:00:33 (1786186833) facet_failover done oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613384 s, 17.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:00:55 (1786186855) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:01:00 (1786186860) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:01:05 (1786186865) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0149779 s, 1.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:01:14 (1786186874) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:01:18 (1786186878) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00900271 s, 455 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.205863 s, 19.9 kB/s PASS 60a (13s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:01:31 (1786186891) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:02:01 (1786186921) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:02:28 (1786186948) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:02:53 (1786186973) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:03:17 (1786186997) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 07:03:24 (1786187004) mirror create pid 54323 mirror split pid 54324 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54323 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54324 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 07:04:33 (1786187073) Seed set to 7075 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1786187075.792612 mapwrite 0x03c71d thru 0x03ffff (0x38e3 bytes) 000004 1786187075.848940 read 0x03270d thru 0x03ffff (0xd8f3 bytes) 000005 1786187075.856615 mapread 0x022622 thru 0x02871b (0x60fa bytes) 000006 1786187075.864672 mapwrite 0x028a55 thru 0x02dc40 (0x51ec bytes) 000007 1786187075.902289 mapread 0x024e82 thru 0x032c98 (0xde17 bytes) 000011 1786187075.906034 write 0x03d10f thru 0x03ffff (0x2ef1 bytes) 000012 1786187075.906914 mapwrite 0x03bb64 thru 0x03ffff (0x449c bytes) 000013 1786187075.942099 mapwrite 0x030a39 thru 0x03ffff (0xf5c7 bytes) 000014 1786187075.977701 write 0x006e38 thru 0x014aea (0xdcb3 bytes) 000016 1786187075.987470 write 0x03085b thru 0x036375 (0x5b1b bytes) 000018 1786187075.988806 mapwrite 0x036068 thru 0x03ffff (0x9f98 bytes) 000019 1786187076.028626 mapread 0x01c2db thru 0x027aab (0xb7d1 bytes) truncating to largest ever: 0xb932 000020 1786187076.029665 trunc from 0x040000 to 0x00b931 (0x346ce bytes) 000021 1786187076.046378 write 0x028691 thru 0x02d8ea (0x525a bytes) 000022 1786187076.048391 read 0x01900d thru 0x01ece2 (0x5cd6 bytes) 000023 1786187076.054350 mapread 0x013dc7 thru 0x02360e (0xf848 bytes) 000024 1786187076.056901 mapread 0x017339 thru 0x01cad0 (0x5798 bytes) truncating to largest ever: 0x21551 000028 1786187076.059068 trunc from 0x02d8eb to 0x021550 (0xc39a bytes) truncating to largest ever: 0x31748 000029 1786187076.070655 trunc from 0x021551 to 0x031747 (0x101f7 bytes) 000030 1786187076.087646 trunc from 0x031748 to 0x007e43 (0x29904 bytes) 000031 1786187076.117261 mapwrite 0x0225c0 thru 0x024c57 (0x2698 bytes) 000032 1786187076.216711 trunc from 0x024c58 to 0x0268b5 (0x1c5e bytes) 000035 1786187076.241597 MIRROR_EXTEND to 2 mirrors 000036 1786187077.240168 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000037 1786187077.440876 MIRROR_EXTEND to 2 mirrors 000041 1786187078.479082 mapread 0x01fbff thru 0x0268b5 (0x6cb7 bytes) 000043 1786187078.510305 mapread 0x025bf7 thru 0x0268b5 (0xcbf bytes) 000045 1786187078.513047 write 0x03b990 thru 0x03e2ca (0x293b bytes) 000047 1786187078.622932 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000048 1786187078.871337 mapread 0x00fbe5 thru 0x015199 (0x55b5 bytes) 000051 1786187078.929541 read 0x0038df thru 0x00fc0e (0xc330 bytes) 000053 1786187078.955160 mapwrite 0x03bc76 thru 0x03ffff (0x438a bytes) 000054 1786187079.163169 mapread 0x02d415 thru 0x03a123 (0xcd0f bytes) 000055 1786187079.195821 read 0x01878b thru 0x01fc29 (0x749f bytes) 000058 1786187079.236172 mapwrite 0x03e840 thru 0x03ffff (0x17c0 bytes) 000060 1786187079.323108 mapwrite 0x03de7f thru 0x03ffff (0x2181 bytes) 000061 1786187079.457551 read 0x00aad1 thru 0x00eb7b (0x40ab bytes) 000064 1786187079.461224 mapwrite 0x03c6d6 thru 0x03ffff (0x392a bytes) 000066 1786187079.586561 mapread 0x034fc1 thru 0x03ffff (0xb03f bytes) 000069 1786187079.597385 read 0x011f6e thru 0x01853d (0x65d0 bytes) 000070 1786187079.599133 trunc from 0x040000 to 0x022002 (0x1dffd bytes) 000071 1786187079.636355 mapwrite 0x020e74 thru 0x022a98 (0x1c25 bytes) 000074 1786187079.795254 read 0x01d889 thru 0x022a98 (0x5210 bytes) 000075 1786187079.795968 write 0x02b1d8 thru 0x02c81e (0x1647 bytes) 000082 1786187079.809734 write 0x011eb8 thru 0x016d56 (0x4e9f bytes) 000085 1786187079.812747 mapwrite 0x038b41 thru 0x03ffff (0x74bf bytes) 000086 1786187080.056857 MIRROR_EXTEND to 2 mirrors 000087 1786187081.164950 trunc from 0x040000 to 0x00d290 (0x32d6f bytes) 000088 1786187081.285659 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000090 1786187081.474347 read 0x00bff6 thru 0x00d290 (0x129b bytes) 000091 1786187081.482033 write 0x0168e1 thru 0x025d59 (0xf479 bytes) 000094 1786187081.489413 read 0x01b63d thru 0x01f51b (0x3edf bytes) 000095 1786187081.490536 trunc from 0x025d5a to 0x010177 (0x15be2 bytes) 000096 1786187081.520908 read 0x00881b thru 0x00c8b7 (0x409d bytes) 000097 1786187081.531341 read 0x00b1f4 thru 0x00b486 (0x293 bytes) truncating to largest ever: 0x35079 000098 1786187081.534082 trunc from 0x010178 to 0x035078 (0x24f01 bytes) 000100 1786187081.556059 read 0x00534c thru 0x00f02d (0x9ce2 bytes) 000101 1786187081.562221 MIRROR_EXTEND to 2 mirrors 000102 1786187082.386707 read 0x023adc thru 0x029d10 (0x6235 bytes) 000103 1786187082.391274 write 0x03ef46 thru 0x03ffff (0x10ba bytes) 000105 1786187082.447382 mapread 0x012b7c thru 0x013a71 (0xef6 bytes) 000106 1786187082.461241 read 0x0006e7 thru 0x00b442 (0xad5c bytes) 000107 1786187082.475915 trunc from 0x040000 to 0x027222 (0x18ddd bytes) 000108 1786187082.508227 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27223): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27223): Mirror 2: 0xecc99b55 (stale) CRC-32 checksum value for chunk [0, 0x27223): Mirror 1: 0xecc99b55 Mirror 3: 0xecc99b55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27223: skipped 000109 1786187083.874879 mapread 0x022bff thru 0x027222 (0x4624 bytes) 000110 1786187083.899992 read 0x0141f0 thru 0x024192 (0xffa3 bytes) 000112 1786187083.911317 mapread 0x019d7e thru 0x0250bb (0xb33e bytes) 000113 1786187083.912444 mapwrite 0x033a26 thru 0x03ffff (0xc5da bytes) 000115 1786187084.131021 mapread 0x01ba19 thru 0x023f16 (0x84fe bytes) 000117 1786187084.149818 mapwrite 0x03f767 thru 0x03ffff (0x899 bytes) 000118 1786187084.194361 trunc from 0x040000 to 0x00dc93 (0x3236c bytes) 000119 1786187084.235459 mapwrite 0x021967 thru 0x02acf1 (0x938b bytes) 000121 1786187084.386325 mapread 0x0109b6 thru 0x010a41 (0x8c bytes) 000123 1786187084.400316 read 0x0210e7 thru 0x02a08c (0x8fa6 bytes) 000126 1786187084.402818 mapread 0x00a0f1 thru 0x01697e (0xc88e bytes) 000129 1786187084.405835 write 0x022b65 thru 0x027a0c (0x4ea8 bytes) 000130 1786187084.410933 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2acf2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acf2): Mirror 2: 0xc73fb743 (stale) Mirror 3: 0xc73fb743 (stale) CRC-32 checksum value for chunk [0, 0x2acf2): Mirror 1: 0x377cde70 Mirror 4: 0x377cde70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acf2: skipped 000133 1786187085.295270 write 0x022b5a thru 0x026840 (0x3ce7 bytes) 000134 1786187085.347367 read 0x004a51 thru 0x00c430 (0x79e0 bytes) 000135 1786187085.356609 mapwrite 0x01256d thru 0x01c070 (0x9b04 bytes) 000137 1786187085.431447 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2acf2): Mirror 3: 0xc73fb743 (stale) Mirror 4: 0x377cde70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acf2): Mirror 3: 0xc73fb743 (stale) Mirror 4: 0x377cde70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acf2: skipped 000139 1786187085.995874 write 0x006f75 thru 0x00845b (0x14e7 bytes) truncating to largest ever: 0x35565 000141 1786187086.024190 trunc from 0x02acf2 to 0x035564 (0xa873 bytes) 000143 1786187086.078926 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35565): Mirror 4: 0x9f89849d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35565): Mirror 4: 0x9f89849d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35565: skipped 000144 1786187086.361189 read 0x031c61 thru 0x035564 (0x3904 bytes) truncating to largest ever: 0x3f9b1 000147 1786187086.376415 trunc from 0x035565 to 0x03f9b0 (0xa44c bytes) 000148 1786187086.396210 mapread 0x01b173 thru 0x01decb (0x2d59 bytes) 000149 1786187086.449392 MIRROR_RESYNC 2 mirrors 000154 1786187087.348016 trunc from 0x03f9b1 to 0x01f172 (0x2083e bytes) 000157 1786187087.436889 MIRROR_RESYNC 2 mirrors 000159 1786187088.422178 mapwrite 0x022aa5 thru 0x029c13 (0x716f bytes) 000160 1786187088.730357 read 0x022238 thru 0x029c13 (0x79dc bytes) 000161 1786187088.731220 trunc from 0x029c14 to 0x01f1d3 (0xaa40 bytes) 000163 1786187088.786960 mapread 0x012ab0 thru 0x01b592 (0x8ae3 bytes) 000166 1786187088.827870 read 0x0129ee thru 0x016e11 (0x4424 bytes) 000168 1786187088.829097 mapwrite 0x017afc thru 0x018df4 (0x12f9 bytes) 000170 1786187088.977394 write 0x002f32 thru 0x00f132 (0xc201 bytes) 000171 1786187089.003605 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000172 1786187089.299210 mapread 0x005a05 thru 0x00fcd6 (0xa2d2 bytes) 000173 1786187089.341873 write 0x0082eb thru 0x00ca60 (0x4776 bytes) 000174 1786187089.343049 trunc from 0x01f1d4 to 0x01e170 (0x1063 bytes) 000176 1786187089.386349 write 0x03fd22 thru 0x03ffff (0x2de bytes) 000177 1786187089.391205 mapwrite 0x008abd thru 0x0172c9 (0xe80d bytes) 000178 1786187089.483416 write 0x035768 thru 0x03ffff (0xa898 bytes) 000179 1786187089.491152 write 0x02e1aa thru 0x03df97 (0xfdee bytes) 000180 1786187089.498840 mapwrite 0x037ac4 thru 0x03acfb (0x3238 bytes) 000182 1786187089.600832 trunc from 0x040000 to 0x02365a (0x1c9a5 bytes) 000185 1786187089.627741 write 0x005498 thru 0x015418 (0xff81 bytes) 000186 1786187089.646605 trunc from 0x02365b to 0x013efa (0xf760 bytes) 000190 1786187089.692318 trunc from 0x013efb to 0x01362d (0x8cd bytes) 000194 1786187089.743386 read 0x00c5eb thru 0x00f7e3 (0x31f9 bytes) 000198 1786187089.765644 read 0x012010 thru 0x01362d (0x161e bytes) 000200 1786187089.772636 mapread 0x000b56 thru 0x00c97a (0xbe25 bytes) 000203 1786187089.795526 read 0x009e7e thru 0x00b967 (0x1aea bytes) 000204 1786187089.800480 mapread 0x002c72 thru 0x0091a6 (0x6535 bytes) 000205 1786187089.814985 mapread 0x0070b9 thru 0x011a86 (0xa9ce bytes) 000206 1786187089.827407 write 0x03e2c0 thru 0x03ffff (0x1d40 bytes) 000208 1786187089.857002 mapwrite 0x03e6fa thru 0x03ffff (0x1906 bytes) 000209 1786187089.998815 trunc from 0x040000 to 0x03e2e4 (0x1d1b bytes) 000213 1786187090.063421 mapread 0x0397ba thru 0x03e2e4 (0x4b2b bytes) 000214 1786187090.104683 trunc from 0x03e2e5 to 0x025e7f (0x18465 bytes) 000215 1786187090.159773 mapwrite 0x02a648 thru 0x02d5b8 (0x2f71 bytes) 000216 1786187090.368476 mapread 0x01c482 thru 0x02723c (0xadbb bytes) 000217 1786187090.400068 read 0x011f09 thru 0x014f3b (0x3033 bytes) 000220 1786187090.416773 read 0x00e4b6 thru 0x01ba89 (0xd5d4 bytes) 000223 1786187090.425013 mapwrite 0x03adbd thru 0x03ffff (0x5243 bytes) 000227 1786187090.582159 read 0x013f22 thru 0x01bd66 (0x7e45 bytes) 000228 1786187090.588988 read 0x01aa42 thru 0x025ec7 (0xb486 bytes) 000230 1786187090.598908 mapread 0x02b94e thru 0x038a10 (0xd0c3 bytes) 000233 1786187090.611887 mapread 0x0040b4 thru 0x0083e2 (0x432f bytes) 000234 1786187090.617333 write 0x008084 thru 0x0082b1 (0x22e bytes) 000236 1786187090.626530 mapread 0x020c20 thru 0x02fcbc (0xf09d bytes) 000238 1786187090.631429 write 0x02e614 thru 0x031cbb (0x36a8 bytes) 000239 1786187090.646091 read 0x004c9f thru 0x012c54 (0xdfb6 bytes) 000241 1786187090.670579 MIRROR_EXTEND to 2 mirrors 000242 1786187092.335584 read 0x02e159 thru 0x0315d6 (0x347e bytes) 000244 1786187092.342062 write 0x03046b thru 0x038c06 (0x879c bytes) 000245 1786187092.462791 write 0x03df4d thru 0x03ffff (0x20b3 bytes) 000246 1786187092.478871 read 0x01f270 thru 0x028793 (0x9524 bytes) 000247 1786187092.492974 mapread 0x01a3d9 thru 0x01b618 (0x1240 bytes) 000249 1786187092.496050 mapwrite 0x01740b thru 0x01b574 (0x416a bytes) 000250 1786187092.584212 trunc from 0x040000 to 0x0329ef (0xd610 bytes) 000253 1786187092.612305 trunc from 0x0329f0 to 0x013650 (0x1f39f bytes) 000254 1786187092.637067 MIRROR_RESYNC 2 mirrors 000255 1786187093.251100 trunc from 0x013651 to 0x008f0d (0xa743 bytes) 000256 1786187093.327712 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000257 1786187093.489887 mapread 0x006650 thru 0x008f0d (0x28be bytes) 000258 1786187093.512825 trunc from 0x008f0e to 0x02a25f (0x21352 bytes) 000259 1786187093.527972 mapwrite 0x033802 thru 0x03de9d (0xa69c bytes) 000260 1786187093.606451 trunc from 0x03de9e to 0x0001fe (0x3dc9f bytes) 000261 1786187093.619126 write 0x002174 thru 0x007b64 (0x59f1 bytes) 000262 1786187093.620549 mapwrite 0x000e8a thru 0x010814 (0xf98b bytes) 000263 1786187093.690671 read 0x00f10d thru 0x00f7c3 (0x6b7 bytes) 000264 1786187093.691699 write 0x01180f thru 0x011c42 (0x434 bytes) 000267 1786187093.694885 write 0x020af2 thru 0x0238eb (0x2dfa bytes) 000268 1786187093.696659 trunc from 0x0238ec to 0x02a269 (0x697e bytes) 000270 1786187093.734040 mapread 0x004bf8 thru 0x011d5b (0xd164 bytes) 000271 1786187093.743099 MIRROR_EXTEND to 2 mirrors 000273 1786187094.662629 trunc from 0x02a26a to 0x037731 (0xd4c8 bytes) 000274 1786187094.735424 trunc from 0x037732 to 0x03a7d8 (0x30a7 bytes) 000275 1786187094.753912 mapread 0x037dc3 thru 0x03a7d8 (0x2a16 bytes) 000276 1786187094.780130 read 0x00f5b5 thru 0x01106e (0x1aba bytes) 000278 1786187094.787275 read 0x001290 thru 0x0033da (0x214b bytes) 000280 1786187094.798834 trunc from 0x03a7d9 to 0x02adde (0xf9fa bytes) 000282 1786187094.819725 write 0x000ea5 thru 0x005a0f (0x4b6b bytes) 000283 1786187094.821348 mapread 0x01ef49 thru 0x02adde (0xbe96 bytes) 000284 1786187094.823786 write 0x02a871 thru 0x034d89 (0xa519 bytes) 000288 1786187094.829393 write 0x02d812 thru 0x034c16 (0x7405 bytes) 000290 1786187094.832458 trunc from 0x034d8a to 0x01a2ea (0x1aa9f bytes) 000292 1786187094.854733 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1a2eb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a2eb): Mirror 2: 0x9e970c2e (stale) CRC-32 checksum value for chunk [0, 0x1a2eb): Mirror 1: 0x5e2fc015 Mirror 3: 0x5e2fc015 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a2eb: skipped 000295 1786187096.109899 mapread 0x013fcd thru 0x01a2ea (0x631e bytes) 000297 1786187096.141363 MIRROR_RESYNC 3 mirrors 000300 1786187097.045863 mapread 0x00cc86 thru 0x0168ba (0x9c35 bytes) 000302 1786187097.061143 trunc from 0x01a2eb to 0x008f9b (0x1134f bytes) 000304 1786187097.197728 trunc from 0x008f9c to 0x019739 (0x1079e bytes) 000305 1786187097.221731 mapread 0x006d93 thru 0x01554c (0xe7ba bytes) 000306 1786187097.241155 mapwrite 0x02417f thru 0x0316d5 (0xd557 bytes) 000307 1786187097.331007 mapwrite 0x0129c0 thru 0x01d3fe (0xaa3f bytes) 000308 1786187097.392562 mapwrite 0x009dcd thru 0x014d16 (0xaf4a bytes) 000309 1786187097.448612 write 0x03a0fb thru 0x03d6c7 (0x35cd bytes) 000310 1786187097.449980 read 0x007fb1 thru 0x01654d (0xe59d bytes) 000312 1786187097.453450 write 0x03def3 thru 0x03ffff (0x210d bytes) 000313 1786187097.460075 write 0x030533 thru 0x0339e5 (0x34b3 bytes) 000314 1786187097.473825 write 0x0309b7 thru 0x03c31f (0xb969 bytes) 000316 1786187097.484912 MIRROR_RESYNC 3 mirrors 000317 1786187098.647511 trunc from 0x040000 to 0x037b3c (0x84c3 bytes) 000319 1786187098.802309 read 0x00ca35 thru 0x00feca (0x3496 bytes) 000320 1786187098.809857 mapwrite 0x001788 thru 0x007aa9 (0x6322 bytes) 000322 1786187099.059302 mapwrite 0x01af47 thru 0x01b443 (0x4fd bytes) 000325 1786187099.305665 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37b3d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b3d): Mirror 2: 0xabe83c1b (stale) Mirror 3: 0xabe83c1b (stale) CRC-32 checksum value for chunk [0, 0x37b3d): Mirror 1: 0x29e3cd29 Mirror 4: 0x29e3cd29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b3d: skipped 000326 1786187101.452099 write 0x03b891 thru 0x03ffff (0x476f bytes) 000329 1786187101.594363 read 0x01f00d thru 0x01f690 (0x684 bytes) 000330 1786187101.604223 mapwrite 0x02693b thru 0x028c7b (0x2341 bytes) 000334 1786187101.801220 mapread 0x02079d thru 0x02fcb9 (0xf51d bytes) 000335 1786187101.817629 mapwrite 0x03cb66 thru 0x03ffff (0x349a bytes) 000337 1786187101.952817 write 0x023b81 thru 0x02f982 (0xbe02 bytes) 000339 1786187101.960480 read 0x014a76 thru 0x01a1fd (0x5788 bytes) 000340 1786187101.985818 read 0x01de42 thru 0x02b452 (0xd611 bytes) 000341 1786187101.989477 mapread 0x0070cb thru 0x00cb7c (0x5ab2 bytes) 000344 1786187102.004415 mapread 0x00979a thru 0x012743 (0x8faa bytes) 000347 1786187102.013125 mapread 0x01b2c5 thru 0x021267 (0x5fa3 bytes) 000348 1786187102.020987 trunc from 0x040000 to 0x0107a1 (0x2f85e bytes) 000351 1786187102.079346 trunc from 0x0107a2 to 0x03eadd (0x2e33c bytes) 000352 1786187102.108453 mapread 0x01ec06 thru 0x0273a0 (0x879b bytes) 000354 1786187102.140868 trunc from 0x03eade to 0x0086ba (0x36423 bytes) 000355 1786187102.191929 trunc from 0x0086bb to 0x00efbc (0x6902 bytes) 000357 1786187102.220737 mapwrite 0x00df95 thru 0x017a9c (0x9b08 bytes) 000358 1786187102.417220 read 0x014a7d thru 0x017a9c (0x3020 bytes) 000359 1786187102.419370 read 0x017404 thru 0x017a9c (0x699 bytes) 000361 1786187102.421714 trunc from 0x017a9d to 0x03e633 (0x26b97 bytes) 000362 1786187102.442059 mapwrite 0x033e35 thru 0x03ff43 (0xc10f bytes) 000363 1786187102.537401 read 0x00a16e thru 0x00a4bb (0x34e bytes) 000364 1786187102.538467 trunc from 0x03ff44 to 0x02ae63 (0x150e0 bytes) 000366 1786187102.567209 MIRROR_RESYNC 4 mirrors 000367 1786187103.754231 read 0x015baa thru 0x016d41 (0x1198 bytes) 000368 1786187103.760358 trunc from 0x02ae64 to 0x037f91 (0xd12e bytes) 000369 1786187103.838554 mapread 0x008c56 thru 0x017435 (0xe7e0 bytes) 000370 1786187103.885319 write 0x0278f1 thru 0x036d82 (0xf492 bytes) 000371 1786187103.922964 mapread 0x00b5c6 thru 0x00e745 (0x3180 bytes) 000372 1786187103.924596 mapread 0x026170 thru 0x02eb3e (0x89cf bytes) 000377 1786187103.959313 trunc from 0x037f92 to 0x0061ad (0x31de4 bytes) 000378 1786187103.995127 trunc from 0x0061ae to 0x00d236 (0x7089 bytes) 000380 1786187104.039671 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd237): Mirror 2: 0xc80fdb98 (stale) Mirror 3: 0xc80fdb98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd237): Mirror 2: 0xc80fdb98 (stale) Mirror 3: 0xc80fdb98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd237: skipped 000382 1786187104.546102 mapwrite 0x034d0f thru 0x035095 (0x387 bytes) 000387 1786187104.672863 mapwrite 0x02db2d thru 0x032540 (0x4a14 bytes) 000388 1786187104.758432 mapread 0x0236e8 thru 0x0255d0 (0x1ee9 bytes) 000390 1786187104.768319 mapread 0x033159 thru 0x035095 (0x1f3d bytes) 000391 1786187104.769406 write 0x0151be thru 0x02251c (0xd35f bytes) 000392 1786187104.776883 write 0x01a288 thru 0x023ffc (0x9d75 bytes) 000394 1786187104.785654 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35096): Mirror 3: 0x987c3c52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35096): Mirror 3: 0x987c3c52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35096: skipped 000396 1786187105.127702 read 0x023880 thru 0x02d0a7 (0x9828 bytes) 000397 1786187105.139932 mapread 0x00400a thru 0x01049f (0xc496 bytes) 000398 1786187105.185651 mapwrite 0x013618 thru 0x016a33 (0x341c bytes) 000399 1786187105.291178 MIRROR_RESYNC 2 mirrors 000400 1786187106.321583 trunc from 0x035096 to 0x01f04a (0x1604b bytes) 000401 1786187106.387183 mapread 0x0107b9 thru 0x01b9b4 (0xb1fc bytes) 000402 1786187106.417377 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f04b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f04b): Mirror 3: 0xc0309fe5 (stale) CRC-32 checksum value for chunk [0, 0x1f04b): Mirror 1: 0xc0309fe5 Mirror 4: 0xc0309fe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f04b: skipped 000404 1786187107.469797 trunc from 0x01f04b to 0x012d49 (0xc301 bytes) 000405 1786187107.632063 trunc from 0x012d4a to 0x00180a (0x1153f bytes) 000406 1786187107.656933 mapread 0x0007c5 thru 0x00180a (0x1046 bytes) 000407 1786187107.670962 write 0x02b232 thru 0x0384e0 (0xd2af bytes) 000408 1786187107.674097 mapread 0x01b53e thru 0x0213db (0x5e9e bytes) 000410 1786187107.690753 trunc from 0x0384e1 to 0x0225a5 (0x15f3b bytes) 000411 1786187107.709375 write 0x022753 thru 0x029d44 (0x75f2 bytes) 000412 1786187107.713259 read 0x021749 thru 0x029d44 (0x85fc bytes) 000413 1786187107.714321 trunc from 0x029d45 to 0x0390e7 (0xf3a3 bytes) 000414 1786187107.746993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x390e8): Mirror 4: 0xd8c1e372 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x390e8): Mirror 4: 0xd8c1e372 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x390e8: skipped 000416 1786187108.105140 mapwrite 0x027ab3 thru 0x0304a4 (0x89f2 bytes) 000417 1786187108.276608 write 0x002b29 thru 0x00db61 (0xb039 bytes) 000418 1786187108.298367 write 0x009085 thru 0x017d4d (0xecc9 bytes) 000419 1786187108.314171 mapwrite 0x033ae6 thru 0x03edc8 (0xb2e3 bytes) 000420 1786187108.573681 mapwrite 0x00ce43 thru 0x019a64 (0xcc22 bytes) 000423 1786187108.632870 trunc from 0x03edc9 to 0x00a8e2 (0x344e6 bytes) 000427 1786187108.656082 trunc from 0x00a8e3 to 0x012b78 (0x8296 bytes) 000432 1786187108.696274 write 0x03ad04 thru 0x03ffff (0x52fc bytes) 000433 1786187108.698623 mapwrite 0x011d8b thru 0x0165c5 (0x483b bytes) 000435 1786187108.779349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000437 1786187109.088763 trunc from 0x040000 to 0x01505f (0x2afa0 bytes) 000440 1786187109.127017 trunc from 0x015060 to 0x038127 (0x230c8 bytes) 000442 1786187109.161292 read 0x02b7fb thru 0x038127 (0xc92d bytes) 000444 1786187109.176187 write 0x02a542 thru 0x034657 (0xa116 bytes) 000445 1786187109.188924 read 0x024ada thru 0x02c518 (0x7a3f bytes) 000448 1786187109.237197 mapwrite 0x02874a thru 0x030e88 (0x873f bytes) 000450 1786187109.369543 read 0x006c24 thru 0x00cc46 (0x6023 bytes) 000453 1786187109.374606 mapwrite 0x021af9 thru 0x02fc0f (0xe117 bytes) 000454 1786187109.486333 mapread 0x030fe4 thru 0x038127 (0x7144 bytes) 000455 1786187109.496025 MIRROR_EXTEND to 2 mirrors 000456 1786187110.659959 read 0x02458f thru 0x02c926 (0x8398 bytes) 000457 1786187110.669711 write 0x01e8fb thru 0x022b1f (0x4225 bytes) 000458 1786187110.743751 trunc from 0x038128 to 0x03f004 (0x6edd bytes) 000459 1786187110.776420 mapwrite 0x024847 thru 0x029b93 (0x534d bytes) 000461 1786187110.885228 read 0x017659 thru 0x020422 (0x8dca bytes) 000463 1786187110.906859 trunc from 0x03f005 to 0x01b197 (0x23e6d bytes) 000464 1786187110.937077 mapread 0x00b9ea thru 0x018545 (0xcb5c bytes) 000467 1786187110.941334 mapwrite 0x02dbfe thru 0x032a38 (0x4e3b bytes) 000468 1786187111.074832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000472 1786187111.302307 write 0x001a95 thru 0x00c284 (0xa7f0 bytes) 000474 1786187111.316910 mapread 0x01935e thru 0x025e10 (0xcab3 bytes) 000478 1786187111.355191 mapwrite 0x030a6b thru 0x037517 (0x6aad bytes) 000479 1786187111.485447 read 0x02a20a thru 0x035d51 (0xbb48 bytes) 000481 1786187111.487032 read 0x02e673 thru 0x037517 (0x8ea5 bytes) 000482 1786187111.487990 mapread 0x015f43 thru 0x020249 (0xa307 bytes) 000483 1786187111.489349 mapwrite 0x01d40c thru 0x020c88 (0x387d bytes) 000484 1786187111.558403 read 0x00098e thru 0x00f204 (0xe877 bytes) 000486 1786187111.578976 trunc from 0x037518 to 0x02fa8f (0x7a88 bytes) 000487 1786187111.597401 mapread 0x00b677 thru 0x00e5a3 (0x2f2d bytes) 000490 1786187111.601356 mapread 0x01e179 thru 0x01fab2 (0x193a bytes) 000493 1786187111.603734 mapread 0x0077fa thru 0x00e28e (0x6a95 bytes) 000495 1786187111.614124 MIRROR_EXTEND to 2 mirrors 000496 1786187113.118840 trunc from 0x02fa90 to 0x02b574 (0x451b bytes) 000497 1786187113.263116 MIRROR_RESYNC 2 mirrors 000498 1786187114.425905 write 0x033b27 thru 0x035e24 (0x22fe bytes) 000499 1786187114.556184 mapwrite 0x0374f7 thru 0x03ffff (0x8b09 bytes) 000500 1786187114.745713 trunc from 0x040000 to 0x02043d (0x1fbc2 bytes) 000501 1786187114.776956 trunc from 0x02043e to 0x020cb1 (0x874 bytes) 000502 1786187114.811174 write 0x02a876 thru 0x03850a (0xdc95 bytes) 000508 1786187114.825308 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3850b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3850b): Mirror 2: 0x75db10c9 (stale) CRC-32 checksum value for chunk [0, 0x3850b): Mirror 1: 0x5c81a11d Mirror 3: 0x5c81a11d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3850b: skipped 000510 1786187116.363364 trunc from 0x03850b to 0x0233e0 (0x1512a bytes) 000512 1786187116.517286 write 0x002fa2 thru 0x00f9f2 (0xca51 bytes) 000513 1786187116.559678 MIRROR_RESYNC 3 mirrors 000514 1786187117.886117 mapwrite 0x01d7cc thru 0x023cdd (0x6512 bytes) 000515 1786187118.063339 read 0x01a1af thru 0x01ca48 (0x289a bytes) 000517 1786187118.064757 mapwrite 0x02028e thru 0x02f413 (0xf186 bytes) 000518 1786187118.228263 write 0x0312d3 thru 0x0343e6 (0x3114 bytes) 000519 1786187118.237302 mapread 0x02cfd0 thru 0x0343e6 (0x7417 bytes) 000520 1786187118.245734 mapwrite 0x018dc1 thru 0x01fa79 (0x6cb9 bytes) 000521 1786187118.310743 read 0x00f7bf thru 0x0149d9 (0x521b bytes) 000523 1786187118.324595 write 0x026a0c thru 0x032bb3 (0xc1a8 bytes) 000524 1786187118.327723 mapwrite 0x02fb4c thru 0x03c364 (0xc819 bytes) 000526 1786187118.437044 read 0x01a30d thru 0x02a2de (0xffd2 bytes) 000529 1786187118.444946 trunc from 0x03c365 to 0x03aea7 (0x14bd bytes) 000531 1786187118.491277 mapread 0x012641 thru 0x01931d (0x6cdd bytes) 000532 1786187118.496095 mapwrite 0x038f3a thru 0x03ffff (0x70c6 bytes) 000534 1786187118.672310 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x93434c39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x93434c39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000535 1786187119.071280 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000538 1786187119.282026 read 0x009586 thru 0x00bfc4 (0x2a3f bytes) 000539 1786187119.296756 write 0x004552 thru 0x00b567 (0x7016 bytes) 000540 1786187119.313015 mapread 0x010142 thru 0x016859 (0x6718 bytes) 000541 1786187119.333852 mapread 0x0336fb thru 0x036ec1 (0x37c7 bytes) 000547 1786187119.349103 write 0x017f47 thru 0x0217b8 (0x9872 bytes) 000548 1786187119.357516 write 0x00e6a6 thru 0x015681 (0x6fdc bytes) 000550 1786187119.359179 mapwrite 0x020d74 thru 0x022098 (0x1325 bytes) 000551 1786187119.466240 read 0x01b726 thru 0x028b28 (0xd403 bytes) 000552 1786187119.467029 trunc from 0x040000 to 0x02e8ce (0x11731 bytes) 000553 1786187119.490682 mapread 0x001017 thru 0x005395 (0x437f bytes) 000554 1786187119.529237 MIRROR_EXTEND to 2 mirrors 000555 1786187120.463860 trunc from 0x02e8cf to 0x02dec5 (0xa09 bytes) 000560 1786187120.584172 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dec6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dec6): Mirror 2: 0x5375dfc (stale) CRC-32 checksum value for chunk [0, 0x2dec6): Mirror 1: 0x5375dfc Mirror 3: 0x5375dfc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dec6: skipped 000561 1786187121.497689 MIRROR_RESYNC 3 mirrors 000562 1786187122.293579 MIRROR_EXTEND to 4 mirrors 000564 1786187123.083255 MIRROR_EXTEND to 5 mirrors 000566 1786187124.119655 write 0x039d73 thru 0x03e8f9 (0x4b87 bytes) 000567 1786187124.205211 read 0x01b58a thru 0x02284a (0x72c1 bytes) 000568 1786187124.223249 MIRROR_RESYNC 5 mirrors 000569 1786187125.538528 trunc from 0x03e8fa to 0x000792 (0x3e167 bytes) 000570 1786187125.611473 mapwrite 0x00aa1b thru 0x015938 (0xaf1e bytes) 000572 1786187125.718747 mapwrite 0x031fac thru 0x039228 (0x727d bytes) 000573 1786187125.797897 mapwrite 0x032374 thru 0x03aecf (0x8b5c bytes) 000574 1786187125.881490 trunc from 0x03aed0 to 0x03853e (0x2991 bytes) 000575 1786187125.899647 trunc from 0x03853f to 0x0230f8 (0x15446 bytes) 000578 1786187125.917702 mapread 0x00fce8 thru 0x010378 (0x691 bytes) 000579 1786187125.921203 mapread 0x018310 thru 0x022cd8 (0xa9c9 bytes) 000580 1786187125.938658 mapread 0x01c97c thru 0x0230f8 (0x677d bytes) 000581 1786187125.942368 read 0x011d15 thru 0x0187d4 (0x6ac0 bytes) 000583 1786187125.949131 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: 0x9747d5fd (stale) Mirror 3: 0x9747d5fd (stale) Mirror 4: 0x9747d5fd (stale) Mirror 5: 0x9747d5fd (stale) CRC-32 checksum value for chunk [0, 0x230f9): Mirror 1: 0x60669e70 Mirror 6: 0x60669e70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x230f9: skipped 000584 1786187127.580390 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: 0x9747d5fd (stale) Mirror 3: 0x9747d5fd (stale) Mirror 4: 0x9747d5fd (stale) Mirror 5: 0x9747d5fd (stale) CRC-32 checksum value for chunk [0, 0x230f9): Mirror 1: 0x60669e70 Mirror 6: 0x60669e70 Mirror 7: 0x60669e70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x230f9: skipped 000585 1786187128.734464 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: 0x9747d5fd (stale) Mirror 3: 0x9747d5fd (stale) Mirror 4: 0x9747d5fd (stale) Mirror 5: 0x9747d5fd (stale) CRC-32 checksum value for chunk [0, 0x230f9): Mirror 1: 0x60669e70 Mirror 6: 0x60669e70 Mirror 7: 0x60669e70 Mirror 8: 0x60669e70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x230f9: skipped 000586 1786187129.723316 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x230f9): Mirror 2: 0x9747d5fd (stale) Mirror 3: 0x9747d5fd (stale) Mirror 4: 0x9747d5fd (stale) Mirror 5: 0x9747d5fd (stale) CRC-32 checksum value for chunk [0, 0x230f9): Mirror 1: 0x60669e70 Mirror 6: 0x60669e70 Mirror 7: 0x60669e70 Mirror 8: 0x60669e70 Mirror 9: 0x60669e70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x230f9: skipped 000588 1786187130.866578 trunc from 0x0230f9 to 0x02a07b (0x6f83 bytes) 000593 1786187130.998739 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a07c): Mirror 2: 0x1ee56bfb (stale) Mirror 3: 0x1ee56bfb (stale) Mirror 4: 0x1ee56bfb (stale) Mirror 5: 0x1ee56bfb (stale) Mirror 6: 0x62bff522 (stale) Mirror 7: 0x62bff522 (stale) Mirror 9: 0x62bff522 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a07c): Mirror 2: 0x1ee56bfb (stale) Mirror 3: 0x1ee56bfb (stale) Mirror 4: 0x1ee56bfb (stale) Mirror 5: 0x1ee56bfb (stale) Mirror 6: 0x62bff522 (stale) Mirror 7: 0x62bff522 (stale) Mirror 9: 0x62bff522 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a07c: skipped 000594 1786187131.665311 trunc from 0x02a07c to 0x030803 (0x6788 bytes) 000596 1786187131.687302 trunc from 0x030804 to 0x032b93 (0x2390 bytes) 000597 1786187131.708231 mapread 0x00a10f thru 0x013be0 (0x9ad2 bytes) 000598 1786187131.745741 read 0x01fd6a thru 0x02a12d (0xa3c4 bytes) 000599 1786187131.756063 trunc from 0x032b94 to 0x037d7c (0x51e9 bytes) 000600 1786187131.783446 MIRROR_RESYNC 8 mirrors 000602 1786187134.524244 write 0x019156 thru 0x01b1a1 (0x204c bytes) 000606 1786187134.627441 mapwrite 0x0035d6 thru 0x0114f6 (0xdf21 bytes) 000609 1786187134.788680 read 0x016404 thru 0x02060c (0xa209 bytes) 000610 1786187134.814989 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37d7d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d7d): Mirror 2: 0x3222f6e8 (stale) Mirror 3: 0x3222f6e8 (stale) Mirror 4: 0x3222f6e8 (stale) Mirror 5: 0x3222f6e8 (stale) Mirror 6: 0x3222f6e8 (stale) Mirror 7: 0x3222f6e8 (stale) Mirror 9: 0x3222f6e8 (stale) CRC-32 checksum value for chunk [0, 0x37d7d): Mirror 1: 0xf5119b15 Mirror 10: 0xf5119b15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d7d: skipped 000611 1786187136.132587 mapwrite 0x013b12 thru 0x01ea38 (0xaf27 bytes) 000612 1786187136.369585 mapread 0x007d62 thru 0x01011c (0x83bb bytes) 000614 1786187136.383026 trunc from 0x037d7d to 0x01471f (0x2365d bytes) 000616 1786187136.416893 MIRROR_RESYNC 9 mirrors 000618 1786187138.074910 trunc from 0x014720 to 0x00a4ff (0xa220 bytes) 000619 1786187138.170914 write 0x0092c8 thru 0x00d73a (0x4473 bytes) 000620 1786187138.177842 trunc from 0x00d73b to 0x03a652 (0x2cf18 bytes) 000621 1786187138.205762 MIRROR_RESYNC 9 mirrors 000623 1786187140.440023 mapread 0x039bd9 thru 0x03a652 (0xa7a bytes) 000625 1786187140.452676 write 0x0106fc thru 0x01e4cf (0xddd4 bytes) 000627 1786187140.564993 mapwrite 0x02f07c thru 0x03be83 (0xce08 bytes) 000628 1786187140.747215 read 0x0027ba thru 0x00b09c (0x88e3 bytes) 000629 1786187140.766725 mapread 0x00000d thru 0x00bd26 (0xbd1a bytes) 000630 1786187140.786311 MIRROR_RESYNC 9 mirrors 000631 1786187143.447217 write 0x02c503 thru 0x0353ed (0x8eeb bytes) 000633 1786187143.529914 read 0x01adce thru 0x021a40 (0x6c73 bytes) 000636 1786187143.541562 read 0x023ac7 thru 0x03356d (0xfaa7 bytes) 000637 1786187143.553779 mapread 0x01dd63 thru 0x0256d4 (0x7972 bytes) 000640 1786187143.557852 write 0x019b23 thru 0x01de55 (0x4333 bytes) 000642 1786187143.561260 trunc from 0x03be84 to 0x03eea2 (0x301f bytes) 000644 1786187143.588330 write 0x031a15 thru 0x03bbc9 (0xa1b5 bytes) 000645 1786187143.590940 read 0x00d721 thru 0x019779 (0xc059 bytes) 000647 1786187143.593247 mapread 0x02abe5 thru 0x036a03 (0xbe1f bytes) 000649 1786187143.596828 read 0x014763 thru 0x01ad48 (0x65e6 bytes) 000650 1786187143.597857 mapread 0x028b72 thru 0x032066 (0x94f5 bytes) 000653 1786187143.606364 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3eea3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eea3): Mirror 2: 0x4c96258d (stale) Mirror 3: 0x4c96258d (stale) Mirror 4: 0x4c96258d (stale) Mirror 5: 0x4c96258d (stale) Mirror 6: 0x4c96258d (stale) Mirror 7: 0x4c96258d (stale) Mirror 9: 0x4c96258d (stale) Mirror 10: 0x4c96258d (stale) CRC-32 checksum value for chunk [0, 0x3eea3): Mirror 1: 0xfaa7881a Mirror 11: 0xfaa7881a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eea3: skipped 000654 1786187145.768057 MIRROR_RESYNC 10 mirrors 000656 1786187148.812258 mapread 0x0184d7 thru 0x0223b3 (0x9edd bytes) 000657 1786187148.841296 read 0x000814 thru 0x004c06 (0x43f3 bytes) 000660 1786187148.849850 trunc from 0x03eea3 to 0x03918e (0x5d14 bytes) 000661 1786187148.928536 read 0x037f02 thru 0x03918e (0x128d bytes) 000662 1786187148.944125 mapread 0x002414 thru 0x0089b8 (0x65a5 bytes) 000664 1786187148.946636 write 0x00662c thru 0x013d80 (0xd755 bytes) 000666 1786187148.960111 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3918f): Mirror 2: 0xba4a7c21 (stale) Mirror 3: 0xba4a7c21 (stale) Mirror 4: 0xba4a7c21 (stale) Mirror 6: 0xba4a7c21 (stale) Mirror 7: 0xba4a7c21 (stale) Mirror 9: 0xba4a7c21 (stale) Mirror 10: 0xba4a7c21 (stale) Mirror 11: 0xba4a7c21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3918f): Mirror 2: 0xba4a7c21 (stale) Mirror 3: 0xba4a7c21 (stale) Mirror 4: 0xba4a7c21 (stale) Mirror 6: 0xba4a7c21 (stale) Mirror 7: 0xba4a7c21 (stale) Mirror 9: 0xba4a7c21 (stale) Mirror 10: 0xba4a7c21 (stale) Mirror 11: 0xba4a7c21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3918f: skipped 000667 1786187149.706972 read 0x032748 thru 0x03918e (0x6a47 bytes) 000668 1786187149.723009 trunc from 0x03918f to 0x03435c (0x4e32 bytes) 000669 1786187149.750673 trunc from 0x03435d to 0x02a56d (0x9def bytes) 000671 1786187149.806083 write 0x0082b1 thru 0x00ace2 (0x2a32 bytes) 000672 1786187149.816645 write 0x016174 thru 0x01b868 (0x56f5 bytes) 000674 1786187149.833915 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2a56e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a56e): Mirror 2: 0xdc282b78 (stale) Mirror 3: 0xdc282b78 (stale) Mirror 4: 0xdc282b78 (stale) Mirror 6: 0xdc282b78 (stale) Mirror 7: 0xdc282b78 (stale) Mirror 9: 0xdc282b78 (stale) Mirror 10: 0xdc282b78 (stale) Mirror 11: 0xdc282b78 (stale) CRC-32 checksum value for chunk [0, 0x2a56e): Mirror 1: 0x4423aac0 Mirror 12: 0x4423aac0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a56e: skipped 000675 1786187151.517376 trunc from 0x02a56e to 0x00956b (0x21002 bytes) 000676 1786187151.642481 write 0x020f40 thru 0x02a43d (0x94fe bytes) 000679 1786187151.651474 trunc from 0x02a43e to 0x017d35 (0x12708 bytes) 000681 1786187151.689666 trunc from 0x017d36 to 0x00371f (0x14616 bytes) 000682 1786187151.713108 trunc from 0x003720 to 0x01650f (0x12df0 bytes) 000685 1786187151.739346 mapread 0x0155a9 thru 0x01650f (0xf67 bytes) 000688 1786187151.755860 trunc from 0x016510 to 0x00fab6 (0x6a59 bytes) 000690 1786187151.798047 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xfab7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfab7): Mirror 2: 0x7c7f3d60 (stale) Mirror 3: 0x7c7f3d60 (stale) Mirror 4: 0x7c7f3d60 (stale) Mirror 6: 0x7c7f3d60 (stale) Mirror 7: 0x7c7f3d60 (stale) Mirror 9: 0x7c7f3d60 (stale) Mirror 10: 0x7c7f3d60 (stale) Mirror 11: 0x7c7f3d60 (stale) Mirror 12: 0xdcdd8364 (stale) CRC-32 checksum value for chunk [0, 0xfab7): Mirror 1: 0x41c4e9e7 Mirror 13: 0x41c4e9e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfab7: skipped 000691 1786187153.275216 mapwrite 0x006cdb thru 0x0099a6 (0x2ccc bytes) 000692 1786187153.552963 trunc from 0x00fab7 to 0x00ce27 (0x2c8f bytes) 000693 1786187153.637288 MIRROR_RESYNC 11 mirrors 000695 1786187155.621210 read 0x00ac78 thru 0x00ce27 (0x21b0 bytes) 000696 1786187155.640891 mapread 0x0086fb thru 0x00b702 (0x3008 bytes) 000697 1786187155.651239 mapread 0x00b770 thru 0x00ce27 (0x16b8 bytes) 000698 1786187155.657114 trunc from 0x00ce28 to 0x015642 (0x881b bytes) 000699 1786187155.819299 read 0x0127b5 thru 0x015642 (0x2e8e bytes) 000700 1786187155.831543 mapwrite 0x015f12 thru 0x01c512 (0x6601 bytes) 000703 1786187155.945107 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1c513): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c513): Mirror 2: 0xd2c1f812 (stale) Mirror 3: 0xd2c1f812 (stale) Mirror 4: 0xd2c1f812 (stale) Mirror 6: 0xd2c1f812 (stale) Mirror 7: 0xd2c1f812 (stale) Mirror 9: 0xd2c1f812 (stale) Mirror 10: 0xd2c1f812 (stale) Mirror 11: 0xd2c1f812 (stale) Mirror 12: 0xd2c1f812 (stale) Mirror 13: 0xd2c1f812 (stale) CRC-32 checksum value for chunk [0, 0x1c513): Mirror 1: 0xf5538793 Mirror 14: 0xf5538793 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c513: skipped 000704 1786187157.877935 mapread 0x00235a thru 0x0046e1 (0x2388 bytes) 000705 1786187157.990233 MIRROR_RESYNC 12 mirrors 000709 1786187161.347015 mapread 0x01afe5 thru 0x01c512 (0x152e bytes) 000710 1786187161.372312 read 0x00c2b4 thru 0x016fc0 (0xad0d bytes) 000711 1786187161.383109 mapwrite 0x00c4eb thru 0x012488 (0x5f9e bytes) 000713 1786187161.594262 read 0x00073c thru 0x0106d8 (0xff9d bytes) 000714 1786187161.615679 mapwrite 0x00d598 thru 0x0104a4 (0x2f0d bytes) 000715 1786187161.671095 read 0x0147ea thru 0x017334 (0x2b4b bytes) 000716 1786187161.681989 MIRROR_RESYNC 12 mirrors 000717 1786187164.039404 MIRROR_RESYNC 12 mirrors 000718 1786187164.842248 mapwrite 0x012b5c thru 0x020b14 (0xdfb9 bytes) 000719 1786187165.121385 mapwrite 0x008056 thru 0x0091a7 (0x1152 bytes) 000720 1786187165.220512 trunc from 0x020b15 to 0x0229c5 (0x1eb1 bytes) 000721 1786187165.290018 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x229c6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x229c6): Mirror 2: 0x13376ec4 (stale) Mirror 3: 0x13376ec4 (stale) Mirror 4: 0x13376ec4 (stale) Mirror 6: 0x13376ec4 (stale) Mirror 7: 0x13376ec4 (stale) Mirror 9: 0x13376ec4 (stale) Mirror 10: 0x13376ec4 (stale) Mirror 11: 0x13376ec4 (stale) Mirror 12: 0x13376ec4 (stale) Mirror 13: 0x13376ec4 (stale) Mirror 14: 0x13376ec4 (stale) CRC-32 checksum value for chunk [0, 0x229c6): Mirror 1: 0x2774d578 Mirror 15: 0x2774d578 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x229c6: skipped 000722 1786187166.671180 trunc from 0x0229c6 to 0x038cf7 (0x16332 bytes) 000723 1786187166.803929 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: 0x627004e1 (stale) Mirror 3: 0x627004e1 (stale) Mirror 4: 0x627004e1 (stale) Mirror 6: 0x627004e1 (stale) Mirror 7: 0x627004e1 (stale) Mirror 9: 0x627004e1 (stale) Mirror 10: 0x627004e1 (stale) Mirror 11: 0x627004e1 (stale) Mirror 12: 0x627004e1 (stale) Mirror 13: 0x627004e1 (stale) Mirror 14: 0x627004e1 (stale) Mirror 15: 0x5ed2ca77 (stale) CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 1: 0x5ed2ca77 Mirror 16: 0x5ed2ca77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cf8: skipped 000725 1786187168.365275 mapread 0x021a4c thru 0x02b303 (0x98b8 bytes) 000726 1786187168.419652 mapread 0x0210ef thru 0x02a0ab (0x8fbd bytes) 000729 1786187168.466965 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: 0x627004e1 (stale) Mirror 3: 0x627004e1 (stale) Mirror 4: 0x627004e1 (stale) Mirror 6: 0x627004e1 (stale) Mirror 7: 0x627004e1 (stale) Mirror 9: 0x627004e1 (stale) Mirror 10: 0x627004e1 (stale) Mirror 11: 0x627004e1 (stale) Mirror 12: 0x627004e1 (stale) Mirror 13: 0x627004e1 (stale) Mirror 14: 0x627004e1 (stale) Mirror 15: 0x5ed2ca77 (stale) CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 1: 0x5ed2ca77 Mirror 16: 0x5ed2ca77 Mirror 17: 0x5ed2ca77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cf8: skipped 000732 1786187170.303666 mapread 0x019b79 thru 0x0278d3 (0xdd5b bytes) 000736 1786187170.357229 mapread 0x034b27 thru 0x038cf7 (0x41d1 bytes) 000737 1786187170.395770 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: 0x627004e1 (stale) Mirror 3: 0x627004e1 (stale) Mirror 4: 0x627004e1 (stale) Mirror 6: 0x627004e1 (stale) Mirror 7: 0x627004e1 (stale) Mirror 9: 0x627004e1 (stale) Mirror 10: 0x627004e1 (stale) Mirror 11: 0x627004e1 (stale) Mirror 12: 0x627004e1 (stale) Mirror 13: 0x627004e1 (stale) Mirror 14: 0x627004e1 (stale) Mirror 15: 0x5ed2ca77 (stale) CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 1: 0x5ed2ca77 Mirror 16: 0x5ed2ca77 Mirror 17: 0x5ed2ca77 Mirror 18: 0x5ed2ca77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cf8: skipped 000738 1786187171.769062 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: 0x627004e1 (stale) Mirror 3: 0x627004e1 (stale) Mirror 4: 0x627004e1 (stale) Mirror 6: 0x627004e1 (stale) Mirror 7: 0x627004e1 (stale) Mirror 9: 0x627004e1 (stale) Mirror 10: 0x627004e1 (stale) Mirror 11: 0x627004e1 (stale) Mirror 12: 0x627004e1 (stale) Mirror 13: 0x627004e1 (stale) Mirror 14: 0x627004e1 (stale) Mirror 15: 0x5ed2ca77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 2: 0x627004e1 (stale) Mirror 3: 0x627004e1 (stale) Mirror 4: 0x627004e1 (stale) Mirror 6: 0x627004e1 (stale) Mirror 7: 0x627004e1 (stale) Mirror 9: 0x627004e1 (stale) Mirror 10: 0x627004e1 (stale) Mirror 11: 0x627004e1 (stale) Mirror 12: 0x627004e1 (stale) Mirror 13: 0x627004e1 (stale) Mirror 14: 0x627004e1 (stale) Mirror 15: 0x5ed2ca77 (stale) CRC-32 checksum value for chunk [0, 0x38cf8): Mirror 1: 0x5ed2ca77 Mirror 16: 0x5ed2ca77 Mirror 17: 0x5ed2ca77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cf8: skipped 000739 1786187172.844539 trunc from 0x038cf8 to 0x028343 (0x109b4 bytes) 000740 1786187172.990536 mapwrite 0x00adaa thru 0x011819 (0x6a70 bytes) 000742 1786187173.165882 MIRROR_RESYNC 15 mirrors 000746 1786187176.423999 MIRROR_RESYNC 15 mirrors 000747 1786187177.721443 trunc from 0x028344 to 0x0185db (0xfd68 bytes) 000750 1786187177.851717 trunc from 0x0185dc to 0x027a5b (0xf480 bytes) 000751 1786187177.888672 write 0x0356a9 thru 0x038a6d (0x33c5 bytes) 000752 1786187177.896739 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a6e): Mirror 2: 0xaf5c1f49 (stale) Mirror 3: 0xaf5c1f49 (stale) Mirror 4: 0xaf5c1f49 (stale) Mirror 6: 0xaf5c1f49 (stale) Mirror 9: 0xaf5c1f49 (stale) Mirror 10: 0xaf5c1f49 (stale) Mirror 11: 0xaf5c1f49 (stale) Mirror 12: 0xaf5c1f49 (stale) Mirror 13: 0xaf5c1f49 (stale) Mirror 14: 0xaf5c1f49 (stale) Mirror 15: 0xaf5c1f49 (stale) Mirror 16: 0xaf5c1f49 (stale) Mirror 17: 0xaf5c1f49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a6e): Mirror 2: 0xaf5c1f49 (stale) Mirror 3: 0xaf5c1f49 (stale) Mirror 4: 0xaf5c1f49 (stale) Mirror 6: 0xaf5c1f49 (stale) Mirror 9: 0xaf5c1f49 (stale) Mirror 10: 0xaf5c1f49 (stale) Mirror 11: 0xaf5c1f49 (stale) Mirror 12: 0xaf5c1f49 (stale) Mirror 13: 0xaf5c1f49 (stale) Mirror 14: 0xaf5c1f49 (stale) Mirror 15: 0xaf5c1f49 (stale) Mirror 16: 0xaf5c1f49 (stale) Mirror 17: 0xaf5c1f49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a6e: skipped 000757 1786187178.785733 write 0x00901a thru 0x013a4c (0xaa33 bytes) 000758 1786187178.802020 trunc from 0x038a6e to 0x03c182 (0x3715 bytes) 000759 1786187178.829246 mapwrite 0x02e043 thru 0x02e637 (0x5f5 bytes) 000762 1786187178.912904 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c183): Mirror 2: 0xdc9313a8 (stale) Mirror 3: 0xdc9313a8 (stale) Mirror 4: 0xdc9313a8 (stale) Mirror 6: 0xdc9313a8 (stale) Mirror 9: 0xdc9313a8 (stale) Mirror 10: 0xdc9313a8 (stale) Mirror 11: 0xdc9313a8 (stale) Mirror 12: 0xdc9313a8 (stale) Mirror 13: 0xdc9313a8 (stale) Mirror 15: 0xdc9313a8 (stale) Mirror 16: 0xdc9313a8 (stale) Mirror 17: 0xdc9313a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c183): Mirror 2: 0xdc9313a8 (stale) Mirror 3: 0xdc9313a8 (stale) Mirror 4: 0xdc9313a8 (stale) Mirror 6: 0xdc9313a8 (stale) Mirror 9: 0xdc9313a8 (stale) Mirror 10: 0xdc9313a8 (stale) Mirror 11: 0xdc9313a8 (stale) Mirror 12: 0xdc9313a8 (stale) Mirror 13: 0xdc9313a8 (stale) Mirror 15: 0xdc9313a8 (stale) Mirror 16: 0xdc9313a8 (stale) Mirror 17: 0xdc9313a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c183: skipped 000763 1786187179.665364 trunc from 0x03c183 to 0x02e047 (0xe13b bytes) 000764 1786187179.691905 write 0x0237b7 thru 0x024e72 (0x16bc bytes) 000765 1786187179.704357 trunc from 0x02e048 to 0x00caef (0x21558 bytes) 000766 1786187179.726677 mapwrite 0x0067f1 thru 0x0163dc (0xfbec bytes) 000767 1786187179.842676 read 0x00e014 thru 0x0163dc (0x83c9 bytes) 000768 1786187179.846923 trunc from 0x0163dd to 0x0229da (0xc5fe bytes) 000771 1786187179.871051 mapread 0x011706 thru 0x019380 (0x7c7b bytes) 000772 1786187179.891177 mapread 0x00d512 thru 0x015879 (0x8368 bytes) 000773 1786187179.892473 read 0x014b45 thru 0x01b021 (0x64dd bytes) 000774 1786187179.902565 MIRROR_RESYNC 13 mirrors 000775 1786187182.386249 write 0x03e565 thru 0x03ffff (0x1a9b bytes) 000777 1786187182.476814 read 0x013d18 thru 0x01d596 (0x987f bytes) 000778 1786187182.488691 trunc from 0x040000 to 0x0119ef (0x2e610 bytes) 000780 1786187182.520454 mapwrite 0x024c10 thru 0x02e26b (0x965c bytes) 000782 1786187182.628290 mapread 0x003962 thru 0x00bee6 (0x8585 bytes) 000783 1786187182.638438 mapread 0x017064 thru 0x0250e9 (0xe086 bytes) 000784 1786187182.648068 read 0x019287 thru 0x01cfc8 (0x3d42 bytes) 000785 1786187182.649014 mapread 0x0195c9 thru 0x01fcd8 (0x6710 bytes) 000787 1786187182.651049 trunc from 0x02e26c to 0x01cb1f (0x1174c bytes) 000788 1786187182.668132 mapread 0x0126fe thru 0x014594 (0x1e97 bytes) 000789 1786187182.669359 trunc from 0x01cb20 to 0x008718 (0x14407 bytes) 000791 1786187182.689816 mapwrite 0x009e92 thru 0x00a492 (0x601 bytes) 000792 1786187182.766942 mapwrite 0x00c2b4 thru 0x01a3b6 (0xe103 bytes) 000793 1786187182.911635 mapread 0x0105a7 thru 0x016b9f (0x65f9 bytes) 000795 1786187182.918205 mapread 0x00f4e0 thru 0x01a3b6 (0xaed7 bytes) 000796 1786187182.926443 MIRROR_RESYNC 13 mirrors 000797 1786187184.863301 MIRROR_RESYNC 13 mirrors 000798 1786187185.609758 write 0x013b63 thru 0x01f48c (0xb92a bytes) 000799 1786187185.680807 trunc from 0x01f48d to 0x03053c (0x110b0 bytes) 000800 1786187185.706284 mapread 0x01ef8b thru 0x021a7f (0x2af5 bytes) 000801 1786187185.719595 trunc from 0x03053d to 0x02b596 (0x4fa6 bytes) 000803 1786187185.746979 MIRROR_RESYNC 13 mirrors 000805 1786187188.377819 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000806 1786187188.895347 write 0x00b558 thru 0x0192a5 (0xdd4e bytes) 000807 1786187188.999131 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b597): Mirror 2: 0x4a321996 (stale) Mirror 4: 0x4a321996 (stale) Mirror 6: 0x4a321996 (stale) Mirror 9: 0x4a321996 (stale) Mirror 10: 0x4a321996 (stale) Mirror 11: 0x4a321996 (stale) Mirror 12: 0x4a321996 (stale) Mirror 13: 0x4a321996 (stale) Mirror 16: 0x4a321996 (stale) Mirror 17: 0x4a321996 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b597): Mirror 2: 0x4a321996 (stale) Mirror 4: 0x4a321996 (stale) Mirror 6: 0x4a321996 (stale) Mirror 9: 0x4a321996 (stale) Mirror 10: 0x4a321996 (stale) Mirror 11: 0x4a321996 (stale) Mirror 12: 0x4a321996 (stale) Mirror 13: 0x4a321996 (stale) Mirror 16: 0x4a321996 (stale) Mirror 17: 0x4a321996 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b597: skipped 000808 1786187189.648239 trunc from 0x02b597 to 0x029094 (0x2502 bytes) 000809 1786187189.675433 MIRROR_RESYNC 11 mirrors 000813 1786187191.712542 mapread 0x00c0ae thru 0x013f30 (0x7e83 bytes) 000814 1786187191.743991 mapwrite 0x03e89e thru 0x03ffff (0x1762 bytes) 000816 1786187191.881892 read 0x02d8e5 thru 0x032d79 (0x5495 bytes) 000817 1786187191.892878 write 0x038bb9 thru 0x03a07e (0x14c6 bytes) 000819 1786187191.894947 write 0x037e00 thru 0x03ffff (0x8200 bytes) 000820 1786187191.897406 trunc from 0x040000 to 0x009a7a (0x36585 bytes) 000824 1786187191.925544 write 0x005af7 thru 0x00c375 (0x687f bytes) 000825 1786187191.932542 mapread 0x00b2d4 thru 0x00c375 (0x10a2 bytes) 000827 1786187191.936265 mapwrite 0x0280e8 thru 0x036343 (0xe25c bytes) 000828 1786187192.030662 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36344): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36344): Mirror 2: 0xe2865e9b (stale) Mirror 4: 0xe2865e9b (stale) Mirror 6: 0xe2865e9b (stale) Mirror 9: 0xe2865e9b (stale) Mirror 10: 0xe2865e9b (stale) Mirror 11: 0xe2865e9b (stale) Mirror 12: 0xe2865e9b (stale) Mirror 13: 0xe2865e9b (stale) Mirror 16: 0xe2865e9b (stale) Mirror 17: 0xe2865e9b (stale) CRC-32 checksum value for chunk [0, 0x36344): Mirror 1: 0xcc8359a7 Mirror 18: 0xcc8359a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36344: skipped 000829 1786187193.537769 trunc from 0x036344 to 0x03bbf9 (0x58b6 bytes) 000830 1786187193.632895 trunc from 0x03bbfa to 0x021628 (0x1a5d1 bytes) 000831 1786187193.658871 mapwrite 0x00e02d thru 0x017ba6 (0x9b7a bytes) 000832 1786187193.735450 trunc from 0x021629 to 0x03242e (0x10e06 bytes) 000838 1786187193.760360 mapwrite 0x002b0f thru 0x00cfe6 (0xa4d8 bytes) 000840 1786187193.821332 read 0x01b8f1 thru 0x01d256 (0x1966 bytes) 000841 1786187193.833987 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3242f): Mirror 2: 0x653156a1 (stale) Mirror 4: 0x653156a1 (stale) Mirror 9: 0x653156a1 (stale) Mirror 10: 0x653156a1 (stale) Mirror 11: 0x653156a1 (stale) Mirror 12: 0x653156a1 (stale) Mirror 13: 0x653156a1 (stale) Mirror 16: 0x653156a1 (stale) Mirror 17: 0x653156a1 (stale) Mirror 18: 0xe8508826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3242f): Mirror 2: 0x653156a1 (stale) Mirror 4: 0x653156a1 (stale) Mirror 9: 0x653156a1 (stale) Mirror 10: 0x653156a1 (stale) Mirror 11: 0x653156a1 (stale) Mirror 12: 0x653156a1 (stale) Mirror 13: 0x653156a1 (stale) Mirror 16: 0x653156a1 (stale) Mirror 17: 0x653156a1 (stale) Mirror 18: 0xe8508826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3242f: skipped 000842 1786187194.827874 read 0x007d93 thru 0x017c5d (0xfecb bytes) 000843 1786187194.854410 MIRROR_RESYNC 11 mirrors 000846 1786187197.972228 MIRROR_RESYNC 11 mirrors 000850 1786187199.199194 write 0x00a23a thru 0x00c3fb (0x21c2 bytes) 000851 1786187199.313445 mapwrite 0x01f6f1 thru 0x01fae3 (0x3f3 bytes) 000852 1786187199.397397 write 0x03f4da thru 0x03ffff (0xb26 bytes) 000853 1786187199.403184 write 0x0279e1 thru 0x034d4f (0xd36f bytes) 000857 1786187199.438410 trunc from 0x040000 to 0x0038b2 (0x3c74d bytes) 000858 1786187199.485067 read 0x0001e5 thru 0x000ada (0x8f6 bytes) 000859 1786187199.504953 mapwrite 0x021f23 thru 0x0302a7 (0xe385 bytes) 000860 1786187199.674447 mapread 0x0049a1 thru 0x013531 (0xeb91 bytes) 000861 1786187199.716277 read 0x008f02 thru 0x00d269 (0x4368 bytes) 000862 1786187199.724897 write 0x0010fa thru 0x001e69 (0xd70 bytes) 000863 1786187199.734058 read 0x01d764 thru 0x027e2f (0xa6cc bytes) 000865 1786187199.738175 mapread 0x00568e thru 0x011fd3 (0xc946 bytes) 000866 1786187199.756824 MIRROR_RESYNC 11 mirrors 000867 1786187204.426099 trunc from 0x0302a8 to 0x01eb9f (0x11708 bytes) 000868 1786187204.618432 trunc from 0x01eba0 to 0x01e526 (0x679 bytes) 000871 1786187204.653724 mapread 0x017100 thru 0x01c5f2 (0x54f3 bytes) 000872 1786187204.684874 read 0x015a0a thru 0x01e526 (0x8b1d bytes) 000874 1786187204.687066 read 0x0064d6 thru 0x015ebc (0xf9e7 bytes) 000875 1786187204.713406 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e527): Mirror 2: 0x9e5e239d (stale) Mirror 4: 0x9e5e239d (stale) Mirror 9: 0x9e5e239d (stale) Mirror 10: 0x9e5e239d (stale) Mirror 11: 0x9e5e239d (stale) Mirror 12: 0x9e5e239d (stale) Mirror 13: 0x9e5e239d (stale) Mirror 17: 0x9e5e239d (stale) Mirror 18: 0x9e5e239d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e527): Mirror 2: 0x9e5e239d (stale) Mirror 4: 0x9e5e239d (stale) Mirror 9: 0x9e5e239d (stale) Mirror 10: 0x9e5e239d (stale) Mirror 11: 0x9e5e239d (stale) Mirror 12: 0x9e5e239d (stale) Mirror 13: 0x9e5e239d (stale) Mirror 17: 0x9e5e239d (stale) Mirror 18: 0x9e5e239d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e527: skipped 000880 1786187205.674296 mapread 0x01463e thru 0x01e526 (0x9ee9 bytes) 000881 1786187205.697025 mapread 0x013cce thru 0x01e526 (0xa859 bytes) 000884 1786187205.710871 MIRROR_RESYNC 10 mirrors 000888 1786187208.720676 trunc from 0x01e527 to 0x01897b (0x5bab bytes) 000889 1786187208.842217 read 0x00e502 thru 0x01897b (0xa47a bytes) 000890 1786187208.860211 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1897c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1897c): Mirror 2: 0xd9ce2e0f (stale) Mirror 4: 0xd9ce2e0f (stale) Mirror 9: 0xd9ce2e0f (stale) Mirror 10: 0xd9ce2e0f (stale) Mirror 11: 0xd9ce2e0f (stale) Mirror 12: 0xd9ce2e0f (stale) Mirror 13: 0xd9ce2e0f (stale) Mirror 17: 0xd9ce2e0f (stale) Mirror 18: 0xd9ce2e0f (stale) CRC-32 checksum value for chunk [0, 0x1897c): Mirror 1: 0xd9ce2e0f Mirror 19: 0xd9ce2e0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1897c: skipped 000891 1786187210.677138 write 0x006c0b thru 0x01197a (0xad70 bytes) 000892 1786187210.839662 read 0x00796f thru 0x0097c4 (0x1e56 bytes) 000897 1786187210.859677 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1897c): Mirror 2: 0xd9ce2e0f (stale) Mirror 4: 0xd9ce2e0f (stale) Mirror 9: 0xd9ce2e0f (stale) Mirror 11: 0xd9ce2e0f (stale) Mirror 12: 0xd9ce2e0f (stale) Mirror 13: 0xd9ce2e0f (stale) Mirror 17: 0xd9ce2e0f (stale) Mirror 18: 0xd9ce2e0f (stale) Mirror 19: 0xd9ce2e0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1897c): Mirror 2: 0xd9ce2e0f (stale) Mirror 4: 0xd9ce2e0f (stale) Mirror 9: 0xd9ce2e0f (stale) Mirror 11: 0xd9ce2e0f (stale) Mirror 12: 0xd9ce2e0f (stale) Mirror 13: 0xd9ce2e0f (stale) Mirror 17: 0xd9ce2e0f (stale) Mirror 18: 0xd9ce2e0f (stale) Mirror 19: 0xd9ce2e0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1897c: skipped 000898 1786187211.799003 mapwrite 0x0307be thru 0x036bab (0x63ee bytes) 000899 1786187211.905622 trunc from 0x036bac to 0x01020d (0x2699e bytes) 000902 1786187211.945924 trunc from 0x01020e to 0x000d58 (0xf4b5 bytes) 000905 1786187211.971701 read 0x00018e thru 0x000d58 (0xbcb bytes) 000908 1786187211.982329 write 0x02d654 thru 0x032b7c (0x5529 bytes) 000910 1786187211.990453 trunc from 0x032b7d to 0x03734a (0x47ce bytes) 000915 1786187212.025585 MIRROR_RESYNC 10 mirrors 000916 1786187215.252292 mapread 0x022a43 thru 0x02a8b8 (0x7e76 bytes) 000919 1786187215.293046 mapwrite 0x032562 thru 0x03e4b3 (0xbf52 bytes) 000920 1786187215.538009 mapwrite 0x0135ff thru 0x01d40a (0x9e0c bytes) 000921 1786187215.669266 mapwrite 0x01bc18 thru 0x02569a (0x9a83 bytes) 000922 1786187215.839733 read 0x029b26 thru 0x02cf3f (0x341a bytes) 000923 1786187215.841181 read 0x02decf thru 0x0382c9 (0xa3fb bytes) 000924 1786187215.844911 read 0x01928f thru 0x01fab4 (0x6826 bytes) 000925 1786187215.851561 write 0x03df59 thru 0x03ffff (0x20a7 bytes) 000926 1786187215.852935 mapwrite 0x02c29f thru 0x039844 (0xd5a6 bytes) 000927 1786187215.919123 mapread 0x0381bd thru 0x03ffff (0x7e43 bytes) 000928 1786187215.922280 read 0x039f6c thru 0x03ffff (0x6094 bytes) 000930 1786187215.924743 mapwrite 0x0373e3 thru 0x03ffff (0x8c1d bytes) 000931 1786187215.998568 write 0x00310d thru 0x003dbc (0xcb0 bytes) 000932 1786187216.011949 read 0x00d3dc thru 0x01c1c5 (0xedea bytes) 000933 1786187216.013104 read 0x029e23 thru 0x035e8b (0xc069 bytes) 000936 1786187216.021133 mapwrite 0x01c629 thru 0x023794 (0x716c bytes) 000937 1786187216.102907 write 0x00497c thru 0x013199 (0xe81e bytes) 000938 1786187216.105560 mapwrite 0x035a71 thru 0x03ffff (0xa58f bytes) 000940 1786187216.175472 mapwrite 0x01ced9 thru 0x027a85 (0xabad bytes) 000941 1786187216.292410 mapread 0x00c6b9 thru 0x017918 (0xb260 bytes) 000942 1786187216.293874 trunc from 0x040000 to 0x03b59a (0x4a65 bytes) 000943 1786187216.364126 mapread 0x020644 thru 0x0293c2 (0x8d7f bytes) 000944 1786187216.370685 read 0x034a0a thru 0x03553d (0xb34 bytes) 000945 1786187216.371442 trunc from 0x03b59b to 0x00e80d (0x2cd8d bytes) 000947 1786187216.420286 mapread 0x000c10 thru 0x004e91 (0x4282 bytes) 000948 1786187216.424820 write 0x025f25 thru 0x02ef14 (0x8ff0 bytes) 000950 1786187216.433636 trunc from 0x02ef15 to 0x0343a3 (0x548f bytes) 000951 1786187216.485616 mapwrite 0x003272 thru 0x004058 (0xde7 bytes) 000952 1786187216.563715 write 0x017c23 thru 0x019241 (0x161f bytes) 000955 1786187216.591291 write 0x023211 thru 0x02ed8a (0xbb7a bytes) 000956 1786187216.601067 mapread 0x02883b thru 0x0343a3 (0xbb69 bytes) 000957 1786187216.603127 read 0x029d06 thru 0x033d14 (0xa00f bytes) 000959 1786187216.609867 read 0x00248a thru 0x002cb7 (0x82e bytes) 000960 1786187216.611834 write 0x00dec6 thru 0x014418 (0x6553 bytes) 000963 1786187216.616149 mapread 0x00aae8 thru 0x014150 (0x9669 bytes) 000964 1786187216.623303 mapwrite 0x033bad thru 0x03ffff (0xc453 bytes) 000965 1786187216.742951 write 0x030cb2 thru 0x039a72 (0x8dc1 bytes) 000968 1786187216.752207 trunc from 0x040000 to 0x0189ea (0x27615 bytes) 000969 1786187216.783740 mapread 0x014e94 thru 0x0189ea (0x3b57 bytes) 000972 1786187216.788485 write 0x029c4e thru 0x02df63 (0x4316 bytes) 000973 1786187216.790401 trunc from 0x02df64 to 0x00b407 (0x22b5c bytes) 000974 1786187216.827965 mapwrite 0x00af6f thru 0x00c4ea (0x157c bytes) 000975 1786187216.908367 read 0x008b9f thru 0x00c4ea (0x394c bytes) 000979 1786187216.915264 trunc from 0x00c4eb to 0x00be85 (0x665 bytes) 000980 1786187216.941900 write 0x00ad3a thru 0x00c649 (0x1910 bytes) 000981 1786187216.945011 trunc from 0x00c64a to 0x008079 (0x45d0 bytes) 000982 1786187216.985351 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x807a): Mirror 2: 0x92a9c699 (stale) Mirror 4: 0x92a9c699 (stale) Mirror 9: 0x92a9c699 (stale) Mirror 11: 0x92a9c699 (stale) Mirror 12: 0x92a9c699 (stale) Mirror 17: 0x92a9c699 (stale) Mirror 18: 0x92a9c699 (stale) Mirror 19: 0x92a9c699 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x807a): Mirror 2: 0x92a9c699 (stale) Mirror 4: 0x92a9c699 (stale) Mirror 9: 0x92a9c699 (stale) Mirror 11: 0x92a9c699 (stale) Mirror 12: 0x92a9c699 (stale) Mirror 17: 0x92a9c699 (stale) Mirror 18: 0x92a9c699 (stale) Mirror 19: 0x92a9c699 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x807a: skipped 000983 1786187217.860309 write 0x01b380 thru 0x0229c8 (0x7649 bytes) 000988 1786187217.870157 mapwrite 0x01e01f thru 0x02763e (0x9620 bytes) 000989 1786187217.997618 mapwrite 0x038249 thru 0x039974 (0x172c bytes) 000990 1786187218.104195 mapread 0x00a6ac thru 0x011b94 (0x74e9 bytes) 000991 1786187218.129019 read 0x03992b thru 0x039974 (0x4a bytes) 000992 1786187218.129740 read 0x01eff1 thru 0x023224 (0x4234 bytes) 000993 1786187218.130373 trunc from 0x039975 to 0x0096a7 (0x302cd bytes) 000994 1786187218.157416 write 0x03b7b5 thru 0x03ba0a (0x256 bytes) 000999 1786187218.166697 read 0x005221 thru 0x009adf (0x48bf bytes) 001000 1786187218.168903 read 0x02178d thru 0x02a8cc (0x9140 bytes) All operations completed A-OK! PASS 70a (151s) == sanity-flr test 71: check mirror extend parent fid ==== 07:07:05 (1786187225) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 07:07:09 (1786187229) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28128 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177924 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 07:07:20 (1786187240) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre2 Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5585632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4045184 .. failed Extending file size to 5950144 .. Extending file size to 6128128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8523968 .. Extending file size to 7459744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8763712 .. Extending file size to 7683552 .. Extending file size to 8593696 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8796576 .. failed Extending file size to 8871552 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57399 57400 57401 57405 57407 done 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) 192.168.202.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (94s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:08:55 (1786187335) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre2 Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6107072 .. Extending file size to 1160768 .. failed Extending file size to 6033472 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7253440 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7638656 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7705472 .. Extending file size to 8006560 .. Extending file size to 8182944 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8387328 .. Extending file size to 8396928 .. Extending file size to 8956448 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9368160 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8956928 .. failed Waiting 58911 58913 58914 58916 58917 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) 192.168.202.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:10:20 (1786187420) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203815 s, 10.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.567031 s, 3.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc2b3bb6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:10:42 (1786187442) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0885124 s, 23.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 07:10:50 (1786187450) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:11:00 (1786187460) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209391 s, 10.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:11:10 (1786187470) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149716 s, 14.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:11:18 (1786187478) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.321673 s, 6.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:11:54 (1786187514) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12984 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:12:59 (1786187579) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.42621 s, 4.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:14:02 (1786187642) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. Creating new pool test_204f oleg211-server: Pool lustre.test_204f created Adding targets to pool oleg211-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg211-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.450443 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg211-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg211-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg211-server: Pool lustre.test_204f destroyed PASS 204f (89s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:15:31 (1786187731) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:15:42 (1786187742) PASS 205b (12s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:15:54 (1786187754) oleg211-server: Pool lustre.test_206 created Adding targets to pool oleg211-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg211-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg211-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg211-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg211-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:16:24 (1786187784) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10328 s, 10.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404012 s, 26.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfcf011e6 Mirror 3: 0xfcf011e6 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:16:35 (1786187795) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:16:38 (1786187798) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:16:42 (1786187802) Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.152083 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19a0000 Read this (4 bytes): BBBB 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:17:24 (1786187844) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0254982 s, 321 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:17:33 (1786187853) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908932 s, 11.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f1 lmm_fid: [0x200000404:0x1f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f1 lmm_fid: [0x200000404:0x1f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834417 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-08 07:17:35.000000000 -0400 Modify: 2026-08-08 07:17:35.000000000 -0400 Change: 2026-08-08 07:17:36.000000000 -0400 Birth: 2026-08-08 07:17:34.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:17:43 (1786187863) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:17:47 (1786187867) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.615393 s, 66.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.122755 s, 33.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 8 07:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 8 07:17 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:17:58 (1786187878) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:18:07 (1786187887) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1912 sec =========== 07:18:18 (1786187898) === sanity-flr: start cleanup 07:18:20 (1786187900) === === sanity-flr: finish cleanup 07:18:27 (1786187907) ===