-----============= acceptance-small: sanity-flr ============----- Fri Apr 10 14:45:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg108-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:45:34 (1775846734) === oleg108-client.virtnet: executing check_config_client /mnt/lustre oleg108-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg108-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93e6d166a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93e6d166a800.idle_timeout=debug disable quota as required oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:45:50 (1775846750) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7562] == sanity-flr test 0a: lfs mirror create with -N option == 14:45:52 (1775846752) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:46:00 (1775846760) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:46:02 (1775846762) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:46:03 (1775846763) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:46:15 (1775846775) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:46:17 (1775846777) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:46:19 (1775846779) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93e6d166a800.stats=clear osc.lustre-OST0001-osc-ffff93e6d166a800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:46:23 (1775846783) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:46:29 (1775846789) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:46:34 (1775846794) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:46:36 (1775846796) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:46:40 (1775846800) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 14:46:46 (1775846806) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:46:50 (1775846810) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:46:52 (1775846812) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:46:56 (1775846816) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:47:01 (1775846821) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:47:05 (1775846825) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.774207 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684698 s, 15.3 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:47:11 (1775846831) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.149886 s, 90.9 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:47:15 (1775846835) Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (42s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:47:57 (1775846877) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.385505 s, 29.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (23s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:48:20 (1775846900) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (41s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:49:01 (1775846941) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.24035 s, 16.2 MB/s -rw-r--r-- 1 root root 52428800 Apr 10 14:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.604s user 0m0.002s sys 0m0.150s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.852s user 0m0.001s sys 0m0.155s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.903s user 0m0.001s sys 0m0.174s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e6d166a800.ost_server_uuid in FULL state after 0 sec PASS 33b (53s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:49:54 (1775846994) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:49:56 (1775846996) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:49:59 (1775846999) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 14:50:01 (1775847001) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 14:50:05 (1775847005) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.328357 s, 38.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:50:13 (1775847013) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183409 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255698 s, 41.0 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:50:17 (1775847017) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:50:23 (1775847023) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147491 s, 694 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00246307 s, 416 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00288106 s, 355 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157268 s, 651 kB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 14:50:29 (1775847029) create test files with size 15737975 .. 3e71bea2ca0dad938562b6d6228aa021 - dab2f81c633fc20d03bad8df53d2c84c - 53b18c01c0c1ad9908a9d4bc201e5172 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 14:50:41 (1775847041) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:50:48 (1775847048) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:50:52 (1775847052) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471108 s, 22.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299256 s, 35.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 14:50:57 (1775847057) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0685725 s, 30.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.201715 s, 20.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118038 s, 17.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 14:51:06 (1775847066) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 14:51:08 (1775847068) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:51:09 (1775847069) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218216 s, 48.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 14:51:13 (1775847073) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:51:15 (1775847075) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x54:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x54:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:51:20 (1775847080) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.121949 s, 86.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:51:33 (1775847093) llite.lustre-ffff93e6e1fe6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.122909 s, 85.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff93e6e0a7e800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:51:40 (1775847100) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (22s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:52:02 (1775847122) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:52:24 (1775847144) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:52:26 (1775847146) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.134464 s, 23.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127302 s, 24.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe635fd88 (stale) Mirror 2: 0xe635fd88 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 4230fb4f4c02a12c40baeb5dbd69c344 - mirror 1 checksum: 3936839b6e386e6884f31bff79882c99 - mirror 2 checksum: 4230fb4f4c02a12c40baeb5dbd69c344 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 10 14:52:27 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 10 14:52:27 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4230fb4f4c02a12c40baeb5dbd69c344 - mirror 1 checksum: 3936839b6e386e6884f31bff79882c99 - mirror 2 checksum: 3936839b6e386e6884f31bff79882c99 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 10 14:52:27 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 10 14:52:27 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:52:33 (1775847153) Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 14:52:37 (1775847157) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 14:52:53 (1775847173) targets are mounted 14:52:53 (1775847173) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173266 s, 60.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:53:02 (1775847182) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:53:05 (1775847185) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:53:07 (1775847187) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00463949 s, 3.5 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:53:11 (1775847191) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:53:14 (1775847194) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:53:15 (1775847195) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (15s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:53:30 (1775847210) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:53:54 (1775847234) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:54:15 (1775847255) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:54:35 (1775847275) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 14:54:39 (1775847279) mirror create pid 52809 mirror split pid 52810 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52809 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52810 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 14:55:43 (1775847343) Seed set to 7344 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1775847344.163507 mapwrite 0x01463c thru 0x01ba2d (0x73f2 bytes) 000002 1775847344.229320 mapwrite 0x02ac80 thru 0x037304 (0xc685 bytes) 000004 1775847344.330369 write 0x0133c5 thru 0x01b12a (0x7d66 bytes) 000005 1775847344.331656 write 0x027e71 thru 0x02883f (0x9cf bytes) 000007 1775847344.333666 mapread 0x0147e9 thru 0x0161b3 (0x19cb bytes) 000013 1775847344.338175 read 0x01f894 thru 0x020f8d (0x16fa bytes) truncating to largest ever: 0xb851 000015 1775847344.347129 trunc from 0x037305 to 0x00b850 (0x2bab4 bytes) 000016 1775847344.357981 mapwrite 0x03fafc thru 0x03ffff (0x504 bytes) 000017 1775847344.413263 mapread 0x0317be thru 0x03aa82 (0x92c5 bytes) 000019 1775847344.423559 read 0x0303ff thru 0x0305b6 (0x1b8 bytes) 000020 1775847344.424261 mapwrite 0x01332a thru 0x017ac3 (0x479a bytes) 000025 1775847344.465692 mapread 0x03c8ac thru 0x03f36e (0x2ac3 bytes) truncating to largest ever: 0x38ee5 000026 1775847344.467099 trunc from 0x040000 to 0x038ee4 (0x711b bytes) 000027 1775847344.483441 write 0x023b21 thru 0x031c5f (0xe13f bytes) truncating to largest ever: 0x39bd0 000028 1775847344.484745 trunc from 0x038ee5 to 0x039bcf (0xceb bytes) 000029 1775847344.496145 read 0x00b2c2 thru 0x00bb53 (0x892 bytes) 000030 1775847344.496838 mapwrite 0x01c654 thru 0x024d33 (0x86e0 bytes) 000032 1775847344.532307 mapwrite 0x01133f thru 0x013fc2 (0x2c84 bytes) 000033 1775847344.558641 mapread 0x0321ab thru 0x0380a3 (0x5ef9 bytes) 000035 1775847344.559748 mapwrite 0x031ab2 thru 0x037669 (0x5bb8 bytes) 000036 1775847344.585922 MIRROR_EXTEND to 2 mirrors 000039 1775847344.892230 trunc from 0x039bd0 to 0x003eb8 (0x35d17 bytes) 000040 1775847344.949609 MIRROR_RESYNC 2 mirrors 000041 1775847345.351090 write 0x00b45f thru 0x011187 (0x5d29 bytes) 000045 1775847345.411043 read 0x007bc2 thru 0x010fa6 (0x93e5 bytes) 000046 1775847345.415532 mapread 0x004da4 thru 0x0083ab (0x3608 bytes) 000049 1775847345.439462 write 0x002314 thru 0x005031 (0x2d1e bytes) 000051 1775847345.446540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11188): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11188): Mirror 2: 0x387c5c24 (stale) CRC-32 checksum value for chunk [0, 0x11188): Mirror 1: 0xdf7cb4d2 Mirror 3: 0xdf7cb4d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11188: skipped 000052 1775847345.857522 mapread 0x0031f3 thru 0x0103ab (0xd1b9 bytes) 000053 1775847345.886322 read 0x00c909 thru 0x011187 (0x487f bytes) 000054 1775847345.893178 write 0x025fbe thru 0x02a2f3 (0x4336 bytes) truncating to largest ever: 0x3fc95 000058 1775847345.941009 trunc from 0x02a2f4 to 0x03fc94 (0x159a1 bytes) 000059 1775847345.962827 mapwrite 0x01e914 thru 0x02e153 (0xf840 bytes) 000061 1775847346.005649 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: 0xff42c880 (stale) Mirror 3: 0x151b1cc (stale) CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 1: 0x8ec1823 Mirror 4: 0x8ec1823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc95: skipped 000062 1775847346.511681 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: 0xff42c880 (stale) Mirror 3: 0x151b1cc (stale) CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 1: 0x8ec1823 Mirror 4: 0x8ec1823 Mirror 5: 0x8ec1823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc95: skipped 000063 1775847347.125140 trunc from 0x03fc95 to 0x0031d1 (0x3cac3 bytes) 000064 1775847347.179912 trunc from 0x0031d2 to 0x02ed44 (0x2bb73 bytes) 000065 1775847347.198420 read 0x017e1e thru 0x027491 (0xf674 bytes) 000066 1775847347.209514 mapwrite 0x02259b thru 0x023862 (0x12c8 bytes) 000069 1775847347.244617 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 2: 0x74b6e229 (stale) Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 2: 0x74b6e229 (stale) Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000070 1775847347.522263 mapread 0x028fc6 thru 0x02ed44 (0x5d7f bytes) 000072 1775847347.570047 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000073 1775847347.795716 read 0x013874 thru 0x01da5f (0xa1ec bytes) 000075 1775847347.805301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 1: 0xaffa06fd Mirror 5: 0xaffa06fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000078 1775847348.412142 mapread 0x00fd7a thru 0x012a7b (0x2d02 bytes) 000080 1775847348.442778 mapwrite 0x00136c thru 0x001458 (0xed bytes) 000081 1775847348.551006 write 0x0344fc thru 0x03de35 (0x993a bytes) 000082 1775847348.552518 trunc from 0x03de36 to 0x025cdd (0x18158 bytes) 000083 1775847348.567553 mapread 0x022370 thru 0x025cdd (0x396e bytes) 000084 1775847348.578501 read 0x01f414 thru 0x025cdd (0x68ca bytes) 000087 1775847348.586157 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25cde): Mirror 4: 0x39482870 (stale) Mirror 5: 0xa8741a0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cde): Mirror 4: 0x39482870 (stale) Mirror 5: 0xa8741a0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cde: skipped 000088 1775847348.800115 mapwrite 0x023f85 thru 0x02bc0a (0x7c86 bytes) 000089 1775847348.871452 write 0x024fc5 thru 0x0263ba (0x13f6 bytes) 000091 1775847348.873914 mapread 0x0137b2 thru 0x016ef2 (0x3741 bytes) 000092 1775847348.885656 trunc from 0x02bc0b to 0x0321dc (0x65d2 bytes) 000093 1775847348.900776 mapwrite 0x02d6b5 thru 0x037e24 (0xa770 bytes) 000097 1775847348.948342 write 0x03cbc1 thru 0x03ffff (0x343f bytes) 000099 1775847348.960449 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x88c51cf7 (stale) Mirror 5: 0xde5f1d65 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0640424 Mirror 6: 0xc0640424 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000100 1775847349.463779 mapread 0x0206b1 thru 0x024da5 (0x46f5 bytes) 000102 1775847349.496429 mapread 0x015910 thru 0x01b41f (0x5b10 bytes) 000105 1775847349.526019 trunc from 0x040000 to 0x00b351 (0x34cae bytes) 000106 1775847349.563105 read 0x0096a6 thru 0x00b351 (0x1cac bytes) 000109 1775847349.571985 trunc from 0x00b352 to 0x029d18 (0x1e9c7 bytes) 000115 1775847349.591647 MIRROR_RESYNC 4 mirrors 000117 1775847350.042565 write 0x0393d7 thru 0x03ffff (0x6c29 bytes) 000118 1775847350.089751 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad817bd4 (stale) Mirror 5: 0xad817bd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad817bd4 (stale) Mirror 5: 0xad817bd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1775847350.318951 read 0x0125f4 thru 0x01e983 (0xc390 bytes) 000123 1775847350.327710 write 0x01e541 thru 0x0223fa (0x3eba bytes) 000124 1775847350.334443 read 0x036968 thru 0x03b7c5 (0x4e5e bytes) 000125 1775847350.344602 write 0x00505f thru 0x005b8e (0xb30 bytes) 000126 1775847350.349131 MIRROR_RESYNC 3 mirrors 000127 1775847350.753208 mapread 0x0391c5 thru 0x03ffff (0x6e3b bytes) 000128 1775847350.765840 mapwrite 0x0209c9 thru 0x02b3a4 (0xa9dc bytes) 000129 1775847350.869453 read 0x022967 thru 0x030aa6 (0xe140 bytes) 000130 1775847350.870117 write 0x015c46 thru 0x0162bf (0x67a bytes) 000131 1775847350.875144 read 0x01c5ab thru 0x028002 (0xba58 bytes) 000134 1775847350.876742 read 0x035c84 thru 0x0379e9 (0x1d66 bytes) 000135 1775847350.886299 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1627409b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1627409b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000136 1775847351.019629 write 0x012613 thru 0x018634 (0x6022 bytes) 000137 1775847351.025323 read 0x004873 thru 0x012689 (0xde17 bytes) 000143 1775847351.034101 trunc from 0x040000 to 0x0326b1 (0xd94e bytes) 000146 1775847351.058710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000147 1775847351.129548 mapread 0x025deb thru 0x02f752 (0x9968 bytes) 000148 1775847351.143435 write 0x012693 thru 0x0135e1 (0xf4f bytes) 000150 1775847351.149422 trunc from 0x0326b2 to 0x009738 (0x28f79 bytes) 000154 1775847351.161950 read 0x008439 thru 0x009738 (0x1300 bytes) 000155 1775847351.167825 write 0x01b5ca thru 0x02184a (0x6281 bytes) 000157 1775847351.172947 write 0x0031d2 thru 0x007357 (0x4186 bytes) 000158 1775847351.179452 mapread 0x0144b9 thru 0x020c6c (0xc7b4 bytes) 000160 1775847351.186268 read 0x0017cc thru 0x007b12 (0x6347 bytes) 000161 1775847351.187058 write 0x000096 thru 0x007e50 (0x7dbb bytes) 000163 1775847351.192238 trunc from 0x02184b to 0x023570 (0x1d26 bytes) 000164 1775847351.201130 mapread 0x00cd8b thru 0x00f123 (0x2399 bytes) 000165 1775847351.202308 mapread 0x01844b thru 0x023570 (0xb126 bytes) 000169 1775847351.208452 trunc from 0x023571 to 0x027480 (0x3f10 bytes) 000171 1775847351.220497 write 0x0207b3 thru 0x02ec74 (0xe4c2 bytes) 000172 1775847351.221951 read 0x00c84f thru 0x01011e (0x38d0 bytes) 000173 1775847351.222557 write 0x025442 thru 0x034cb2 (0xf871 bytes) 000174 1775847351.227861 MIRROR_EXTEND to 2 mirrors 000175 1775847351.556432 mapwrite 0x003fc8 thru 0x00d538 (0x9571 bytes) 000177 1775847351.695167 read 0x028dd1 thru 0x02ae37 (0x2067 bytes) 000178 1775847351.703152 trunc from 0x034cb3 to 0x01feae (0x14e04 bytes) 000179 1775847351.735006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000180 1775847351.846155 mapwrite 0x01e8d7 thru 0x029cfe (0xb428 bytes) 000182 1775847351.932169 read 0x0244da thru 0x029cfe (0x5825 bytes) 000183 1775847351.932952 mapwrite 0x012c15 thru 0x014f8b (0x2377 bytes) 000184 1775847351.968636 mapread 0x024b5b thru 0x029cfe (0x51a4 bytes) 000187 1775847351.971245 mapread 0x00eec1 thru 0x01e5f8 (0xf738 bytes) 000189 1775847351.973091 trunc from 0x029cff to 0x01e2bc (0xba42 bytes) 000190 1775847351.988148 read 0x01a2b8 thru 0x01e2bc (0x4005 bytes) 000191 1775847351.989041 mapwrite 0x034112 thru 0x0367c8 (0x26b7 bytes) 000192 1775847352.036974 write 0x02fbdd thru 0x03df48 (0xe36c bytes) 000196 1775847352.040973 trunc from 0x03df49 to 0x008b62 (0x353e6 bytes) 000199 1775847352.057014 trunc from 0x008b63 to 0x02dd65 (0x25203 bytes) 000200 1775847352.073930 read 0x028949 thru 0x02dd65 (0x541d bytes) 000201 1775847352.090664 trunc from 0x02dd66 to 0x00aa60 (0x23305 bytes) 000204 1775847352.108042 trunc from 0x00aa61 to 0x00fefc (0x549c bytes) 000206 1775847352.118759 mapwrite 0x028df7 thru 0x0338a4 (0xaaae bytes) 000209 1775847352.212504 write 0x03c51d thru 0x03ffff (0x3ae3 bytes) 000210 1775847352.214221 write 0x007a84 thru 0x00e4eb (0x6a68 bytes) 000211 1775847352.234381 write 0x020213 thru 0x029a0a (0x97f8 bytes) 000214 1775847352.247140 mapwrite 0x035588 thru 0x03b2d6 (0x5d4f bytes) 000215 1775847352.299300 trunc from 0x040000 to 0x03706c (0x8f93 bytes) 000217 1775847352.316406 write 0x01b013 thru 0x028ea1 (0xde8f bytes) 000219 1775847352.328718 mapwrite 0x010ef3 thru 0x0111f6 (0x304 bytes) 000220 1775847352.361920 mapwrite 0x03a68b thru 0x03ffff (0x5975 bytes) 000221 1775847352.413194 MIRROR_EXTEND to 2 mirrors 000222 1775847352.823562 mapread 0x0240ee thru 0x0293e2 (0x52f5 bytes) 000224 1775847352.855179 mapread 0x029ead thru 0x03678c (0xc8e0 bytes) 000225 1775847352.868938 MIRROR_EXTEND to 3 mirrors 000227 1775847353.302848 trunc from 0x040000 to 0x031753 (0xe8ac bytes) 000229 1775847353.389319 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) CRC-32 checksum value for chunk [0, 0x31754): Mirror 1: 0x6e9bb559 Mirror 4: 0x6e9bb559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000232 1775847354.049697 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000235 1775847354.273158 mapread 0x000846 thru 0x003e86 (0x3641 bytes) 000236 1775847354.294260 read 0x0182e4 thru 0x01f46c (0x7189 bytes) 000237 1775847354.312254 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) CRC-32 checksum value for chunk [0, 0x31754): Mirror 1: 0x6e9bb559 Mirror 4: 0x6e9bb559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000238 1775847355.031469 read 0x02e283 thru 0x031753 (0x34d1 bytes) 000239 1775847355.045665 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000240 1775847355.277422 trunc from 0x031754 to 0x023ec6 (0xd88d bytes) 000241 1775847355.292198 write 0x03edfe thru 0x03ffff (0x1202 bytes) 000242 1775847355.296084 MIRROR_RESYNC 3 mirrors 000243 1775847355.773608 MIRROR_EXTEND to 4 mirrors 000244 1775847356.207635 read 0x01c566 thru 0x02bff6 (0xfa91 bytes) 000246 1775847356.215568 read 0x01d9b8 thru 0x0216e5 (0x3d2e bytes) 000247 1775847356.218601 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000248 1775847356.486167 read 0x013769 thru 0x01adf1 (0x7689 bytes) 000249 1775847356.510738 write 0x019f67 thru 0x02647e (0xc518 bytes) 000251 1775847356.582058 trunc from 0x040000 to 0x008329 (0x37cd6 bytes) 000253 1775847356.615928 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x832a): Mirror 4: 0x130bde8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x832a): Mirror 4: 0x130bde8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x832a: skipped 000254 1775847356.793604 trunc from 0x00832a to 0x0095b7 (0x128e bytes) 000256 1775847356.803513 mapwrite 0x012691 thru 0x01ea39 (0xc3a9 bytes) 000257 1775847356.863906 mapwrite 0x03bc61 thru 0x03ffff (0x439f bytes) 000258 1775847356.908543 trunc from 0x040000 to 0x00e462 (0x31b9d bytes) 000259 1775847356.928650 trunc from 0x00e463 to 0x029d82 (0x1b920 bytes) 000260 1775847356.947962 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29d83): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d83): Mirror 4: 0x781db661 (stale) CRC-32 checksum value for chunk [0, 0x29d83): Mirror 1: 0xcfe87690 Mirror 5: 0xcfe87690 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d83: skipped 000261 1775847357.560934 trunc from 0x029d83 to 0x014bbb (0x151c7 bytes) 000262 1775847357.599398 read 0x002432 thru 0x00cc34 (0xa803 bytes) 000263 1775847357.610438 trunc from 0x014bbc to 0x018f32 (0x4377 bytes) 000264 1775847357.628880 mapread 0x011ab0 thru 0x018f32 (0x7483 bytes) 000267 1775847357.655977 mapwrite 0x00137d thru 0x003d80 (0x2a04 bytes) 000268 1775847357.696353 mapwrite 0x02862a thru 0x02fce5 (0x76bc bytes) 000269 1775847357.747175 mapread 0x018d8e thru 0x02687b (0xdaee bytes) 000270 1775847357.765580 mapwrite 0x034ff2 thru 0x03f202 (0xa211 bytes) 000271 1775847357.842846 mapwrite 0x01a271 thru 0x024250 (0x9fe0 bytes) 000273 1775847357.867992 mapwrite 0x02048b thru 0x023faf (0x3b25 bytes) 000274 1775847357.890891 trunc from 0x03f203 to 0x007e31 (0x373d1 bytes) 000276 1775847357.909713 mapread 0x0077b5 thru 0x007e31 (0x67d bytes) 000277 1775847357.911095 read 0x006f1b thru 0x00785f (0x945 bytes) 000279 1775847357.912461 read 0x006111 thru 0x007e31 (0x1d21 bytes) 000281 1775847357.914256 read 0x001f5d thru 0x005d20 (0x3dc4 bytes) 000288 1775847357.919206 read 0x0021a4 thru 0x007e31 (0x5c8e bytes) 000289 1775847357.920348 write 0x019c0a thru 0x023e09 (0xa200 bytes) 000290 1775847357.922277 mapwrite 0x03cdc8 thru 0x03ffff (0x3238 bytes) 000291 1775847357.978437 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4e046353 (stale) Mirror 5: 0x76275e60 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b0df39a Mirror 6: 0x4b0df39a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1775847358.577989 mapwrite 0x03a462 thru 0x03d606 (0x31a5 bytes) 000294 1775847358.715927 trunc from 0x040000 to 0x02b449 (0x14bb6 bytes) 000295 1775847358.740573 mapread 0x016574 thru 0x01699e (0x42b bytes) 000296 1775847358.755077 mapwrite 0x03427d thru 0x03dd72 (0x9af6 bytes) 000298 1775847358.828862 read 0x0009c1 thru 0x00586c (0x4eac bytes) 000300 1775847358.836254 mapwrite 0x01fac0 thru 0x02ec8f (0xf1d0 bytes) 000302 1775847358.891929 mapread 0x03a92c thru 0x03aac9 (0x19e bytes) 000303 1775847358.893010 read 0x01242e thru 0x0179c3 (0x5596 bytes) 000304 1775847358.897331 MIRROR_RESYNC 4 mirrors 000307 1775847359.412751 read 0x01679a thru 0x026186 (0xf9ed bytes) 000308 1775847359.426195 read 0x00f78f thru 0x01e6e8 (0xef5a bytes) 000309 1775847359.434123 trunc from 0x03dd73 to 0x00d034 (0x30d3e bytes) 000310 1775847359.472769 write 0x00de38 thru 0x014151 (0x631a bytes) 000311 1775847359.477167 mapread 0x00b2ed thru 0x00e643 (0x3357 bytes) 000312 1775847359.486177 mapwrite 0x017f35 thru 0x01b89c (0x3968 bytes) 000314 1775847359.553222 read 0x018234 thru 0x01b89c (0x3669 bytes) 000315 1775847359.554018 trunc from 0x01b89d to 0x0320df (0x16843 bytes) 000316 1775847359.573720 write 0x00fd81 thru 0x01626e (0x64ee bytes) 000317 1775847359.578874 read 0x02a04b thru 0x0320df (0x8095 bytes) 000319 1775847359.584549 read 0x02e2d4 thru 0x0320df (0x3e0c bytes) 000320 1775847359.585063 read 0x03003c thru 0x0320df (0x20a4 bytes) 000321 1775847359.585429 write 0x023387 thru 0x03327e (0xfef8 bytes) 000322 1775847359.586411 read 0x02bc4a thru 0x03327e (0x7635 bytes) 000324 1775847359.587363 write 0x00b5f8 thru 0x01361b (0x8024 bytes) 000327 1775847359.589730 trunc from 0x03327f to 0x005d0e (0x2d570 bytes) 000329 1775847359.603778 write 0x00c369 thru 0x0111ce (0x4e66 bytes) 000332 1775847359.606305 trunc from 0x0111cf to 0x033a84 (0x228b6 bytes) 000333 1775847359.620918 mapwrite 0x016414 thru 0x022204 (0xbdf1 bytes) 000334 1775847359.706899 MIRROR_RESYNC 4 mirrors 000335 1775847360.171457 read 0x02f085 thru 0x033a84 (0x4a00 bytes) 000336 1775847360.185222 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000337 1775847360.408321 mapwrite 0x03a39e thru 0x03ffff (0x5c62 bytes) 000338 1775847360.542299 read 0x01cbb8 thru 0x029fb8 (0xd401 bytes) 000341 1775847360.552999 MIRROR_RESYNC 3 mirrors 000342 1775847360.919897 mapwrite 0x02586b thru 0x034fb7 (0xf74d bytes) 000348 1775847361.036826 trunc from 0x040000 to 0x029596 (0x16a69 bytes) 000349 1775847361.064495 trunc from 0x029597 to 0x00afba (0x1e5dc bytes) 000350 1775847361.072721 read 0x00a30f thru 0x00afba (0xcac bytes) 000351 1775847361.075957 mapread 0x003837 thru 0x00afba (0x7784 bytes) 000352 1775847361.080894 read 0x007c57 thru 0x00afba (0x3364 bytes) 000355 1775847361.089832 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xafbb): Mirror 6: 0xcad00a9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xafbb): Mirror 6: 0xcad00a9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xafbb: skipped 000357 1775847361.239456 mapread 0x001903 thru 0x0098fb (0x7ff9 bytes) 000358 1775847361.251161 read 0x000ab1 thru 0x00afba (0xa50a bytes) 000359 1775847361.261268 mapread 0x0078b8 thru 0x00afba (0x3703 bytes) 000360 1775847361.263042 write 0x020cff thru 0x028031 (0x7333 bytes) 000361 1775847361.264663 mapwrite 0x028565 thru 0x02d773 (0x520f bytes) 000363 1775847361.332086 mapwrite 0x000e2d thru 0x00bcaf (0xae83 bytes) 000365 1775847361.373635 mapread 0x01d175 thru 0x01f7c6 (0x2652 bytes) 000367 1775847361.384028 mapread 0x00915b thru 0x00d230 (0x40d6 bytes) 000368 1775847361.385826 trunc from 0x02d774 to 0x0203cd (0xd3a6 bytes) 000369 1775847361.403486 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000370 1775847361.511088 write 0x0343b8 thru 0x03f238 (0xae81 bytes) 000371 1775847361.518552 MIRROR_EXTEND to 2 mirrors 000373 1775847361.876963 trunc from 0x03f239 to 0x000602 (0x3ec36 bytes) 000376 1775847361.979570 MIRROR_RESYNC 2 mirrors 000379 1775847362.349063 read 0x00029f thru 0x000602 (0x364 bytes) 000380 1775847362.353299 write 0x032dbb thru 0x039fe9 (0x722f bytes) 000381 1775847362.431323 MIRROR_RESYNC 2 mirrors 000382 1775847362.886645 write 0x02f13b thru 0x03796b (0x8831 bytes) 000385 1775847362.948242 mapread 0x018ca5 thru 0x02826c (0xf5c8 bytes) 000386 1775847362.966496 read 0x01048f thru 0x01a9d1 (0xa543 bytes) 000388 1775847362.971267 mapread 0x035afd thru 0x039fe9 (0x44ed bytes) 000390 1775847362.982347 write 0x01c6bc thru 0x02310f (0x6a54 bytes) 000392 1775847362.984753 mapread 0x02bd19 thru 0x032d7a (0x7062 bytes) 000393 1775847362.999524 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39fea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39fea): Mirror 2: 0xf003e260 (stale) CRC-32 checksum value for chunk [0, 0x39fea): Mirror 1: 0xe97d35af Mirror 3: 0xe97d35af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39fea: skipped 000394 1775847363.627341 mapread 0x025f99 thru 0x028c5b (0x2cc3 bytes) 000400 1775847363.716707 MIRROR_RESYNC 3 mirrors 000401 1775847364.215460 read 0x013211 thru 0x01df9a (0xad8a bytes) 000402 1775847364.227079 write 0x035a5c thru 0x03ffff (0xa5a4 bytes) 000403 1775847364.289567 write 0x00ec44 thru 0x01b5bd (0xc97a bytes) 000404 1775847364.300659 read 0x02b2ec thru 0x0354d3 (0xa1e8 bytes) 000405 1775847364.312716 trunc from 0x040000 to 0x00f2d8 (0x30d27 bytes) 000409 1775847364.348765 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2d9): Mirror 2: 0xac9e39e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2d9): Mirror 2: 0xac9e39e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2d9: skipped 000410 1775847364.583514 trunc from 0x00f2d9 to 0x01def5 (0xec1d bytes) 000412 1775847364.606115 trunc from 0x01def6 to 0x027cfa (0x9e05 bytes) 000413 1775847364.642989 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 2: 0xb91e017f (stale) CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 1: 0xa9ce5778 Mirror 3: 0xa9ce5778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cfb: skipped 000415 1775847365.315243 trunc from 0x027cfb to 0x022bf2 (0x5108 bytes) 000417 1775847365.392998 write 0x03ffce thru 0x03ffff (0x32 bytes) 000418 1775847365.401428 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce60e22f (stale) Mirror 3: 0x77dac87a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb54adf66 Mirror 4: 0xb54adf66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000419 1775847366.109640 write 0x01b1bd thru 0x020e05 (0x5c49 bytes) 000422 1775847366.179261 MIRROR_RESYNC 4 mirrors 000423 1775847366.759674 MIRROR_RESYNC 4 mirrors 000424 1775847367.062552 mapwrite 0x00daa5 thru 0x019917 (0xbe73 bytes) 000426 1775847367.208679 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8d569f5d (stale) Mirror 3: 0x8d569f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8d569f5d (stale) Mirror 3: 0x8d569f5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000429 1775847367.490557 mapread 0x030a92 thru 0x03ffff (0xf56e bytes) 000430 1775847367.512696 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d569f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d569f5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000432 1775847367.711167 mapwrite 0x03c17d thru 0x03ffff (0x3e83 bytes) 000433 1775847367.769716 trunc from 0x040000 to 0x019384 (0x26c7b bytes) 000434 1775847367.785152 mapread 0x00a97c thru 0x01414f (0x97d4 bytes) 000435 1775847367.802198 write 0x01906f thru 0x01bd3e (0x2cd0 bytes) 000436 1775847367.803652 write 0x02a61a thru 0x03364f (0x9036 bytes) 000440 1775847367.810056 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: 0xe2370161 (stale) CRC-32 checksum value for chunk [0, 0x33650): Mirror 1: 0x7fd24549 Mirror 4: 0x7fd24549 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33650: skipped 000441 1775847368.347305 read 0x02bca9 thru 0x03364f (0x79a7 bytes) 000444 1775847368.376429 mapwrite 0x03112f thru 0x0334c9 (0x239b bytes) 000445 1775847368.523716 write 0x00c8c7 thru 0x015206 (0x8940 bytes) 000452 1775847368.543925 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: 0xe2370161 (stale) Mirror 4: 0x7fd24549 (stale) CRC-32 checksum value for chunk [0, 0x33650): Mirror 1: 0x5f6f9e39 Mirror 5: 0x5f6f9e39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33650: skipped 000454 1775847369.148431 trunc from 0x033650 to 0x011d7f (0x218d0 bytes) 000456 1775847369.209047 write 0x039048 thru 0x03ec29 (0x5be2 bytes) 000458 1775847369.210409 trunc from 0x03ec2a to 0x006c9b (0x37f8e bytes) 000459 1775847369.224172 mapwrite 0x00471d thru 0x00ea13 (0xa2f7 bytes) 000460 1775847369.283527 mapread 0x00019a thru 0x00e5ca (0xe431 bytes) 000462 1775847369.290073 mapwrite 0x0108c6 thru 0x01e066 (0xd7a1 bytes) 000463 1775847369.360852 trunc from 0x01e067 to 0x00ba7f (0x125e7 bytes) 000465 1775847369.374099 mapread 0x00512e thru 0x0054c7 (0x39a bytes) 000467 1775847369.375169 read 0x00b866 thru 0x00ba7f (0x21a bytes) 000468 1775847369.375630 mapwrite 0x03aa5a thru 0x03ffff (0x55a6 bytes) 000469 1775847369.423412 write 0x039ee1 thru 0x03ffff (0x611f bytes) 000471 1775847369.424941 mapread 0x001845 thru 0x00e78c (0xcf48 bytes) 000472 1775847369.434298 read 0x014a4c thru 0x0235c8 (0xeb7d bytes) 000473 1775847369.440498 read 0x00c5b4 thru 0x015f61 (0x99ae bytes) 000474 1775847369.441109 read 0x03b347 thru 0x03ffff (0x4cb9 bytes) 000475 1775847369.441637 mapwrite 0x031f66 thru 0x0377b4 (0x584f bytes) 000476 1775847369.476986 mapwrite 0x00f22a thru 0x010cd7 (0x1aae bytes) 000477 1775847369.518270 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b525cea (stale) Mirror 5: 0x9be714aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b525cea (stale) Mirror 5: 0x9be714aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1775847369.788230 trunc from 0x040000 to 0x016f06 (0x290f9 bytes) 000481 1775847369.808015 read 0x008ba4 thru 0x0160dc (0xd539 bytes) 000483 1775847369.819238 trunc from 0x016f07 to 0x00cd2d (0xa1d9 bytes) 000484 1775847369.829735 mapwrite 0x004d36 thru 0x00a160 (0x542b bytes) 000488 1775847369.869138 read 0x0069c4 thru 0x00cd2d (0x636a bytes) 000491 1775847369.870832 read 0x00b389 thru 0x00cd2d (0x19a5 bytes) 000493 1775847369.871919 mapwrite 0x006cb9 thru 0x00e196 (0x74de bytes) 000494 1775847369.905844 write 0x006418 thru 0x014cc2 (0xe8ab bytes) 000495 1775847369.907570 mapwrite 0x0352ba thru 0x03711d (0x1e64 bytes) 000497 1775847369.959905 mapread 0x00f2cf thru 0x011e06 (0x2b38 bytes) 000498 1775847369.960813 trunc from 0x03711e to 0x025821 (0x118fc bytes) 000500 1775847369.974998 mapwrite 0x03d7dc thru 0x03ffff (0x2824 bytes) 000503 1775847370.016041 trunc from 0x040000 to 0x026452 (0x19bad bytes) 000504 1775847370.032653 mapwrite 0x01399a thru 0x016840 (0x2ea7 bytes) 000506 1775847370.072241 mapwrite 0x0101f5 thru 0x012b71 (0x297d bytes) 000507 1775847370.100337 mapread 0x021084 thru 0x026452 (0x53cf bytes) 000508 1775847370.105450 mapread 0x00c78a thru 0x013085 (0x68fc bytes) 000511 1775847370.112309 trunc from 0x026453 to 0x00bee9 (0x1a569 bytes) 000512 1775847370.133891 read 0x00affb thru 0x00bee9 (0xeef bytes) 000513 1775847370.134485 write 0x002600 thru 0x00db49 (0xb54a bytes) 000514 1775847370.140781 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 4: 0x27314a82 (stale) Mirror 5: 0xef8d2c87 (stale) CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 1: 0x358e3ef9 Mirror 6: 0x358e3ef9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb4a: skipped 000515 1775847370.731036 trunc from 0x00db4a to 0x038b3f (0x2aff6 bytes) 000516 1775847370.829688 read 0x01f470 thru 0x020fc8 (0x1b59 bytes) 000518 1775847370.838001 write 0x005247 thru 0x014290 (0xf04a bytes) 000520 1775847370.855107 write 0x038013 thru 0x03f352 (0x7340 bytes) 000521 1775847370.867131 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f353): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f353): Mirror 4: 0x769c2196 (stale) Mirror 5: 0xbfab1cf6 (stale) Mirror 6: 0x384544fc (stale) CRC-32 checksum value for chunk [0, 0x3f353): Mirror 1: 0x4faa14f1 Mirror 7: 0x4faa14f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f353: skipped 000522 1775847371.494420 trunc from 0x03f353 to 0x0228e0 (0x1ca72 bytes) 000524 1775847371.562098 read 0x01ed97 thru 0x0228e0 (0x3b4a bytes) 000525 1775847371.569308 read 0x014aaa thru 0x0185d4 (0x3b2b bytes) 000526 1775847371.581555 mapread 0x0091ae thru 0x00cd1d (0x3b70 bytes) 000529 1775847371.585276 read 0x000e3d thru 0x01007b (0xf23f bytes) 000532 1775847371.587111 mapwrite 0x02c005 thru 0x0325da (0x65d6 bytes) 000534 1775847371.699144 trunc from 0x0325db to 0x00e196 (0x24444 bytes) 000535 1775847371.722765 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe197): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe197): Mirror 4: 0xc6382e6c (stale) Mirror 5: 0xf38c9b9c (stale) Mirror 6: 0x59d1ca45 (stale) Mirror 7: 0x19f06de (stale) CRC-32 checksum value for chunk [0, 0xe197): Mirror 1: 0x19f06de Mirror 8: 0x19f06de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe197: skipped 000536 1775847372.186308 mapwrite 0x0371e6 thru 0x03ffff (0x8e1a bytes) 000537 1775847372.324921 mapwrite 0x017d0c thru 0x01a2d5 (0x25ca bytes) 000538 1775847372.375455 MIRROR_RESYNC 6 mirrors 000539 1775847373.121716 mapwrite 0x0216ea thru 0x0288e6 (0x71fd bytes) 000540 1775847373.298373 mapwrite 0x0142c8 thru 0x01705e (0x2d97 bytes) 000542 1775847373.359433 write 0x0266ef thru 0x029636 (0x2f48 bytes) 000549 1775847373.365311 mapwrite 0x01260b thru 0x01581a (0x3210 bytes) 000550 1775847373.406132 mapwrite 0x03fa02 thru 0x03ffff (0x5fe bytes) 000551 1775847373.450051 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76c16d51 Mirror 9: 0x76c16d51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000553 1775847374.076163 mapwrite 0x01a095 thru 0x01f42b (0x5397 bytes) 000554 1775847374.224239 trunc from 0x040000 to 0x027cf9 (0x18306 bytes) 000558 1775847374.258837 mapwrite 0x002e3e thru 0x010ff0 (0xe1b3 bytes) 000560 1775847374.301231 trunc from 0x027cfa to 0x02c203 (0x450a bytes) 000562 1775847374.321963 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c204): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c204): Mirror 4: 0xa40b9160 (stale) Mirror 5: 0xa40b9160 (stale) Mirror 6: 0xa40b9160 (stale) Mirror 7: 0xa40b9160 (stale) Mirror 8: 0xa40b9160 (stale) Mirror 9: 0x6c2c991c (stale) CRC-32 checksum value for chunk [0, 0x2c204): Mirror 1: 0x7144042d Mirror 10: 0x7144042d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c204: skipped 000563 1775847375.116258 write 0x0179e2 thru 0x023f3b (0xc55a bytes) 000564 1775847375.181977 read 0x0272fb thru 0x02c203 (0x4f09 bytes) 000565 1775847375.188432 mapread 0x001b8f thru 0x00ca39 (0xaeab bytes) 000570 1775847375.197647 read 0x0205fc thru 0x02b845 (0xb24a bytes) 000571 1775847375.204239 mapwrite 0x03a070 thru 0x03ffff (0x5f90 bytes) 000572 1775847375.263352 mapread 0x03462a thru 0x03ffff (0xb9d6 bytes) 000577 1775847375.267845 read 0x03d33c thru 0x03ffff (0x2cc4 bytes) 000579 1775847375.269204 write 0x018e71 thru 0x0236e0 (0xa870 bytes) 000581 1775847375.271522 mapwrite 0x003927 thru 0x0120c9 (0xe7a3 bytes) 000582 1775847375.315979 read 0x013b92 thru 0x01f32b (0xb79a bytes) 000584 1775847375.325227 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) Mirror 10: 0x89ca4300 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) Mirror 10: 0x89ca4300 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000585 1775847375.617358 write 0x033cb1 thru 0x03ffff (0xc34f bytes) 000586 1775847375.623576 trunc from 0x040000 to 0x02c62d (0x139d2 bytes) 000587 1775847375.644598 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c62e): Mirror 5: 0xd68facc (stale) Mirror 6: 0xd68facc (stale) Mirror 7: 0xd68facc (stale) Mirror 8: 0xd68facc (stale) Mirror 10: 0x3ec4d2e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c62e): Mirror 5: 0xd68facc (stale) Mirror 6: 0xd68facc (stale) Mirror 7: 0xd68facc (stale) Mirror 8: 0xd68facc (stale) Mirror 10: 0x3ec4d2e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c62e: skipped 000589 1775847375.973222 mapread 0x01f045 thru 0x023017 (0x3fd3 bytes) 000590 1775847376.003418 trunc from 0x02c62e to 0x01d0f2 (0xf53b bytes) 000591 1775847376.015694 mapwrite 0x01ea7f thru 0x01fd8d (0x130f bytes) 000593 1775847376.077310 write 0x0243a9 thru 0x02578e (0x13e6 bytes) 000595 1775847376.078624 mapread 0x018a09 thru 0x020ebe (0x84b6 bytes) 000597 1775847376.098262 MIRROR_RESYNC 6 mirrors 000599 1775847376.758545 mapread 0x01dd7d thru 0x0248dd (0x6b61 bytes) 000600 1775847376.774850 mapwrite 0x03c96e thru 0x03ffff (0x3692 bytes) 000601 1775847376.861271 write 0x011d08 thru 0x013dd1 (0x20ca bytes) 000602 1775847376.870032 write 0x026f06 thru 0x02b43b (0x4536 bytes) 000603 1775847376.880569 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8db3291d (stale) Mirror 6: 0x8db3291d (stale) Mirror 7: 0x8db3291d (stale) Mirror 8: 0x8db3291d (stale) Mirror 10: 0x8db3291d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd6b5eb5 Mirror 11: 0xfd6b5eb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1775847377.593109 mapwrite 0x02d4da thru 0x03a22c (0xcd53 bytes) 000607 1775847377.792984 trunc from 0x040000 to 0x03aeea (0x5115 bytes) 000609 1775847377.814472 MIRROR_RESYNC 7 mirrors 000611 1775847378.666349 mapread 0x027512 thru 0x031e2a (0xa919 bytes) 000613 1775847378.703259 mapwrite 0x02d37c thru 0x032f06 (0x5b8b bytes) 000615 1775847378.839884 read 0x00e85d thru 0x011c42 (0x33e6 bytes) 000616 1775847378.856680 mapwrite 0x0205ff thru 0x027991 (0x7393 bytes) 000617 1775847378.921836 write 0x03b41c thru 0x03ffff (0x4be4 bytes) 000618 1775847378.923997 trunc from 0x040000 to 0x008081 (0x37f7e bytes) 000619 1775847378.942761 mapwrite 0x001b4e thru 0x007b24 (0x5fd7 bytes) 000621 1775847378.995624 mapwrite 0x030ac8 thru 0x03848f (0x79c8 bytes) 000622 1775847379.066533 MIRROR_RESYNC 7 mirrors 000623 1775847379.955428 trunc from 0x038490 to 0x030d3a (0x7755 bytes) 000625 1775847380.025191 read 0x00bfc7 thru 0x00c719 (0x753 bytes) 000626 1775847380.034355 mapwrite 0x015128 thru 0x01bdd4 (0x6cad bytes) 000627 1775847380.104775 trunc from 0x030d3b to 0x0241ac (0xcb8e bytes) 000628 1775847380.142816 mapwrite 0x0398f5 thru 0x03d5a8 (0x3cb4 bytes) 000629 1775847380.252627 trunc from 0x03d5a9 to 0x00d432 (0x30176 bytes) 000631 1775847380.279602 mapwrite 0x03239c thru 0x03c853 (0xa4b8 bytes) 000632 1775847380.372702 read 0x0175ff thru 0x01d0eb (0x5aed bytes) 000633 1775847380.390325 read 0x027a0b thru 0x02fc72 (0x8268 bytes) 000635 1775847380.392111 write 0x027b8c thru 0x02f097 (0x750c bytes) 000636 1775847380.396151 mapread 0x02ee46 thru 0x0364cc (0x7687 bytes) 000640 1775847380.401708 read 0x025a1d thru 0x035707 (0xfceb bytes) 000644 1775847380.406686 read 0x032631 thru 0x033bad (0x157d bytes) 000645 1775847380.408174 trunc from 0x03c854 to 0x00d207 (0x2f64c bytes) 000648 1775847380.431537 mapwrite 0x00ae84 thru 0x00c270 (0x13ed bytes) 000649 1775847380.486946 write 0x00c881 thru 0x00cc45 (0x3c5 bytes) 000650 1775847380.488077 read 0x00c157 thru 0x00d1f9 (0x10a3 bytes) 000651 1775847380.489775 mapwrite 0x01a242 thru 0x0213d0 (0x718f bytes) 000652 1775847380.593738 mapwrite 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000653 1775847380.678094 trunc from 0x040000 to 0x00aa8d (0x35572 bytes) 000654 1775847380.689140 read 0x0067e5 thru 0x00aa8d (0x42a9 bytes) 000656 1775847380.696514 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaa8e): Mirror 5: 0x90b1124f (stale) Mirror 6: 0x90b1124f (stale) Mirror 7: 0x90b1124f (stale) Mirror 8: 0x90b1124f (stale) Mirror 11: 0x90b1124f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa8e): Mirror 5: 0x90b1124f (stale) Mirror 6: 0x90b1124f (stale) Mirror 7: 0x90b1124f (stale) Mirror 8: 0x90b1124f (stale) Mirror 11: 0x90b1124f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa8e: skipped 000660 1775847381.017701 mapwrite 0x029e3a thru 0x036e9c (0xd063 bytes) 000661 1775847381.140500 trunc from 0x036e9d to 0x02036d (0x16b2f bytes) 000662 1775847381.161378 read 0x002933 thru 0x00ef03 (0xc5d1 bytes) 000663 1775847381.196884 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2036e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2036e): Mirror 5: 0x20a124a7 (stale) Mirror 6: 0x20a124a7 (stale) Mirror 7: 0x20a124a7 (stale) Mirror 8: 0x20a124a7 (stale) Mirror 11: 0x20a124a7 (stale) CRC-32 checksum value for chunk [0, 0x2036e): Mirror 1: 0x20a124a7 Mirror 12: 0x20a124a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2036e: skipped 000664 1775847382.098319 mapwrite 0x03b79f thru 0x03ffff (0x4861 bytes) 000665 1775847382.302370 trunc from 0x040000 to 0x0070d0 (0x38f2f bytes) 000667 1775847382.336400 write 0x03bb28 thru 0x03ffff (0x44d8 bytes) 000674 1775847382.343365 write 0x02ae2b thru 0x037a10 (0xcbe6 bytes) 000675 1775847382.359650 mapwrite 0x0373c1 thru 0x03fbf7 (0x8837 bytes) 000677 1775847382.429072 MIRROR_RESYNC 7 mirrors 000679 1775847383.306428 MIRROR_RESYNC 7 mirrors 000680 1775847383.744125 mapwrite 0x008231 thru 0x01148f (0x925f bytes) 000681 1775847383.889746 mapread 0x01c028 thru 0x020068 (0x4041 bytes) 000682 1775847383.900069 write 0x034faa thru 0x0354a4 (0x4fb bytes) 000683 1775847383.931760 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 7: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3eb00aa Mirror 13: 0x3eb00aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1775847384.664300 write 0x0388d3 thru 0x03ffff (0x772d bytes) 000689 1775847384.747690 mapread 0x036629 thru 0x037b08 (0x14e0 bytes) 000692 1775847384.759148 write 0x009652 thru 0x012c17 (0x95c6 bytes) 000697 1775847384.773458 trunc from 0x040000 to 0x0319b8 (0xe647 bytes) 000698 1775847384.796387 trunc from 0x0319b9 to 0x039d24 (0x836c bytes) 000699 1775847384.814171 mapread 0x006dab thru 0x013e9f (0xd0f5 bytes) 000701 1775847384.822966 trunc from 0x039d25 to 0x03f641 (0x591d bytes) 000702 1775847384.840558 trunc from 0x03f642 to 0x02602c (0x19615 bytes) 000704 1775847384.858332 write 0x03e94b thru 0x03ffff (0x16b5 bytes) 000705 1775847384.860227 mapwrite 0x008328 thru 0x009431 (0x110a bytes) 000707 1775847384.938417 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000710 1775847385.494059 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000711 1775847385.820615 write 0x01d5fb thru 0x02304c (0x5a52 bytes) 000712 1775847385.832397 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b987b2 Mirror 14: 0x9b987b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1775847386.668344 mapread 0x00f1aa thru 0x01c784 (0xd5db bytes) 000715 1775847386.745728 mapread 0x03012c thru 0x03a469 (0xa33e bytes) 000718 1775847386.837232 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b987b2 Mirror 14: 0x9b987b2 Mirror 15: 0x9b987b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000720 1775847387.742771 write 0x015685 thru 0x01eff1 (0x996d bytes) 000722 1775847387.852645 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) Mirror 14: 0x9b987b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) Mirror 14: 0x9b987b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000723 1775847388.284684 trunc from 0x040000 to 0x02e9ed (0x11612 bytes) 000727 1775847388.311454 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e9ee): Mirror 6: 0xcd139a6e (stale) Mirror 8: 0xcd139a6e (stale) Mirror 11: 0xcd139a6e (stale) Mirror 13: 0xf4fc8554 (stale) Mirror 14: 0x946531eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9ee): Mirror 6: 0xcd139a6e (stale) Mirror 8: 0xcd139a6e (stale) Mirror 11: 0xcd139a6e (stale) Mirror 13: 0xf4fc8554 (stale) Mirror 14: 0x946531eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9ee: skipped 000728 1775847388.675406 MIRROR_RESYNC 6 mirrors 000730 1775847389.569376 read 0x01983e thru 0x01a5e0 (0xda3 bytes) 000733 1775847389.577363 mapwrite 0x0093a4 thru 0x00ecce (0x592b bytes) 000734 1775847389.710727 mapread 0x015807 thru 0x020e5f (0xb659 bytes) 000735 1775847389.745407 mapwrite 0x021285 thru 0x029d67 (0x8ae3 bytes) 000736 1775847389.829463 mapread 0x0082d8 thru 0x01496d (0xc696 bytes) 000738 1775847389.834148 mapwrite 0x03f65e thru 0x03ffff (0x9a2 bytes) 000739 1775847389.920461 trunc from 0x040000 to 0x00018b (0x3fe74 bytes) 000740 1775847389.942398 read 0x00011e thru 0x00018b (0x6e bytes) 000742 1775847389.955280 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x18c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18c): Mirror 6: 0x64f1bd4c (stale) Mirror 8: 0x64f1bd4c (stale) Mirror 11: 0x64f1bd4c (stale) Mirror 13: 0x64f1bd4c (stale) Mirror 14: 0x64f1bd4c (stale) CRC-32 checksum value for chunk [0, 0x18c): Mirror 1: 0x64f1bd4c Mirror 15: 0x64f1bd4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18c: skipped 000744 1775847390.635483 mapwrite 0x001ad5 thru 0x008394 (0x68c0 bytes) 000745 1775847390.792280 MIRROR_RESYNC 7 mirrors 000747 1775847391.847293 read 0x00225c thru 0x007801 (0x55a6 bytes) 000748 1775847391.856353 write 0x02c9b0 thru 0x038cf6 (0xc347 bytes) 000749 1775847391.927642 write 0x036b7f thru 0x03ffff (0x9481 bytes) 000751 1775847391.931499 mapread 0x00c72e thru 0x017f14 (0xb7e7 bytes) 000753 1775847391.979272 MIRROR_RESYNC 7 mirrors 000754 1775847393.059737 read 0x0152fa thru 0x0228c0 (0xd5c7 bytes) 000755 1775847393.066381 mapread 0x01aede thru 0x01c389 (0x14ac bytes) 000756 1775847393.067129 trunc from 0x040000 to 0x037a9a (0x8565 bytes) 000757 1775847393.108755 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 6: 0x8ea20359 (stale) Mirror 8: 0x8ea20359 (stale) Mirror 11: 0x8ea20359 (stale) Mirror 13: 0x8ea20359 (stale) Mirror 14: 0x8ea20359 (stale) Mirror 15: 0x8ea20359 (stale) CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 1: 0x8ea20359 Mirror 16: 0x8ea20359 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a9b: skipped 000758 1775847393.875307 trunc from 0x037a9b to 0x01566f (0x2242b bytes) 000759 1775847393.979716 read 0x007da4 thru 0x0142bc (0xc519 bytes) 000760 1775847393.987823 trunc from 0x015670 to 0x02b4a4 (0x15e35 bytes) 000761 1775847394.008509 MIRROR_RESYNC 8 mirrors 000763 1775847394.955027 MIRROR_RESYNC 8 mirrors 000764 1775847395.411281 mapwrite 0x02d86b thru 0x031682 (0x3e18 bytes) 000765 1775847395.543848 write 0x006c30 thru 0x015516 (0xe8e7 bytes) 000767 1775847395.558342 trunc from 0x031683 to 0x03ab53 (0x94d1 bytes) 000769 1775847395.575441 mapread 0x00ddd8 thru 0x0175ad (0x97d6 bytes) 000772 1775847395.587432 mapwrite 0x019fea thru 0x024347 (0xa35e bytes) 000774 1775847395.651598 write 0x02a1dc thru 0x0345b6 (0xa3db bytes) 000775 1775847395.666467 read 0x030ab0 thru 0x034844 (0x3d95 bytes) 000776 1775847395.680041 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab54): Mirror 6: 0x415e8c67 (stale) Mirror 8: 0x415e8c67 (stale) Mirror 11: 0x415e8c67 (stale) Mirror 14: 0x415e8c67 (stale) Mirror 15: 0x415e8c67 (stale) Mirror 16: 0x415e8c67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab54): Mirror 6: 0x415e8c67 (stale) Mirror 8: 0x415e8c67 (stale) Mirror 11: 0x415e8c67 (stale) Mirror 14: 0x415e8c67 (stale) Mirror 15: 0x415e8c67 (stale) Mirror 16: 0x415e8c67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab54: skipped 000777 1775847396.180150 mapread 0x002003 thru 0x009b69 (0x7b67 bytes) 000778 1775847396.205044 read 0x00f632 thru 0x01313e (0x3b0d bytes) 000779 1775847396.209361 trunc from 0x03ab54 to 0x008388 (0x327cb bytes) 000780 1775847396.219201 write 0x00bbfc thru 0x019c55 (0xe05a bytes) 000782 1775847396.220724 write 0x00f538 thru 0x012680 (0x3149 bytes) 000784 1775847396.221933 write 0x01484f thru 0x01ce0a (0x85bc bytes) 000786 1775847396.223711 read 0x00a690 thru 0x015ba8 (0xb519 bytes) 000787 1775847396.232761 MIRROR_RESYNC 7 mirrors 000788 1775847397.106548 read 0x003a35 thru 0x004c2d (0x11f9 bytes) 000790 1775847397.113449 write 0x000ac4 thru 0x004ca5 (0x41e2 bytes) 000791 1775847397.179633 write 0x031e40 thru 0x03ffff (0xe1c0 bytes) 000792 1775847397.181449 read 0x02c1ce thru 0x02d2c6 (0x10f9 bytes) 000794 1775847397.193820 read 0x01b4ec thru 0x023557 (0x806c bytes) 000796 1775847397.196596 read 0x0284c3 thru 0x037eab (0xf9e9 bytes) 000798 1775847397.198113 mapread 0x028746 thru 0x029668 (0xf23 bytes) 000800 1775847397.200650 mapwrite 0x028b6c thru 0x031895 (0x8d2a bytes) 000804 1775847397.252084 read 0x039374 thru 0x03a972 (0x15ff bytes) 000805 1775847397.252953 trunc from 0x040000 to 0x01ce21 (0x231de bytes) 000806 1775847397.299042 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ce22): Mirror 6: 0xd2f99587 (stale) Mirror 8: 0xd2f99587 (stale) Mirror 11: 0xd2f99587 (stale) Mirror 15: 0xd2f99587 (stale) Mirror 16: 0xd2f99587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ce22): Mirror 6: 0xd2f99587 (stale) Mirror 8: 0xd2f99587 (stale) Mirror 11: 0xd2f99587 (stale) Mirror 15: 0xd2f99587 (stale) Mirror 16: 0xd2f99587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ce22: skipped 000807 1775847397.648665 mapwrite 0x00ec5f thru 0x0125fc (0x399e bytes) 000809 1775847397.722020 mapread 0x018ee5 thru 0x01c190 (0x32ac bytes) 000810 1775847397.729468 read 0x007a7e thru 0x007e73 (0x3f6 bytes) 000814 1775847397.741368 read 0x01b23e thru 0x01ce21 (0x1be4 bytes) 000817 1775847397.744194 write 0x00bb48 thru 0x00db0f (0x1fc8 bytes) 000818 1775847397.745725 trunc from 0x01ce22 to 0x00d686 (0xf79b bytes) 000819 1775847397.765274 mapread 0x008685 thru 0x00d686 (0x5002 bytes) 000821 1775847397.780309 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd687): Mirror 8: 0x15ff4592 (stale) Mirror 11: 0x15ff4592 (stale) Mirror 15: 0x15ff4592 (stale) Mirror 16: 0x15ff4592 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd687): Mirror 8: 0x15ff4592 (stale) Mirror 11: 0x15ff4592 (stale) Mirror 15: 0x15ff4592 (stale) Mirror 16: 0x15ff4592 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd687: skipped 000825 1775847398.104540 read 0x003fd8 thru 0x00d686 (0x96af bytes) 000826 1775847398.111399 trunc from 0x00d687 to 0x03b99e (0x2e318 bytes) 000827 1775847398.124035 mapwrite 0x0319bb thru 0x03443d (0x2a83 bytes) 000828 1775847398.185058 mapwrite 0x01b06e thru 0x020dd7 (0x5d6a bytes) 000829 1775847398.228955 mapread 0x0135e2 thru 0x017437 (0x3e56 bytes) 000831 1775847398.240966 trunc from 0x03b99f to 0x003cdf (0x37cbf bytes) 000832 1775847398.264032 trunc from 0x003ce0 to 0x0009d7 (0x3308 bytes) 000833 1775847398.290670 trunc from 0x0009d8 to 0x026f42 (0x2656b bytes) 000834 1775847398.309492 trunc from 0x026f43 to 0x00c1de (0x1ad64 bytes) 000836 1775847398.321699 read 0x0078e8 thru 0x00c1de (0x48f7 bytes) 000837 1775847398.335687 mapread 0x005c1a thru 0x00c1de (0x65c5 bytes) 000838 1775847398.337200 mapread 0x00bc5f thru 0x00c1de (0x580 bytes) 000839 1775847398.346148 mapread 0x0062ba thru 0x009787 (0x34ce bytes) 000840 1775847398.354129 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc1df): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc1df): Mirror 8: 0xc9acd4c8 (stale) Mirror 11: 0xc9acd4c8 (stale) Mirror 15: 0xc9acd4c8 (stale) Mirror 16: 0xc9acd4c8 (stale) CRC-32 checksum value for chunk [0, 0xc1df): Mirror 1: 0x5de5ca47 Mirror 17: 0x5de5ca47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc1df: skipped 000841 1775847399.032439 mapread 0x000b0e thru 0x007439 (0x692c bytes) 000843 1775847399.065086 mapread 0x00109e thru 0x0031d9 (0x213c bytes) 000845 1775847399.076180 trunc from 0x00c1df to 0x02cd61 (0x20b83 bytes) 000846 1775847399.227276 mapwrite 0x028d1c thru 0x02dafc (0x4de1 bytes) 000847 1775847399.341522 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dafd): Mirror 11: 0x70b2e605 (stale) Mirror 15: 0x70b2e605 (stale) Mirror 16: 0x70b2e605 (stale) Mirror 17: 0x30e5393a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dafd): Mirror 11: 0x70b2e605 (stale) Mirror 15: 0x70b2e605 (stale) Mirror 16: 0x70b2e605 (stale) Mirror 17: 0x30e5393a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dafd: skipped 000849 1775847399.760958 write 0x011ccb thru 0x020b69 (0xee9f bytes) 000850 1775847399.769752 mapread 0x013999 thru 0x0201b7 (0xc81f bytes) 000851 1775847399.772335 trunc from 0x02dafd to 0x013d03 (0x19df9 bytes) 000852 1775847399.788837 mapwrite 0x03de68 thru 0x03ffff (0x2198 bytes) 000853 1775847399.858061 read 0x00938c thru 0x012ff2 (0x9c67 bytes) 000856 1775847399.869377 write 0x003a96 thru 0x00af90 (0x74fb bytes) 000857 1775847399.882738 read 0x0127da thru 0x0137fd (0x1024 bytes) 000858 1775847399.884542 trunc from 0x040000 to 0x021fa3 (0x1e05c bytes) 000860 1775847399.914568 mapwrite 0x029c6b thru 0x032c88 (0x901e bytes) 000861 1775847400.067423 mapwrite 0x015d77 thru 0x0229ac (0xcc36 bytes) 000863 1775847400.136200 mapread 0x0268df thru 0x026a1a (0x13c bytes) 000865 1775847400.137555 write 0x01f7db thru 0x023a46 (0x426c bytes) 000866 1775847400.141609 mapwrite 0x02b731 thru 0x0344b1 (0x8d81 bytes) 000868 1775847400.206087 trunc from 0x0344b2 to 0x00a29d (0x2a214 bytes) 000870 1775847400.233217 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa29e): Mirror 11: 0x70884a18 (stale) Mirror 15: 0x70884a18 (stale) Mirror 16: 0x70884a18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa29e): Mirror 11: 0x70884a18 (stale) Mirror 15: 0x70884a18 (stale) Mirror 16: 0x70884a18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa29e: skipped 000871 1775847400.496872 mapwrite 0x00c082 thru 0x01618f (0xa10e bytes) 000872 1775847400.589701 mapwrite 0x03e0eb thru 0x03ffff (0x1f15 bytes) 000875 1775847400.658326 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf9180896 (stale) Mirror 15: 0xf9180896 (stale) Mirror 16: 0xf9180896 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79e928ed Mirror 17: 0x79e928ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000876 1775847401.419167 mapread 0x03fe7d thru 0x03ffff (0x183 bytes) 000877 1775847401.432223 trunc from 0x040000 to 0x0136ad (0x2c952 bytes) 000879 1775847401.568727 mapwrite 0x023d8a thru 0x030f8a (0xd201 bytes) 000880 1775847401.692851 trunc from 0x030f8b to 0x01098e (0x205fc bytes) 000882 1775847401.726634 read 0x007021 thru 0x01098e (0x996e bytes) 000883 1775847401.745342 trunc from 0x01098f to 0x018e3b (0x84ad bytes) 000887 1775847401.763703 write 0x028786 thru 0x038740 (0xffbb bytes) 000888 1775847401.768570 read 0x004cd8 thru 0x011470 (0xc799 bytes) 000890 1775847401.789831 read 0x00aad2 thru 0x0154e4 (0xaa13 bytes) 000892 1775847401.798892 MIRROR_RESYNC 5 mirrors 000893 1775847402.635514 MIRROR_RESYNC 5 mirrors 000895 1775847402.995522 trunc from 0x038741 to 0x0095c6 (0x2f17a bytes) 000897 1775847403.069634 mapread 0x008147 thru 0x0095c6 (0x1480 bytes) 000898 1775847403.080675 write 0x023b2c thru 0x028141 (0x4616 bytes) 000900 1775847403.082454 write 0x016bf8 thru 0x02601b (0xf424 bytes) 000902 1775847403.096360 trunc from 0x028142 to 0x013fad (0x14194 bytes) 000904 1775847403.119773 mapwrite 0x0292c9 thru 0x036aac (0xd7e4 bytes) 000905 1775847403.183341 trunc from 0x036aad to 0x00482b (0x32281 bytes) 000907 1775847403.196670 write 0x00bebb thru 0x010e3e (0x4f84 bytes) 000909 1775847403.198566 trunc from 0x010e3f to 0x036bd2 (0x25d94 bytes) 000910 1775847403.210496 read 0x006160 thru 0x00e747 (0x85e8 bytes) 000911 1775847403.215777 write 0x00117c thru 0x00bee6 (0xad6b bytes) 000912 1775847403.221003 trunc from 0x036bd3 to 0x030833 (0x639f bytes) 000913 1775847403.231696 mapread 0x00d68e thru 0x01503d (0x79b0 bytes) 000915 1775847403.240415 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 11: 0x856f2f18 (stale) Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 11: 0x856f2f18 (stale) Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000916 1775847403.557716 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000919 1775847403.819417 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) CRC-32 checksum value for chunk [0, 0x30834): Mirror 1: 0x942ec6af Mirror 17: 0x942ec6af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000920 1775847404.649181 mapread 0x014e82 thru 0x01e8fb (0x9a7a bytes) 000921 1775847404.724185 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 16: 0x856f2f18 (stale) CRC-32 checksum value for chunk [0, 0x30834): Mirror 1: 0x942ec6af Mirror 17: 0x942ec6af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000922 1775847405.304962 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000923 1775847405.703129 trunc from 0x030834 to 0x02385b (0xcfd8 bytes) 000924 1775847405.776586 write 0x035e0e thru 0x03ffff (0xa1f2 bytes) 000925 1775847405.777920 mapwrite 0x02f446 thru 0x0318da (0x2495 bytes) 000928 1775847405.814938 mapwrite 0x0235bc thru 0x024375 (0xdba bytes) 000930 1775847405.869007 mapwrite 0x00523a thru 0x00d421 (0x81e8 bytes) 000931 1775847405.924840 mapread 0x011594 thru 0x01c231 (0xac9e bytes) 000934 1775847405.942560 read 0x035c07 thru 0x03ffff (0xa3f9 bytes) 000938 1775847405.946363 trunc from 0x040000 to 0x00dcaf (0x32350 bytes) 000941 1775847405.982451 write 0x0375a4 thru 0x03ffff (0x8a5c bytes) 000942 1775847405.995268 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) 000944 1775847406.128220 write 0x025890 thru 0x031280 (0xb9f1 bytes) 000946 1775847406.136117 trunc from 0x040000 to 0x01f5d8 (0x20a27 bytes) 000949 1775847406.170681 read 0x0037ce thru 0x00d59d (0x9dd0 bytes) 000950 1775847406.187024 mapread 0x0015cd thru 0x005ad1 (0x4505 bytes) 000951 1775847406.222707 MIRROR_EXTEND to 2 mirrors 000954 1775847406.692124 MIRROR_EXTEND to 3 mirrors 000955 1775847407.198578 mapwrite 0x0022e6 thru 0x00f508 (0xd223 bytes) 000956 1775847407.364612 mapread 0x01a70c thru 0x01eb45 (0x443a bytes) 000959 1775847407.375240 mapwrite 0x03a992 thru 0x03ffff (0x566e bytes) 000966 1775847407.437536 trunc from 0x040000 to 0x027fa9 (0x18056 bytes) 000967 1775847407.447063 read 0x01083a thru 0x0157ed (0x4fb4 bytes) 000969 1775847407.452838 mapread 0x002b0b thru 0x00340e (0x904 bytes) 000970 1775847407.454100 mapread 0x0153eb thru 0x024a85 (0xf69b bytes) 000974 1775847407.463316 mapread 0x00d688 thru 0x013e9f (0x6818 bytes) 000976 1775847407.464682 mapread 0x006026 thru 0x008b44 (0x2b1f bytes) 000977 1775847407.469433 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27faa): Mirror 3: 0xee8b9eb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27faa): Mirror 3: 0xee8b9eb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27faa: skipped 000978 1775847407.681475 write 0x03896c thru 0x03ffff (0x7694 bytes) 000981 1775847407.685723 mapread 0x003935 thru 0x004782 (0xe4e bytes) 000982 1775847407.697672 mapread 0x030096 thru 0x03f0ea (0xf055 bytes) 000983 1775847407.709339 mapwrite 0x000012 thru 0x009b63 (0x9b52 bytes) 000986 1775847407.761484 trunc from 0x040000 to 0x003e16 (0x3c1e9 bytes) 000987 1775847407.791691 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e17): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e17): Mirror 3: 0xb26403a (stale) CRC-32 checksum value for chunk [0, 0x3e17): Mirror 1: 0x39a923bd Mirror 4: 0x39a923bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e17: skipped 000988 1775847408.407509 mapread 0x00091e thru 0x003e16 (0x34f9 bytes) 000989 1775847408.439314 mapread 0x0002ad thru 0x0036a7 (0x33fb bytes) 000990 1775847408.449731 mapwrite 0x032521 thru 0x03a661 (0x8141 bytes) 000992 1775847408.562276 mapread 0x022cb4 thru 0x023d12 (0x105f bytes) 000997 1775847408.575638 MIRROR_RESYNC 3 mirrors 000999 1775847409.131855 write 0x0260d3 thru 0x032086 (0xbfb4 bytes) All operations completed A-OK! PASS 70a (68s) == sanity-flr test 71: check mirror extend parent fid ==== 14:56:51 (1775847411) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 14:56:53 (1775847413) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147318 s, 14.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120726 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3ec:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3ec:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3e9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 14:56:59 (1775847419) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8868864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6567072 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7664928 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8333984 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8355168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9230752 .. Extending file size to 9276512 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56733 56734 56735 56736 56741 done 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (71s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:58:10 (1775847490) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5733792 .. Extending file size to 956672 .. failed Extending file size to 1255776 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7057376 .. Extending file size to 8010720 .. Extending file size to 7860256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8092608 .. Extending file size to 8598656 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8575136 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8646624 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8659936 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8818912 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58722 58723 58724 58728 58729 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:59:18 (1775847558) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0513373 s, 40.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1451 s, 14.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x59c01168 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:59:27 (1775847567) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0531377 s, 39.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 14:59:31 (1775847571) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x31e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x402:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x402:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x403:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x3fe:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:59:35 (1775847575) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0991705 s, 21.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x404:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x400:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:59:39 (1775847579) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13737 s, 15.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x405:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x406:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x401:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x405:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x406:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x401:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x402:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:59:44 (1775847584) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168863 s, 12.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x407:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x403:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x407:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x403:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x404:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:00:11 (1775847611) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603851 s, 17.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x408:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x406:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x405:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x408:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x406:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x405:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x409:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:01:02 (1775847662) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x407:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x408:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0867165 s, 24.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x407:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x408:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x407:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x408:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:01:51 (1775847711) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. Creating new pool test_204f oleg108-server: Pool lustre.test_204f created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169858 s, 12.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x409:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x409:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg108-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:03:06 (1775847786) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:03:13 (1775847793) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:03:22 (1775847802) oleg108-server: Pool lustre.test_206 created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg108-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:03:45 (1775847825) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711468 s, 14.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309292 s, 33.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xceb579d Mirror 3: 0xceb579d PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:03:53 (1775847833) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:03:55 (1775847835) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:03:58 (1775847838) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.10705 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1830000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 209a (26s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:04:24 (1775847864) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0085048 s, 963 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:04:28 (1775847868) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280165 s, 37.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x343 lmm_fid: [0x200000404:0x343:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x343 lmm_fid: [0x200000404:0x343:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x415:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834755 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-10 15:04:29.000000000 -0400 Modify: 2026-04-10 15:04:29.000000000 -0400 Change: 2026-04-10 15:04:30.000000000 -0400 Birth: 2026-04-10 15:04:28.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:04:33 (1775847873) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:04:35 (1775847875) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.311599 s, 131 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0375157 s, 109 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 10 15:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x345:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 10 15:04 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:04:40 (1775847880) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:04:44 (1775847884) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1163 sec =========== 15:04:50 (1775847890) === sanity-flr: start cleanup 15:04:51 (1775847891) === === sanity-flr: finish cleanup 15:04:55 (1775847895) ===