-----============= acceptance-small: sanity-flr ============----- Wed Jul 1 13:42:21 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg218-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:42:27 (1782927747) === oleg218-client.virtnet: executing check_config_client /mnt/lustre oleg218-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg218-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9490460e5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9490460e5800.idle_timeout=debug disable quota as required oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:42:37 (1782927757) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7571] == sanity-flr test 0a: lfs mirror create with -N option == 13:42:38 (1782927758) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:42:42 (1782927762) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:42:43 (1782927763) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:42:44 (1782927764) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:42:51 (1782927771) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:42:52 (1782927772) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 13:42:53 (1782927773) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9490460e5800.stats=clear osc.lustre-OST0001-osc-ffff9490460e5800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:42:55 (1782927775) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:42:58 (1782927778) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:43:01 (1782927781) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:43:02 (1782927782) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:43:04 (1782927784) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 13:43:06 (1782927786) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:43:08 (1782927788) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:43:09 (1782927789) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:43:11 (1782927791) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:43:14 (1782927794) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:43:15 (1782927795) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.405611 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276396 s, 37.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:43:19 (1782927799) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0794167 s, 106 MB/s PASS 22 (2s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:43:21 (1782927801) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:43:24 (1782927804) Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:43:46 (1782927826) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0733545 s, 57.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:44:00 (1782927840) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (22s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:44:22 (1782927862) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14792 s, 45.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 1 13:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.217s user 0m0.001s sys 0m0.134s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.243s user 0m0.001s sys 0m0.151s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.246s user 0m0.001s sys 0m0.146s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490460e5800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:44:50 (1782927890) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:44:51 (1782927891) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:44:52 (1782927892) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:44:53 (1782927893) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 13:44:55 (1782927895) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.143261 s, 87.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:44:58 (1782927898) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111019 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152252 s, 68.9 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:45:00 (1782927900) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:45:04 (1782927904) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000713733 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000555859 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000575771 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000590057 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 13:45:06 (1782927906) create test files with size 15744818 .. cb56ac035a5276be71ef35c9ea9e183f - 9f717625a98f1a7b7801872f05c31230 - 479d061675cc6939ce7e90281d9f7b08 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 13:45:12 (1782927912) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:45:16 (1782927916) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:45:18 (1782927918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337961 s, 31.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103027 s, 102 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 13:45:21 (1782927921) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0318489 s, 65.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0943944 s, 44.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0368111 s, 57.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 13:45:24 (1782927924) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:45:25 (1782927925) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:45:26 (1782927926) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889768 s, 118 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 13:45:28 (1782927928) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:45:29 (1782927929) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:45:32 (1782927932) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0980446 s, 107 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:45:54 (1782927954) llite.lustre-ffff949044c0d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0739597 s, 142 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff949046026000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:45:58 (1782927958) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:46:09 (1782927969) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:46:20 (1782927980) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:46:21 (1782927981) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0684484 s, 46.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0607997 s, 51.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4b74bb6f (nosync,stale) Mirror 2: 0x4b74bb6f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7141843c1b60041081fdc272baef1a6e - mirror 1 checksum: aeda580a890329fa2489960b0b43ea10 - mirror 2 checksum: 7141843c1b60041081fdc272baef1a6e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-01T13:46:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-01T13:46:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7141843c1b60041081fdc272baef1a6e - mirror 1 checksum: aeda580a890329fa2489960b0b43ea10 - mirror 2 checksum: aeda580a890329fa2489960b0b43ea10 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-01T13:46:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-01T13:46:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:46:24 (1782927984) Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 13:46:26 (1782927986) shut down facet: mds1 facet_host: oleg218-server facet_failover_host: oleg218-server Failover mds1 to oleg218-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 13:46:40 (1782928000) targets are mounted 13:46:40 (1782928000) facet_failover done oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113382 s, 92.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (23s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:46:47 (1782928007) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:46:48 (1782928008) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:46:49 (1782928009) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00247245 s, 6.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:46:51 (1782928011) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:46:52 (1782928012) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:46:53 (1782928013) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:46:59 (1782928019) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:47:17 (1782928037) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:47:35 (1782928055) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:47:53 (1782928073) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 13:47:55 (1782928075) mirror create pid 53641 mirror split pid 53644 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53641 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53644 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 13:48:57 (1782928137) Seed set to 8138 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000011 1782928138.215694 mapwrite 0x0179c9 thru 0x01b128 (0x3760 bytes) 000012 1782928138.239996 mapread 0x000373 thru 0x0093d9 (0x9067 bytes) 000015 1782928138.245184 mapread 0x00510f thru 0x012370 (0xd262 bytes) truncating to largest ever: 0x1a24b 000017 1782928138.248340 trunc from 0x01b129 to 0x01a24a (0xede bytes) 000019 1782928138.254599 mapwrite 0x03c849 thru 0x03d93a (0x10f2 bytes) 000020 1782928138.275144 mapread 0x000999 thru 0x0054e4 (0x4b4c bytes) 000021 1782928138.276334 read 0x009d6e thru 0x00e26f (0x4502 bytes) 000024 1782928138.278338 write 0x0027d0 thru 0x011943 (0xf174 bytes) 000025 1782928138.279894 read 0x01ea13 thru 0x02cbc2 (0xe1b0 bytes) 000030 1782928138.287579 write 0x030071 thru 0x0325aa (0x253a bytes) 000031 1782928138.288679 mapwrite 0x03e3b5 thru 0x03ffff (0x1c4b bytes) truncating to largest ever: 0x34b27 000036 1782928138.309683 trunc from 0x040000 to 0x034b26 (0xb4d9 bytes) 000037 1782928138.314130 mapread 0x001a68 thru 0x00f862 (0xddfb bytes) 000039 1782928138.315691 write 0x01a42b thru 0x01ef55 (0x4b2b bytes) 000040 1782928138.317886 MIRROR_EXTEND to 2 mirrors 000046 1782928138.478548 mapread 0x023016 thru 0x02aab6 (0x7aa1 bytes) 000047 1782928138.486145 read 0x033ca0 thru 0x034b26 (0xe87 bytes) 000050 1782928138.491298 mapwrite 0x0256ba thru 0x02590f (0x256 bytes) 000051 1782928138.538236 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000054 1782928138.572865 mapwrite 0x03da99 thru 0x03e09f (0x607 bytes) 000055 1782928138.595800 mapwrite 0x0093f3 thru 0x019131 (0xfd3f bytes) 000060 1782928138.618612 read 0x0132c2 thru 0x01c6aa (0x93e9 bytes) 000061 1782928138.621089 write 0x02df2d thru 0x034e10 (0x6ee4 bytes) 000062 1782928138.624386 read 0x00dec3 thru 0x010a60 (0x2b9e bytes) 000064 1782928138.625590 read 0x02c5bb thru 0x033b79 (0x75bf bytes) 000066 1782928138.629734 write 0x02aab2 thru 0x02f06d (0x45bc bytes) 000068 1782928138.632588 MIRROR_EXTEND to 2 mirrors 000070 1782928138.805034 trunc from 0x03e0a0 to 0x02bd9b (0x12304 bytes) 000072 1782928138.829230 read 0x011643 thru 0x0117b7 (0x175 bytes) 000074 1782928138.831672 mapread 0x012c33 thru 0x015c78 (0x3046 bytes) 000076 1782928138.837333 trunc from 0x02bd9c to 0x00d7c7 (0x1e5d4 bytes) 000077 1782928138.843073 write 0x01548a thru 0x0231f4 (0xdd6b bytes) 000078 1782928138.844848 write 0x03593c thru 0x03ffff (0xa6c4 bytes) truncating to largest ever: 0x3e007 000079 1782928138.845798 trunc from 0x040000 to 0x03e006 (0x1ff9 bytes) 000080 1782928138.850407 read 0x039717 thru 0x03e006 (0x48f0 bytes) 000082 1782928138.852588 MIRROR_RESYNC 2 mirrors 000083 1782928139.016234 mapread 0x02db99 thru 0x03b883 (0xdceb bytes) 000084 1782928139.023766 write 0x0234e1 thru 0x02c608 (0x9128 bytes) 000085 1782928139.045235 read 0x03d0a0 thru 0x03e006 (0xf67 bytes) 000087 1782928139.047688 write 0x01bf2e thru 0x02180a (0x58dd bytes) 000090 1782928139.052421 write 0x0073a2 thru 0x0089a0 (0x15ff bytes) 000091 1782928139.055420 mapwrite 0x034e9b thru 0x038f0f (0x4075 bytes) 000093 1782928139.073402 mapwrite 0x03d3b3 thru 0x03f163 (0x1db1 bytes) 000096 1782928139.098803 mapwrite 0x00bb41 thru 0x01601b (0xa4db bytes) 000097 1782928139.118284 read 0x02c721 thru 0x03668e (0x9f6e bytes) 000098 1782928139.120380 MIRROR_RESYNC 2 mirrors 000103 1782928139.277460 mapread 0x03bda1 thru 0x03c054 (0x2b4 bytes) 000104 1782928139.280479 trunc from 0x03f164 to 0x02679c (0x189c7 bytes) 000106 1782928139.302171 trunc from 0x02679d to 0x01edfc (0x79a0 bytes) 000108 1782928139.307622 mapread 0x011f6a thru 0x01d7b7 (0xb84e bytes) 000109 1782928139.312591 mapread 0x01e7f7 thru 0x01edfc (0x606 bytes) 000110 1782928139.313196 mapwrite 0x023ab7 thru 0x026b43 (0x308d bytes) 000112 1782928139.331471 trunc from 0x026b44 to 0x012d83 (0x13dc0 bytes) 000113 1782928139.336417 write 0x008d4f thru 0x00fe00 (0x70b2 bytes) 000114 1782928139.339007 write 0x001e11 thru 0x00373d (0x192d bytes) 000115 1782928139.343674 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000116 1782928139.381432 write 0x01e3e7 thru 0x02734d (0x8f67 bytes) 000117 1782928139.382405 mapread 0x004f96 thru 0x013e55 (0xeec0 bytes) 000122 1782928139.391287 trunc from 0x02734e to 0x02a898 (0x354b bytes) 000123 1782928139.395605 trunc from 0x02a899 to 0x026249 (0x464f bytes) 000124 1782928139.399738 trunc from 0x02624a to 0x02c544 (0x62fb bytes) 000125 1782928139.403725 read 0x027cdd thru 0x028a6c (0xd90 bytes) 000126 1782928139.405774 trunc from 0x02c545 to 0x007dfa (0x2474a bytes) 000127 1782928139.410655 read 0x001eb7 thru 0x007dfa (0x5f44 bytes) 000128 1782928139.411231 read 0x002260 thru 0x007dfa (0x5b9b bytes) 000130 1782928139.413545 MIRROR_EXTEND to 2 mirrors 000134 1782928139.565957 write 0x00b1a3 thru 0x011f2c (0x6d8a bytes) 000135 1782928139.584889 trunc from 0x011f2d to 0x0049f3 (0xd539 bytes) 000136 1782928139.595108 read 0x0028c0 thru 0x0049f3 (0x2134 bytes) 000137 1782928139.597465 read 0x0020ad thru 0x0049f3 (0x2947 bytes) 000139 1782928139.598652 write 0x034319 thru 0x037a28 (0x3710 bytes) 000143 1782928139.600554 read 0x0034f0 thru 0x00607b (0x2b8c bytes) 000144 1782928139.604213 write 0x011083 thru 0x018478 (0x73f6 bytes) 000145 1782928139.607077 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37a29): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a29): Mirror 2: 0x9e84707a (stale) CRC-32 checksum value for chunk [0, 0x37a29): Mirror 1: 0xdeff9a32 Mirror 3: 0xdeff9a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a29: skipped 000146 1782928139.817455 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37a29): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a29): Mirror 2: 0x9e84707a (stale) CRC-32 checksum value for chunk [0, 0x37a29): Mirror 1: 0xdeff9a32 Mirror 3: 0xdeff9a32 Mirror 4: 0xdeff9a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a29: skipped 000147 1782928140.024636 write 0x001fa9 thru 0x011d88 (0xfde0 bytes) 000150 1782928140.046857 mapwrite 0x02f6cb thru 0x032f6d (0x38a3 bytes) 000151 1782928140.066186 read 0x02704f thru 0x035f0c (0xeebe bytes) 000154 1782928140.069384 mapwrite 0x018b20 thru 0x0239dd (0xaebe bytes) 000158 1782928140.088960 mapread 0x01fdd9 thru 0x02e274 (0xe49c bytes) 000161 1782928140.090966 mapwrite 0x02cda6 thru 0x037192 (0xa3ed bytes) 000162 1782928140.107937 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a29): Mirror 3: 0xdeff9a32 (stale) Mirror 4: 0xdeff9a32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a29): Mirror 3: 0xdeff9a32 (stale) Mirror 4: 0xdeff9a32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a29: skipped 000163 1782928140.198844 trunc from 0x037a29 to 0x010aa2 (0x26f86 bytes) 000164 1782928140.207885 trunc from 0x010aa3 to 0x013362 (0x28c0 bytes) 000165 1782928140.212293 read 0x00018d thru 0x008b67 (0x89db bytes) 000166 1782928140.215370 mapread 0x01281a thru 0x013362 (0xb49 bytes) 000169 1782928140.219438 trunc from 0x013363 to 0x03a389 (0x27027 bytes) 000171 1782928140.223978 write 0x00bc29 thru 0x016ee0 (0xb2b8 bytes) 000174 1782928140.228559 read 0x000ae9 thru 0x00b0ee (0xa606 bytes) 000178 1782928140.232921 write 0x026739 thru 0x02e146 (0x7a0e bytes) 000180 1782928140.234209 mapwrite 0x0027ef thru 0x00d891 (0xb0a3 bytes) 000182 1782928140.248818 write 0x02d48f thru 0x0323de (0x4f50 bytes) 000185 1782928140.250366 mapwrite 0x0055d4 thru 0x008eb8 (0x38e5 bytes) 000186 1782928140.262792 trunc from 0x03a38a to 0x020a65 (0x19924 bytes) 000188 1782928140.267796 read 0x00c466 thru 0x011b47 (0x56e2 bytes) 000189 1782928140.268289 read 0x0057ac thru 0x012e66 (0xd6bb bytes) 000190 1782928140.268717 mapread 0x01d712 thru 0x020a65 (0x3354 bytes) 000192 1782928140.269781 write 0x013737 thru 0x01e98e (0xb258 bytes) 000195 1782928140.272695 MIRROR_RESYNC 3 mirrors 000196 1782928140.457624 write 0x03b217 thru 0x03ffff (0x4de9 bytes) 000198 1782928140.473976 mapread 0x01192d thru 0x01856b (0x6c3f bytes) 000199 1782928140.479712 mapread 0x03b642 thru 0x03ffff (0x49be bytes) 000201 1782928140.483249 MIRROR_RESYNC 3 mirrors 000202 1782928140.656440 mapwrite 0x02564f thru 0x02f507 (0x9eb9 bytes) 000203 1782928140.707503 mapread 0x038f68 thru 0x03d867 (0x4900 bytes) 000205 1782928140.711029 write 0x03c2ea thru 0x03ffff (0x3d16 bytes) truncating to largest ever: 0x3fb2d 000208 1782928140.712507 trunc from 0x040000 to 0x03fb2c (0x4d3 bytes) 000209 1782928140.722374 mapwrite 0x0309ad thru 0x03b3fd (0xaa51 bytes) 000210 1782928140.739168 trunc from 0x03fb2d to 0x0281f4 (0x17938 bytes) 000211 1782928140.745113 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: 0x6a323740 (stale) Mirror 4: 0x6a323740 (stale) CRC-32 checksum value for chunk [0, 0x281f5): Mirror 1: 0x454efcd3 Mirror 5: 0x454efcd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x281f5: skipped 000212 1782928140.956326 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: 0x6a323740 (stale) Mirror 4: 0x6a323740 (stale) CRC-32 checksum value for chunk [0, 0x281f5): Mirror 1: 0x454efcd3 Mirror 5: 0x454efcd3 Mirror 6: 0x454efcd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x281f5: skipped 000214 1782928141.188039 write 0x004a50 thru 0x00829a (0x384b bytes) 000216 1782928141.209602 write 0x01d437 thru 0x024336 (0x6f00 bytes) 000220 1782928141.214005 trunc from 0x0281f5 to 0x009781 (0x1ea73 bytes) 000221 1782928141.223807 read 0x007a17 thru 0x009781 (0x1d6b bytes) 000222 1782928141.227344 MIRROR_RESYNC 5 mirrors 000223 1782928141.444449 trunc from 0x009782 to 0x003cb2 (0x5acf bytes) 000225 1782928141.468961 mapread 0x001330 thru 0x003cb2 (0x2983 bytes) 000226 1782928141.475258 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb3): Mirror 3: 0xcfb51072 (stale) Mirror 4: 0xcfb51072 (stale) Mirror 6: 0xcfb51072 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3): Mirror 3: 0xcfb51072 (stale) Mirror 4: 0xcfb51072 (stale) Mirror 6: 0xcfb51072 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3: skipped 000227 1782928141.553020 write 0x03fe8e thru 0x03ffff (0x172 bytes) 000230 1782928141.554475 read 0x001340 thru 0x0110f9 (0xfdba bytes) 000231 1782928141.556937 read 0x02b0ed thru 0x02b166 (0x7a bytes) 000233 1782928141.561393 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa7b74782 (stale) Mirror 6: 0xa7b74782 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa7b74782 (stale) Mirror 6: 0xa7b74782 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000234 1782928141.635884 mapwrite 0x0283b4 thru 0x035377 (0xcfc4 bytes) 000235 1782928141.656872 mapwrite 0x0196ed thru 0x020521 (0x6e35 bytes) 000236 1782928141.673666 trunc from 0x040000 to 0x008364 (0x37c9b bytes) 000238 1782928141.678880 trunc from 0x008365 to 0x020055 (0x17cf1 bytes) 000239 1782928141.682858 trunc from 0x020056 to 0x02123b (0x11e6 bytes) 000242 1782928141.688567 MIRROR_RESYNC 3 mirrors 000244 1782928141.858805 read 0x0121af thru 0x0178f8 (0x574a bytes) 000245 1782928141.862262 MIRROR_RESYNC 3 mirrors 000246 1782928141.969527 MIRROR_EXTEND to 4 mirrors 000247 1782928142.134260 mapread 0x00ae91 thru 0x00b92f (0xa9f bytes) 000248 1782928142.137356 write 0x035e8b thru 0x0369b6 (0xb2c bytes) 000249 1782928142.153274 read 0x032fc8 thru 0x0369b6 (0x39ef bytes) 000250 1782928142.155381 mapread 0x00c01a thru 0x019948 (0xd92f bytes) 000253 1782928142.162458 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x369b7): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369b7): Mirror 4: 0x3dbdaa45 (stale) Mirror 6: 0x3dbdaa45 (stale) Mirror 7: 0x3dbdaa45 (stale) CRC-32 checksum value for chunk [0, 0x369b7): Mirror 1: 0x359ffc36 Mirror 8: 0x359ffc36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369b7: skipped 000254 1782928142.383466 trunc from 0x0369b7 to 0x030d80 (0x5c36 bytes) 000255 1782928142.409709 write 0x021958 thru 0x02fdb2 (0xe45b bytes) 000257 1782928142.413741 read 0x024130 thru 0x02db0e (0x99df bytes) 000259 1782928142.414709 mapwrite 0x028407 thru 0x02ee2d (0x6a27 bytes) 000261 1782928142.430445 write 0x03da7c thru 0x03ffff (0x2584 bytes) 000265 1782928142.432684 read 0x023ec1 thru 0x0334b2 (0xf5f2 bytes) 000266 1782928142.436519 read 0x025370 thru 0x0263b6 (0x1047 bytes) 000269 1782928142.438189 mapwrite 0x025095 thru 0x02ee8e (0x9dfa bytes) 000270 1782928142.452933 write 0x00c5ef thru 0x0158ea (0x92fc bytes) 000272 1782928142.454231 trunc from 0x040000 to 0x0225b5 (0x1da4a bytes) 000273 1782928142.461172 mapwrite 0x03f860 thru 0x03ffff (0x7a0 bytes) 000275 1782928142.481994 mapwrite 0x02a492 thru 0x037e0f (0xd97e bytes) 000276 1782928142.500089 write 0x02a1cf thru 0x039577 (0xf3a9 bytes) 000277 1782928142.502871 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5863cdc1 (stale) Mirror 6: 0x5863cdc1 (stale) Mirror 7: 0x5863cdc1 (stale) Mirror 8: 0x459e24c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x894eee18 Mirror 9: 0x894eee18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000279 1782928142.732437 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5863cdc1 (stale) Mirror 6: 0x5863cdc1 (stale) Mirror 7: 0x5863cdc1 (stale) Mirror 8: 0x459e24c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x894eee18 Mirror 9: 0x894eee18 Mirror 10: 0x894eee18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1782928142.963529 trunc from 0x040000 to 0x037ed3 (0x812c bytes) 000281 1782928142.991243 mapread 0x00621a thru 0x00ffa1 (0x9d88 bytes) 000283 1782928142.997256 mapread 0x01a702 thru 0x01c70e (0x200d bytes) 000284 1782928142.999955 trunc from 0x037ed4 to 0x03b2de (0x340b bytes) 000285 1782928143.005871 read 0x0361cc thru 0x03776c (0x15a1 bytes) 000286 1782928143.009179 MIRROR_RESYNC 7 mirrors 000287 1782928143.307298 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000289 1782928143.410218 read 0x01dea3 thru 0x0243e0 (0x653e bytes) 000290 1782928143.412784 trunc from 0x03b2df to 0x03a597 (0xd47 bytes) 000291 1782928143.434756 trunc from 0x03a598 to 0x02ebd3 (0xb9c4 bytes) 000294 1782928143.440777 trunc from 0x02ebd4 to 0x03e69d (0xfaca bytes) 000295 1782928143.445113 mapwrite 0x033874 thru 0x0351d1 (0x195e bytes) 000296 1782928143.464104 trunc from 0x03e69e to 0x02a195 (0x14508 bytes) 000298 1782928143.470215 write 0x005058 thru 0x0136e7 (0xe690 bytes) 000301 1782928143.476862 read 0x00096c thru 0x009c2c (0x92c1 bytes) 000303 1782928143.480470 trunc from 0x02a196 to 0x03f276 (0x150e1 bytes) 000304 1782928143.485371 trunc from 0x03f277 to 0x01374c (0x2bb2a bytes) 000305 1782928143.491819 MIRROR_RESYNC 6 mirrors 000306 1782928143.731335 mapwrite 0x028d50 thru 0x031806 (0x8ab7 bytes) 000309 1782928143.771788 mapwrite 0x00b352 thru 0x01a70c (0xf3bb bytes) 000310 1782928143.791251 mapwrite 0x01bcb8 thru 0x029b83 (0xdecc bytes) 000311 1782928143.809558 trunc from 0x031807 to 0x0254ee (0xc318 bytes) 000313 1782928143.816251 trunc from 0x0254ef to 0x039040 (0x13b52 bytes) 000314 1782928143.820126 trunc from 0x039041 to 0x029de9 (0xf257 bytes) 000316 1782928143.824669 mapread 0x00964f thru 0x00dbe7 (0x4599 bytes) 000318 1782928143.825791 read 0x01a283 thru 0x0213e3 (0x7161 bytes) 000319 1782928143.826349 write 0x00ff67 thru 0x01fc85 (0xfd1f bytes) 000320 1782928143.827332 read 0x00372e thru 0x00f3fc (0xbccf bytes) 000323 1782928143.830791 read 0x007538 thru 0x0163d2 (0xee9b bytes) 000325 1782928143.831735 read 0x00a3dc thru 0x00e4b2 (0x40d7 bytes) 000326 1782928143.832185 write 0x02767a thru 0x028833 (0x11ba bytes) 000327 1782928143.832864 mapread 0x0006ac thru 0x009987 (0x92dc bytes) 000332 1782928143.836637 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29dea): Mirror 6: 0x87921679 (stale) Mirror 7: 0x87921679 (stale) Mirror 8: 0x87921679 (stale) Mirror 10: 0x87921679 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29dea): Mirror 6: 0x87921679 (stale) Mirror 7: 0x87921679 (stale) Mirror 8: 0x87921679 (stale) Mirror 10: 0x87921679 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29dea: skipped 000334 1782928143.933800 mapwrite 0x02b703 thru 0x035fc6 (0xa8c4 bytes) 000336 1782928143.957281 mapread 0x027251 thru 0x02727e (0x2e bytes) 000337 1782928143.959180 MIRROR_RESYNC 5 mirrors 000338 1782928144.185467 write 0x03ef22 thru 0x03ffff (0x10de bytes) 000339 1782928144.202635 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xed9210d3 (stale) Mirror 8: 0xed9210d3 (stale) Mirror 10: 0xed9210d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xed9210d3 (stale) Mirror 8: 0xed9210d3 (stale) Mirror 10: 0xed9210d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000344 1782928144.297702 mapwrite 0x010b30 thru 0x015311 (0x47e2 bytes) 000345 1782928144.316415 read 0x03ffb9 thru 0x03ffff (0x47 bytes) 000346 1782928144.318363 write 0x027a88 thru 0x035a72 (0xdfeb bytes) 000349 1782928144.322351 read 0x02eb2b thru 0x02ff52 (0x1428 bytes) 000353 1782928144.325356 MIRROR_RESYNC 4 mirrors 000355 1782928144.525304 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000356 1782928144.617765 mapread 0x02cca5 thru 0x033a5c (0x6db8 bytes) 000358 1782928144.623879 read 0x02d9af thru 0x037da2 (0xa3f4 bytes) 000361 1782928144.626380 read 0x00acaf thru 0x00c3eb (0x173d bytes) 000362 1782928144.629657 mapread 0x00b9d2 thru 0x00f6f6 (0x3d25 bytes) 000363 1782928144.630400 trunc from 0x040000 to 0x03c6fb (0x3904 bytes) 000365 1782928144.660856 trunc from 0x03c6fc to 0x030826 (0xbed5 bytes) 000366 1782928144.666828 mapwrite 0x02cb9b thru 0x037ef7 (0xb35d bytes) 000367 1782928144.700715 mapread 0x02b3ad thru 0x02c5f6 (0x124a bytes) 000368 1782928144.702925 trunc from 0x037ef8 to 0x00a5b4 (0x2d943 bytes) 000369 1782928144.707384 mapread 0x002456 thru 0x00a5b4 (0x815f bytes) 000370 1782928144.719064 mapwrite 0x0098e9 thru 0x00ac2b (0x1343 bytes) 000372 1782928144.736623 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xac2c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xac2c): Mirror 8: 0x83ab02b6 (stale) Mirror 10: 0x83ab02b6 (stale) CRC-32 checksum value for chunk [0, 0xac2c): Mirror 1: 0x6a4e8912 Mirror 11: 0x6a4e8912 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xac2c: skipped 000373 1782928144.947197 mapread 0x00347e thru 0x0040d0 (0xc53 bytes) 000374 1782928144.952398 trunc from 0x00ac2c to 0x03e180 (0x33555 bytes) 000375 1782928144.979875 read 0x030936 thru 0x033d84 (0x344f bytes) 000376 1782928144.983852 MIRROR_RESYNC 4 mirrors 000377 1782928145.187370 write 0x00d82a thru 0x015276 (0x7a4d bytes) 000378 1782928145.203970 trunc from 0x03e181 to 0x0188e4 (0x2589c bytes) 000379 1782928145.209009 read 0x01367b thru 0x01431e (0xca4 bytes) 000382 1782928145.210740 trunc from 0x0188e5 to 0x010fc9 (0x791b bytes) 000386 1782928145.216091 mapwrite 0x01d09b thru 0x02bdaa (0xed10 bytes) 000388 1782928145.243809 write 0x029359 thru 0x02be4a (0x2af2 bytes) 000389 1782928145.244843 read 0x008089 thru 0x00e429 (0x63a1 bytes) 000391 1782928145.248359 mapwrite 0x010a61 thru 0x01f50f (0xeaaf bytes) 000394 1782928145.263363 trunc from 0x02be4b to 0x01a660 (0x117ea bytes) 000395 1782928145.268795 mapwrite 0x003692 thru 0x00509a (0x1a09 bytes) 000396 1782928145.281975 trunc from 0x01a661 to 0x000e0e (0x19852 bytes) 000397 1782928145.288663 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe0f): Mirror 8: 0x81dd832d (stale) Mirror 10: 0x81dd832d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe0f): Mirror 8: 0x81dd832d (stale) Mirror 10: 0x81dd832d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe0f: skipped 000398 1782928145.364683 mapwrite 0x02a2b1 thru 0x033ac2 (0x9812 bytes) 000400 1782928145.388146 mapread 0x02261b thru 0x0239c9 (0x13af bytes) 000401 1782928145.391126 mapwrite 0x02a20c thru 0x038392 (0xe187 bytes) 000403 1782928145.412443 mapread 0x0315f6 thru 0x038392 (0x6d9d bytes) 000404 1782928145.413603 write 0x01e08c thru 0x02dc21 (0xfb96 bytes) 000407 1782928145.415977 trunc from 0x038393 to 0x007788 (0x30c0a bytes) 000409 1782928145.422334 MIRROR_RESYNC 3 mirrors 000412 1782928145.600175 MIRROR_EXTEND to 4 mirrors 000414 1782928145.749410 read 0x00665f thru 0x007788 (0x112a bytes) 000415 1782928145.751009 write 0x0006fe thru 0x001801 (0x1104 bytes) 000417 1782928145.771920 mapread 0x001a83 thru 0x005931 (0x3eaf bytes) 000419 1782928145.778567 MIRROR_RESYNC 4 mirrors 000420 1782928145.971144 trunc from 0x007789 to 0x036567 (0x2eddf bytes) 000421 1782928145.996165 MIRROR_RESYNC 4 mirrors 000422 1782928146.204447 trunc from 0x036568 to 0x0050fd (0x3146a bytes) 000424 1782928146.221417 read 0x0000d6 thru 0x0050fd (0x5028 bytes) 000426 1782928146.225947 MIRROR_RESYNC 4 mirrors 000427 1782928146.422025 write 0x00bc71 thru 0x017e4e (0xc1de bytes) 000428 1782928146.440946 mapwrite 0x01fc75 thru 0x0265fd (0x6989 bytes) 000429 1782928146.471123 read 0x01c9a8 thru 0x0265fd (0x9c56 bytes) 000433 1782928146.473591 read 0x018bae thru 0x0265fd (0xda50 bytes) 000435 1782928146.477236 mapwrite 0x021aa8 thru 0x022bed (0x1146 bytes) 000436 1782928146.488978 write 0x00fafa thru 0x01460f (0x4b16 bytes) 000438 1782928146.490431 mapread 0x004ef6 thru 0x0068ce (0x19d9 bytes) 000441 1782928146.492070 mapwrite 0x01e4a2 thru 0x02be37 (0xd996 bytes) 000442 1782928146.514493 write 0x005e7c thru 0x013863 (0xd9e8 bytes) 000443 1782928146.515683 read 0x0136b0 thru 0x022b9f (0xf4f0 bytes) 000449 1782928146.518096 mapwrite 0x0387aa thru 0x03ffff (0x7856 bytes) 000453 1782928146.539551 mapwrite 0x00c6f7 thru 0x01a41d (0xdd27 bytes) 000457 1782928146.555063 read 0x00c602 thru 0x016c36 (0xa635 bytes) 000458 1782928146.555545 read 0x0331f1 thru 0x03ffff (0xce0f bytes) 000459 1782928146.557548 MIRROR_RESYNC 4 mirrors 000465 1782928146.759317 read 0x0354ef thru 0x03ffff (0xab11 bytes) 000467 1782928146.762137 trunc from 0x040000 to 0x01472c (0x2b8d3 bytes) 000471 1782928146.789348 mapread 0x013c3b thru 0x01472c (0xaf2 bytes) 000472 1782928146.792823 trunc from 0x01472d to 0x0021e0 (0x1254c bytes) 000473 1782928146.798775 trunc from 0x0021e1 to 0x000907 (0x18d9 bytes) 000474 1782928146.803352 read 0x0008fe thru 0x000907 (0xa bytes) 000475 1782928146.805140 mapread 0x0001a6 thru 0x000907 (0x762 bytes) 000476 1782928146.805852 read 0x0000e7 thru 0x000907 (0x821 bytes) 000479 1782928146.809322 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x908): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x908): Mirror 8: 0x25842072 (stale) Mirror 10: 0x25842072 (stale) Mirror 11: 0x25842072 (stale) CRC-32 checksum value for chunk [0, 0x908): Mirror 1: 0x25842072 Mirror 12: 0x25842072 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x908: skipped 000480 1782928147.011096 read 0x00056e thru 0x000907 (0x39a bytes) 000481 1782928147.013548 trunc from 0x000908 to 0x01b67e (0x1ad77 bytes) 000484 1782928147.039581 read 0x00783e thru 0x0144dd (0xcca0 bytes) 000486 1782928147.042976 trunc from 0x01b67f to 0x00416a (0x17514 bytes) 000489 1782928147.050630 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x416b): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x416b): Mirror 8: 0x91633027 (stale) Mirror 10: 0x91633027 (stale) Mirror 11: 0x91633027 (stale) Mirror 12: 0xba6107a0 (stale) CRC-32 checksum value for chunk [0, 0x416b): Mirror 1: 0xba6107a0 Mirror 13: 0xba6107a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x416b: skipped 000490 1782928147.259899 mapread 0x000a64 thru 0x00416a (0x3707 bytes) 000491 1782928147.268129 mapread 0x002785 thru 0x00416a (0x19e6 bytes) 000493 1782928147.271781 mapwrite 0x03f914 thru 0x03ffff (0x6ec bytes) 000494 1782928147.312619 mapread 0x0392e8 thru 0x03e28a (0x4fa3 bytes) 000495 1782928147.313407 mapwrite 0x012e3b thru 0x01c5a7 (0x976d bytes) 000497 1782928147.334875 write 0x023b62 thru 0x02844f (0x48ee bytes) 000498 1782928147.337425 mapread 0x014da6 thru 0x014f1f (0x17a bytes) 000501 1782928147.340831 MIRROR_RESYNC 6 mirrors 000502 1782928147.608721 mapwrite 0x01c346 thru 0x01d899 (0x1554 bytes) 000503 1782928147.658175 mapread 0x00bd52 thru 0x016143 (0xa3f2 bytes) 000504 1782928147.662901 trunc from 0x040000 to 0x038ae9 (0x7516 bytes) 000506 1782928147.675603 MIRROR_RESYNC 6 mirrors 000507 1782928147.922741 mapread 0x0148e9 thru 0x0240a9 (0xf7c1 bytes) 000510 1782928147.930743 write 0x02476e thru 0x033d62 (0xf5f5 bytes) 000511 1782928147.954420 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38aea): Mirror 8: 0x7900a6f1 (stale) Mirror 10: 0x7900a6f1 (stale) Mirror 11: 0x7900a6f1 (stale) Mirror 12: 0x7900a6f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38aea): Mirror 8: 0x7900a6f1 (stale) Mirror 10: 0x7900a6f1 (stale) Mirror 11: 0x7900a6f1 (stale) Mirror 12: 0x7900a6f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38aea: skipped 000515 1782928148.065596 mapread 0x00f846 thru 0x01e343 (0xeafe bytes) 000520 1782928148.076468 trunc from 0x038aea to 0x0222ca (0x1681f bytes) 000521 1782928148.086633 write 0x018ab6 thru 0x019bc7 (0x1112 bytes) 000526 1782928148.090612 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x222cb): Mirror 10: 0x55e82e8f (stale) Mirror 11: 0x55e82e8f (stale) Mirror 12: 0x55e82e8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222cb): Mirror 10: 0x55e82e8f (stale) Mirror 11: 0x55e82e8f (stale) Mirror 12: 0x55e82e8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222cb: skipped 000530 1782928148.183789 trunc from 0x0222cb to 0x03d2b9 (0x1afef bytes) 000531 1782928148.189882 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: 0xf611239d (stale) Mirror 11: 0xf611239d (stale) Mirror 12: 0xf611239d (stale) CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 1: 0x7c0e2c08 Mirror 13: 0x7c0e2c08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2ba: skipped 000532 1782928148.442366 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: 0xf611239d (stale) Mirror 11: 0xf611239d (stale) Mirror 12: 0xf611239d (stale) CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 1: 0x7c0e2c08 Mirror 13: 0x7c0e2c08 Mirror 14: 0x7c0e2c08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2ba: skipped 000533 1782928148.688550 mapread 0x02900d thru 0x029735 (0x729 bytes) 000535 1782928148.691949 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: 0xf611239d (stale) Mirror 12: 0xf611239d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 10: 0xf611239d (stale) Mirror 12: 0xf611239d (stale) CRC-32 checksum value for chunk [0, 0x3d2ba): Mirror 1: 0x7c0e2c08 Mirror 13: 0x7c0e2c08 Mirror 14: 0x7c0e2c08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2ba: skipped 000538 1782928148.859318 mapwrite 0x01a9cf thru 0x01c07a (0x16ac bytes) 000539 1782928148.911175 trunc from 0x03d2ba to 0x01f832 (0x1da87 bytes) 000540 1782928148.917212 trunc from 0x01f833 to 0x00c545 (0x132ed bytes) 000541 1782928148.921633 write 0x013494 thru 0x022143 (0xecb0 bytes) 000542 1782928148.922894 mapread 0x00f2db thru 0x013fbd (0x4ce3 bytes) 000546 1782928148.929929 MIRROR_RESYNC 5 mirrors 000548 1782928149.177161 MIRROR_RESYNC 5 mirrors 000549 1782928149.303782 read 0x014841 thru 0x016b87 (0x2347 bytes) 000552 1782928149.306289 trunc from 0x022144 to 0x03f458 (0x1d315 bytes) 000553 1782928149.325000 read 0x0073ab thru 0x01584e (0xe4a4 bytes) 000554 1782928149.328681 write 0x02da90 thru 0x033e92 (0x6403 bytes) 000556 1782928149.331335 MIRROR_RESYNC 5 mirrors 000557 1782928149.567589 mapread 0x03bd00 thru 0x03f458 (0x3759 bytes) 000558 1782928149.572850 write 0x03a084 thru 0x03ffff (0x5f7c bytes) 000559 1782928149.591903 mapwrite 0x034c63 thru 0x03ffff (0xb39d bytes) 000562 1782928149.611350 write 0x02640e thru 0x026c81 (0x874 bytes) 000563 1782928149.615067 MIRROR_RESYNC 5 mirrors 000564 1782928149.841867 MIRROR_RESYNC 5 mirrors 000565 1782928149.970228 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000566 1782928150.070409 write 0x025a9c thru 0x034ab4 (0xf019 bytes) 000567 1782928150.092520 write 0x0357b4 thru 0x03aaa3 (0x52f0 bytes) 000571 1782928150.097009 read 0x034f57 thru 0x03eb37 (0x9be1 bytes) 000573 1782928150.099810 trunc from 0x040000 to 0x03e79b (0x1864 bytes) 000574 1782928150.110481 read 0x0315aa thru 0x037016 (0x5a6d bytes) 000576 1782928150.111529 trunc from 0x03e79c to 0x00515b (0x39640 bytes) 000577 1782928150.116255 write 0x011a31 thru 0x01f2d5 (0xd8a5 bytes) 000579 1782928150.119284 MIRROR_RESYNC 4 mirrors 000580 1782928150.325659 trunc from 0x01f2d6 to 0x00d4ae (0x11e27 bytes) 000582 1782928150.353221 mapwrite 0x0205e6 thru 0x023f69 (0x3984 bytes) 000586 1782928150.379808 trunc from 0x023f6a to 0x002195 (0x21dd4 bytes) 000587 1782928150.384930 mapread 0x00150b thru 0x002195 (0xc8b bytes) 000590 1782928150.390017 MIRROR_RESYNC 4 mirrors 000591 1782928150.585643 read 0x002019 thru 0x002195 (0x17d bytes) 000592 1782928150.587758 read 0x001891 thru 0x002195 (0x905 bytes) 000594 1782928150.591330 MIRROR_RESYNC 4 mirrors 000597 1782928150.701294 mapread 0x00145b thru 0x002195 (0xd3b bytes) 000600 1782928150.702151 read 0x000729 thru 0x002195 (0x1a6d bytes) 000601 1782928150.702392 mapwrite 0x0301e8 thru 0x033ca7 (0x3ac0 bytes) 000602 1782928150.743903 mapwrite 0x02cb9e thru 0x03bd25 (0xf188 bytes) 000603 1782928150.767886 write 0x00e58c thru 0x016f74 (0x89e9 bytes) 000604 1782928150.771599 trunc from 0x03bd26 to 0x030bb0 (0xb175 bytes) 000606 1782928150.776723 trunc from 0x030bb1 to 0x02acc6 (0x5eea bytes) 000608 1782928150.782277 trunc from 0x02acc7 to 0x026065 (0x4c61 bytes) 000609 1782928150.786747 write 0x008dc6 thru 0x017b45 (0xed80 bytes) 000610 1782928150.790647 mapread 0x025ed0 thru 0x026065 (0x196 bytes) 000611 1782928150.793961 read 0x011504 thru 0x0209ab (0xf4a8 bytes) 000613 1782928150.797447 write 0x00e52a thru 0x01c0bb (0xdb92 bytes) 000614 1782928150.798502 write 0x02625a thru 0x0276bd (0x1464 bytes) 000615 1782928150.799246 mapread 0x01a7d2 thru 0x01dcbf (0x34ee bytes) 000616 1782928150.799990 trunc from 0x0276be to 0x02526e (0x244f bytes) 000617 1782928150.804793 write 0x0020ef thru 0x01095a (0xe86c bytes) 000618 1782928150.805935 write 0x02edac thru 0x031edb (0x3130 bytes) 000620 1782928150.807080 mapwrite 0x0333cc thru 0x03bf0f (0x8b44 bytes) 000621 1782928150.830087 trunc from 0x03bf10 to 0x00f9f7 (0x2c518 bytes) 000623 1782928150.835398 write 0x014c29 thru 0x022de0 (0xe1b8 bytes) 000624 1782928150.837844 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22de1): Mirror 12: 0x8674abab (stale) Mirror 14: 0x8674abab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22de1): Mirror 12: 0x8674abab (stale) Mirror 14: 0x8674abab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22de1: skipped 000625 1782928150.922230 trunc from 0x022de1 to 0x014c02 (0xe1de bytes) 000627 1782928150.926958 mapwrite 0x01e8cd thru 0x02187b (0x2faf bytes) 000628 1782928150.950115 MIRROR_RESYNC 3 mirrors 000630 1782928151.144074 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000631 1782928151.235030 mapread 0x005f37 thru 0x01219d (0xc267 bytes) 000632 1782928151.243947 mapwrite 0x033436 thru 0x0352c8 (0x1e93 bytes) 000633 1782928151.284388 mapread 0x03368e thru 0x0352c8 (0x1c3b bytes) 000634 1782928151.285567 mapwrite 0x02ef51 thru 0x031e64 (0x2f14 bytes) 000635 1782928151.299173 mapread 0x004ebe thru 0x00a972 (0x5ab5 bytes) 000636 1782928151.303040 write 0x01bf7d thru 0x01ffd5 (0x4059 bytes) 000637 1782928151.306107 trunc from 0x0352c9 to 0x02e933 (0x6995 bytes) 000638 1782928151.312244 mapread 0x01030b thru 0x01d6c4 (0xd3ba bytes) 000639 1782928151.319296 MIRROR_RESYNC 2 mirrors 000640 1782928151.477072 mapread 0x013705 thru 0x01f21a (0xbb16 bytes) 000642 1782928151.484831 read 0x0222cb thru 0x02a841 (0x8577 bytes) 000643 1782928151.486692 mapwrite 0x00113f thru 0x002e7c (0x1d3e bytes) 000644 1782928151.533371 read 0x01e99f thru 0x02e745 (0xfda7 bytes) 000646 1782928151.536873 trunc from 0x02e934 to 0x007d77 (0x26bbc bytes) 000649 1782928151.549220 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000650 1782928151.589505 write 0x004912 thru 0x0135e9 (0xecd8 bytes) 000651 1782928151.592286 mapwrite 0x022848 thru 0x027ad8 (0x5291 bytes) 000652 1782928151.616378 mapread 0x01dd15 thru 0x022784 (0x4a70 bytes) 000653 1782928151.617215 trunc from 0x027ad9 to 0x02de8a (0x63b2 bytes) 000654 1782928151.621735 mapwrite 0x03c4f6 thru 0x03ffff (0x3b0a bytes) 000656 1782928151.641234 mapread 0x0150cc thru 0x023af1 (0xea26 bytes) 000657 1782928151.644541 trunc from 0x040000 to 0x015ce5 (0x2a31a bytes) 000658 1782928151.649314 mapwrite 0x0019cf thru 0x00561c (0x3c4e bytes) 000661 1782928151.666098 read 0x00f25f thru 0x01405f (0x4e01 bytes) 000662 1782928151.666590 write 0x02ce20 thru 0x02ce5a (0x3b bytes) 000663 1782928151.667352 write 0x028880 thru 0x02ce38 (0x45b9 bytes) 000664 1782928151.669741 mapread 0x02902f thru 0x02ce5a (0x3e2c bytes) 000666 1782928151.670933 read 0x0011be thru 0x009a3a (0x887d bytes) 000668 1782928151.672250 write 0x03a140 thru 0x03ffff (0x5ec0 bytes) 000669 1782928151.673483 write 0x030edc thru 0x03fa6b (0xeb90 bytes) 000671 1782928151.676984 write 0x025c77 thru 0x02abc4 (0x4f4e bytes) 000672 1782928151.677788 mapwrite 0x036dc5 thru 0x03ffff (0x923b bytes) 000673 1782928151.691582 mapwrite 0x027f8a thru 0x03361a (0xb691 bytes) 000674 1782928151.705350 write 0x00f224 thru 0x0174c6 (0x82a3 bytes) 000675 1782928151.706655 trunc from 0x040000 to 0x028a41 (0x175be bytes) 000676 1782928151.711885 mapread 0x002d5c thru 0x003c10 (0xeb5 bytes) 000677 1782928151.712630 write 0x0235c2 thru 0x027795 (0x41d4 bytes) 000678 1782928151.713634 trunc from 0x028a42 to 0x0371b4 (0xe773 bytes) 000679 1782928151.718453 mapread 0x019385 thru 0x01a977 (0x15f3 bytes) 000680 1782928151.719434 trunc from 0x0371b5 to 0x014353 (0x22e61 bytes) 000681 1782928151.724126 mapread 0x005c5b thru 0x00b9dc (0x5d82 bytes) 000682 1782928151.724800 trunc from 0x014354 to 0x028e8f (0x14b3c bytes) 000683 1782928151.728835 mapread 0x01499c thru 0x01c6c4 (0x7d29 bytes) 000684 1782928151.740032 write 0x023b16 thru 0x02dc6e (0xa159 bytes) 000685 1782928151.742342 write 0x027033 thru 0x030826 (0x97f4 bytes) 000688 1782928151.744713 write 0x019d75 thru 0x01c7ef (0x2a7b bytes) 000689 1782928151.745764 mapwrite 0x02b542 thru 0x038cde (0xd79d bytes) 000690 1782928151.778245 mapread 0x0052ea thru 0x006dc9 (0x1ae0 bytes) 000691 1782928151.778964 read 0x024313 thru 0x026a6e (0x275c bytes) 000696 1782928151.781014 read 0x01810c thru 0x01acd3 (0x2bc8 bytes) 000699 1782928151.782300 mapwrite 0x028317 thru 0x02be78 (0x3b62 bytes) 000703 1782928151.795260 mapread 0x015bc4 thru 0x01d529 (0x7966 bytes) 000704 1782928151.798286 trunc from 0x038cdf to 0x03be4c (0x316e bytes) 000706 1782928151.803467 read 0x007d81 thru 0x012afc (0xad7c bytes) 000708 1782928151.804444 trunc from 0x03be4d to 0x0193e7 (0x22a65 bytes) 000710 1782928151.809034 write 0x0074f2 thru 0x01241c (0xaf2b bytes) 000712 1782928151.810406 mapread 0x018954 thru 0x0193e7 (0xa94 bytes) 000713 1782928151.811058 trunc from 0x0193e8 to 0x0220a7 (0x8cc0 bytes) 000714 1782928151.815364 write 0x0111a8 thru 0x01d68d (0xc4e6 bytes) 000718 1782928151.819465 trunc from 0x0220a8 to 0x027d77 (0x5cd0 bytes) 000723 1782928151.826379 mapread 0x01918b thru 0x01a3e2 (0x1258 bytes) 000725 1782928151.827560 trunc from 0x027d78 to 0x0203d2 (0x79a5 bytes) 000728 1782928151.833016 trunc from 0x0203d3 to 0x008269 (0x18169 bytes) 000730 1782928151.838189 mapread 0x006969 thru 0x008269 (0x1901 bytes) 000731 1782928151.838874 mapwrite 0x02ea21 thru 0x0358b0 (0x6e90 bytes) 000734 1782928151.860717 trunc from 0x0358b1 to 0x01a684 (0x1b22c bytes) 000735 1782928151.865513 mapread 0x019e4f thru 0x01a684 (0x836 bytes) 000736 1782928151.866300 read 0x015538 thru 0x01779b (0x2264 bytes) 000737 1782928151.866813 mapread 0x00fb20 thru 0x01a684 (0xab65 bytes) 000738 1782928151.869981 trunc from 0x01a685 to 0x03221a (0x17b96 bytes) 000740 1782928151.874675 write 0x011c3a thru 0x0143bf (0x2786 bytes) 000742 1782928151.875744 mapwrite 0x03b975 thru 0x03c4d9 (0xb65 bytes) 000744 1782928151.897323 mapread 0x02722e thru 0x030a24 (0x97f7 bytes) 000746 1782928151.898578 mapread 0x03235a thru 0x037e22 (0x5ac9 bytes) 000747 1782928151.899331 read 0x03019e thru 0x0349e5 (0x4848 bytes) 000748 1782928151.899792 trunc from 0x03c4da to 0x004b27 (0x379b2 bytes) 000751 1782928151.905714 read 0x0011f6 thru 0x0028b3 (0x16be bytes) 000752 1782928151.906283 trunc from 0x004b28 to 0x031e8d (0x2d366 bytes) 000753 1782928151.911227 trunc from 0x031e8e to 0x03aad3 (0x8c46 bytes) 000756 1782928151.917123 mapread 0x00f273 thru 0x01e442 (0xf1d0 bytes) 000757 1782928151.921738 write 0x0227c5 thru 0x02d95a (0xb196 bytes) 000759 1782928151.922930 mapwrite 0x01ae80 thru 0x01f4b6 (0x4637 bytes) 000760 1782928151.936116 trunc from 0x03aad4 to 0x03efab (0x44d8 bytes) 000761 1782928151.942364 MIRROR_EXTEND to 2 mirrors 000764 1782928152.117359 read 0x00437c thru 0x00bec1 (0x7b46 bytes) 000765 1782928152.121621 read 0x00a8da thru 0x0113be (0x6ae5 bytes) 000766 1782928152.126290 trunc from 0x03efac to 0x03d534 (0x1a77 bytes) 000768 1782928152.146828 write 0x013209 thru 0x01bcfc (0x8af4 bytes) 000769 1782928152.150227 mapwrite 0x01af9b thru 0x021909 (0x696f bytes) 000771 1782928152.172776 mapread 0x01617b thru 0x01c5f2 (0x6478 bytes) 000772 1782928152.173526 write 0x03a713 thru 0x03ffff (0x58ed bytes) 000773 1782928152.175888 mapwrite 0x035dbb thru 0x03ffff (0xa245 bytes) 000774 1782928152.197027 trunc from 0x040000 to 0x00a1ad (0x35e52 bytes) 000775 1782928152.203249 mapread 0x000d3b thru 0x0074e3 (0x67a9 bytes) 000777 1782928152.213982 trunc from 0x00a1ae to 0x03e320 (0x34173 bytes) 000779 1782928152.220555 MIRROR_RESYNC 2 mirrors 000784 1782928152.401434 write 0x011fe3 thru 0x01a4c4 (0x84e2 bytes) 000785 1782928152.419326 write 0x02ad34 thru 0x035466 (0xa733 bytes) 000787 1782928152.425067 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000789 1782928152.464627 read 0x02f234 thru 0x03e320 (0xf0ed bytes) 000790 1782928152.467493 write 0x019fec thru 0x02158d (0x75a2 bytes) 000791 1782928152.470640 read 0x02515b thru 0x02c4df (0x7385 bytes) 000792 1782928152.473955 mapwrite 0x02ecb8 thru 0x03b9f5 (0xcd3e bytes) 000793 1782928152.488576 read 0x000825 thru 0x0059b4 (0x5190 bytes) 000795 1782928152.489587 write 0x0031f0 thru 0x007587 (0x4398 bytes) 000796 1782928152.490440 trunc from 0x03e321 to 0x00d333 (0x30fed bytes) 000797 1782928152.495710 write 0x009a11 thru 0x00dad1 (0x40c1 bytes) 000798 1782928152.496493 trunc from 0x00dad2 to 0x00f65d (0x1b8c bytes) 000800 1782928152.500978 trunc from 0x00f65e to 0x015c75 (0x6618 bytes) 000802 1782928152.505661 mapread 0x0069d9 thru 0x00d9d6 (0x6ffe bytes) 000804 1782928152.506963 mapwrite 0x0229ed thru 0x031a82 (0xf096 bytes) 000805 1782928152.527871 mapread 0x00f8e8 thru 0x01f18a (0xf8a3 bytes) 000806 1782928152.528780 mapwrite 0x0248c9 thru 0x02826e (0x39a6 bytes) 000810 1782928152.541396 read 0x0027b0 thru 0x00ec92 (0xc4e3 bytes) 000813 1782928152.542786 read 0x006457 thru 0x00cd1a (0x68c4 bytes) 000815 1782928152.543884 mapread 0x00e1ca thru 0x0113dc (0x3213 bytes) 000821 1782928152.547865 MIRROR_EXTEND to 2 mirrors 000822 1782928152.700667 read 0x019e59 thru 0x01b697 (0x183f bytes) 000823 1782928152.703621 mapwrite 0x0127f9 thru 0x01ce5a (0xa662 bytes) 000824 1782928152.758958 read 0x00f1bc thru 0x0139f2 (0x4837 bytes) 000825 1782928152.759499 write 0x03481d thru 0x039b08 (0x52ec bytes) 000826 1782928152.760380 write 0x02fa5e thru 0x031bde (0x2181 bytes) 000827 1782928152.763478 read 0x007307 thru 0x01659a (0xf294 bytes) 000828 1782928152.763949 mapread 0x031cdf thru 0x039b08 (0x7e2a bytes) 000829 1782928152.764793 trunc from 0x039b09 to 0x020267 (0x198a1 bytes) 000830 1782928152.774888 trunc from 0x020268 to 0x02b08f (0xae28 bytes) 000832 1782928152.780979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: 0xba46b969 (stale) CRC-32 checksum value for chunk [0, 0x2b090): Mirror 1: 0x30bd3670 Mirror 3: 0x30bd3670 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b090: skipped 000833 1782928152.990931 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: 0xba46b969 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: 0xba46b969 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b090: skipped 000834 1782928153.066587 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b090): Mirror 2: 0xba46b969 (stale) CRC-32 checksum value for chunk [0, 0x2b090): Mirror 1: 0x30bd3670 Mirror 3: 0x30bd3670 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b090: skipped 000836 1782928153.293157 read 0x00b9b0 thru 0x017195 (0xb7e6 bytes) 000837 1782928153.296438 mapwrite 0x030bab thru 0x03a945 (0x9d9b bytes) 000838 1782928153.338577 MIRROR_RESYNC 3 mirrors 000839 1782928153.528248 trunc from 0x03a946 to 0x0301ce (0xa777 bytes) 000844 1782928153.553783 MIRROR_RESYNC 3 mirrors 000845 1782928153.730743 MIRROR_RESYNC 3 mirrors 000846 1782928153.849508 MIRROR_EXTEND to 4 mirrors 000848 1782928154.019924 mapread 0x0199b1 thru 0x0296ba (0xfd0a bytes) 000849 1782928154.029524 mapread 0x01d15a thru 0x02ccb0 (0xfb57 bytes) 000850 1782928154.030272 mapwrite 0x01add7 thru 0x01da98 (0x2cc2 bytes) 000852 1782928154.081204 mapwrite 0x0083ca thru 0x00b368 (0x2f9f bytes) 000854 1782928154.101341 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301cf): Mirror 3: 0x72033d9c (stale) Mirror 4: 0x72033d9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301cf): Mirror 3: 0x72033d9c (stale) Mirror 4: 0x72033d9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301cf: skipped 000856 1782928154.198930 write 0x03f130 thru 0x03f1de (0xaf bytes) 000857 1782928154.199872 read 0x02e12b thru 0x030b67 (0x2a3d bytes) 000858 1782928154.202493 read 0x0219d1 thru 0x02b390 (0x99c0 bytes) 000861 1782928154.207005 write 0x035169 thru 0x03ffff (0xae97 bytes) 000862 1782928154.208128 read 0x01a378 thru 0x028c77 (0xe900 bytes) 000865 1782928154.209481 mapread 0x0307f9 thru 0x032657 (0x1e5f bytes) 000866 1782928154.210331 write 0x03f28f thru 0x03ffff (0xd71 bytes) 000867 1782928154.213137 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd0ad98dc (stale) Mirror 4: 0xd0ad98dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5ae486b Mirror 5: 0xb5ae486b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000868 1782928154.490345 read 0x005810 thru 0x00ddf9 (0x85ea bytes) 000871 1782928154.499234 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd0ad98dc (stale) Mirror 4: 0xd0ad98dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5ae486b Mirror 5: 0xb5ae486b Mirror 6: 0xb5ae486b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000873 1782928154.748694 trunc from 0x040000 to 0x03c79c (0x3863 bytes) 000874 1782928154.777985 read 0x00f3a8 thru 0x0134cd (0x4126 bytes) 000875 1782928154.781145 mapwrite 0x02293b thru 0x0308ca (0xdf90 bytes) 000876 1782928154.806904 read 0x0253fe thru 0x026e19 (0x1a1c bytes) 000877 1782928154.809151 MIRROR_RESYNC 5 mirrors 000879 1782928155.086697 mapread 0x00ae53 thru 0x00d839 (0x29e7 bytes) 000880 1782928155.092658 mapwrite 0x0254c8 thru 0x034de8 (0xf921 bytes) 000883 1782928155.152987 trunc from 0x03c79d to 0x0246fa (0x180a2 bytes) 000885 1782928155.165369 read 0x012a50 thru 0x01d0ca (0xa67b bytes) 000886 1782928155.168752 read 0x00ae10 thru 0x0183fe (0xd5ef bytes) 000888 1782928155.171991 trunc from 0x0246fb to 0x01c5ff (0x80fb bytes) 000889 1782928155.176605 trunc from 0x01c600 to 0x01f65b (0x305c bytes) 000890 1782928155.181012 mapread 0x015044 thru 0x01b4c5 (0x6482 bytes) 000893 1782928155.184222 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f65c): Mirror 3: 0x2eb48af (stale) Mirror 4: 0x2eb48af (stale) Mirror 5: 0x2eb48af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f65c): Mirror 3: 0x2eb48af (stale) Mirror 4: 0x2eb48af (stale) Mirror 5: 0x2eb48af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f65c: skipped 000895 1782928155.286199 write 0x02fb15 thru 0x03e8d8 (0xedc4 bytes) 000896 1782928155.287894 mapread 0x033643 thru 0x035a0d (0x23cb bytes) 000897 1782928155.288624 write 0x0008ea thru 0x0017f5 (0xf0c bytes) 000900 1782928155.293134 trunc from 0x03e8d9 to 0x0120e0 (0x2c7f8 bytes) 000901 1782928155.298139 mapread 0x009506 thru 0x0120e0 (0x8bdb bytes) 000902 1782928155.303781 mapwrite 0x033dc4 thru 0x03adff (0x703c bytes) 000905 1782928155.325671 write 0x036591 thru 0x03bf1e (0x598e bytes) 000907 1782928155.326893 read 0x01ac8b thru 0x025836 (0xabac bytes) 000910 1782928155.330763 write 0x03625e thru 0x03b21a (0x4fbd bytes) 000911 1782928155.331524 mapread 0x009dac thru 0x011292 (0x74e7 bytes) 000912 1782928155.332303 trunc from 0x03bf1f to 0x01b37a (0x20ba4 bytes) 000914 1782928155.337813 read 0x012cba thru 0x017612 (0x4959 bytes) 000915 1782928155.341364 mapwrite 0x02789d thru 0x029c5d (0x23c1 bytes) 000916 1782928155.361320 write 0x018682 thru 0x01a069 (0x19e8 bytes) 000917 1782928155.362122 read 0x00e00f thru 0x00e247 (0x239 bytes) 000918 1782928155.364092 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29c5e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c5e): Mirror 3: 0x82659f03 (stale) Mirror 4: 0x82659f03 (stale) Mirror 5: 0x82659f03 (stale) CRC-32 checksum value for chunk [0, 0x29c5e): Mirror 1: 0x3fae4e93 Mirror 6: 0x3fae4e93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c5e: skipped 000919 1782928155.599938 write 0x011a38 thru 0x017f6e (0x6537 bytes) 000921 1782928155.623553 trunc from 0x029c5e to 0x012de0 (0x16e7d bytes) 000922 1782928155.637591 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12de1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12de1): Mirror 3: 0x1d6b49fc (stale) Mirror 4: 0x1d6b49fc (stale) Mirror 5: 0x1d6b49fc (stale) Mirror 6: 0x696d6086 (stale) CRC-32 checksum value for chunk [0, 0x12de1): Mirror 1: 0xa91e8914 Mirror 7: 0xa91e8914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12de1: skipped 000923 1782928155.868215 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12de1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12de1): Mirror 3: 0x1d6b49fc (stale) Mirror 4: 0x1d6b49fc (stale) Mirror 5: 0x1d6b49fc (stale) Mirror 6: 0x696d6086 (stale) CRC-32 checksum value for chunk [0, 0x12de1): Mirror 1: 0xa91e8914 Mirror 7: 0xa91e8914 Mirror 8: 0xa91e8914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12de1: skipped 000925 1782928156.097767 read 0x0029fa thru 0x00ee86 (0xc48d bytes) 000927 1782928156.101101 mapread 0x0008d0 thru 0x002b83 (0x22b4 bytes) 000931 1782928156.105612 trunc from 0x012de1 to 0x016bd5 (0x3df5 bytes) 000932 1782928156.139006 MIRROR_RESYNC 7 mirrors 000936 1782928156.428717 mapwrite 0x00ff02 thru 0x01f5c1 (0xf6c0 bytes) 000938 1782928156.477150 read 0x006f3a thru 0x01476d (0xd834 bytes) 000941 1782928156.480489 mapread 0x0118d7 thru 0x01cf4b (0xb675 bytes) 000943 1782928156.481745 read 0x014589 thru 0x01b458 (0x6ed0 bytes) 000944 1782928156.482224 trunc from 0x01f5c2 to 0x0003ed (0x1f1d4 bytes) 000945 1782928156.490471 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee): Mirror 3: 0xae3e055a (stale) Mirror 4: 0xae3e055a (stale) Mirror 5: 0xae3e055a (stale) Mirror 7: 0xae3e055a (stale) Mirror 8: 0xae3e055a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee): Mirror 3: 0xae3e055a (stale) Mirror 4: 0xae3e055a (stale) Mirror 5: 0xae3e055a (stale) Mirror 7: 0xae3e055a (stale) Mirror 8: 0xae3e055a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee: skipped 000946 1782928156.582909 mapwrite 0x016417 thru 0x025f6d (0xfb57 bytes) 000947 1782928156.609690 mapread 0x016717 thru 0x024dbf (0xe6a9 bytes) 000948 1782928156.610892 mapread 0x025171 thru 0x025234 (0xc4 bytes) 000949 1782928156.611689 mapwrite 0x00b8c4 thru 0x013f74 (0x86b1 bytes) 000950 1782928156.631274 MIRROR_RESYNC 6 mirrors 000952 1782928156.894520 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x642:0x0]: Invalid argument (22) 000953 1782928156.998587 mapwrite 0x03cb8d thru 0x03ffff (0x3473 bytes) 000954 1782928157.046472 mapwrite 0x03f8bf thru 0x03ffff (0x741 bytes) 000955 1782928157.057441 trunc from 0x040000 to 0x03bfa5 (0x405a bytes) 000956 1782928157.063694 trunc from 0x03bfa6 to 0x027644 (0x14961 bytes) 000957 1782928157.068189 mapwrite 0x0386d3 thru 0x03ffff (0x792d bytes) 000958 1782928157.088207 mapwrite 0x02f634 thru 0x0362fd (0x6cca bytes) 000961 1782928157.107378 trunc from 0x040000 to 0x01f731 (0x208ce bytes) 000964 1782928157.115117 mapread 0x0109f7 thru 0x015def (0x53f9 bytes) 000967 1782928157.120121 mapwrite 0x00b32a thru 0x01888f (0xd566 bytes) 000968 1782928157.137506 trunc from 0x01f732 to 0x016567 (0x91ca bytes) 000970 1782928157.143263 mapwrite 0x033c71 thru 0x03ec76 (0xb006 bytes) 000971 1782928157.167428 read 0x0182f4 thru 0x0271fb (0xef08 bytes) 000974 1782928157.172795 mapwrite 0x03c9ce thru 0x03ffff (0x3632 bytes) 000976 1782928157.192104 mapwrite 0x02e4c5 thru 0x02f235 (0xd71 bytes) 000978 1782928157.204267 read 0x02f767 thru 0x03c87c (0xd116 bytes) 000979 1782928157.204753 mapread 0x018e22 thru 0x026e7f (0xe05e bytes) 000980 1782928157.205597 read 0x03708c thru 0x03ffff (0x8f74 bytes) 000981 1782928157.206001 trunc from 0x040000 to 0x0266dc (0x19923 bytes) 000982 1782928157.210992 mapwrite 0x025065 thru 0x026a66 (0x1a02 bytes) 000983 1782928157.227754 write 0x021ec8 thru 0x0243a0 (0x24d9 bytes) 000986 1782928157.229449 write 0x02eff5 thru 0x037877 (0x8883 bytes) 000987 1782928157.230783 trunc from 0x037878 to 0x02d56b (0xa30c bytes) 000988 1782928157.235954 mapread 0x00e9a8 thru 0x01bc17 (0xd270 bytes) 000989 1782928157.239567 write 0x03a951 thru 0x03efea (0x469a bytes) 000991 1782928157.240889 mapread 0x0303c4 thru 0x033d44 (0x3981 bytes) 000995 1782928157.245696 write 0x016bea thru 0x023ec3 (0xd2da bytes) 000997 1782928157.247729 mapwrite 0x00da9e thru 0x00f8a8 (0x1e0b bytes) 000999 1782928157.264196 trunc from 0x03efeb to 0x00dc71 (0x31379 bytes) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 13:49:19 (1782928159) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 13:49:20 (1782928160) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0721452 s, 29.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498366 s, 21.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9c8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 13:49:22 (1782928162) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2795840 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4598784 .. Extending file size to 8843648 .. Extending file size to 7454208 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9004960 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8291712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8550080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9065760 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8693120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9024704 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9302432 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60228 60229 60231 60232 60235 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:50:28 (1782928228) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 949792 .. Extending file size to 5432128 .. Extending file size to 7768096 .. Extending file size to 8288384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8794720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9293440 .. Extending file size to 8419168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8422528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8683840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9020768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9205632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 63409 63410 63412 63413 63415 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:51:32 (1782928292) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3584 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3584 2204544 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0378505 s, 55.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0735613 s, 28.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x44cb28c3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:51:38 (1782928298) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0269168 s, 77.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 13:51:40 (1782928300) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x685:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9d9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9d9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9da:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x9dd:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:51:42 (1782928302) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0391755 s, 53.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:51:44 (1782928304) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0401675 s, 52.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e1:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:51:46 (1782928306) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0432778 s, 48.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e3:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:52:10 (1782928330) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233402 s, 44.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e0:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:52:54 (1782928374) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.040876 s, 51.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:53:39 (1782928419) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. Creating new pool test_204f oleg218-server: Pool lustre.test_204f created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0447347 s, 46.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg218-server: Pool lustre.test_204f destroyed PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:54:39 (1782928479) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:54:41 (1782928481) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:54:44 (1782928484) oleg218-server: Pool lustre.test_206 created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg218-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:54:58 (1782928498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283354 s, 37.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117572 s, 89.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfe27ca27 Mirror 3: 0xfe27ca27 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:55:00 (1782928500) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:55:02 (1782928502) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:55:03 (1782928503) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0210918 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x760000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x25f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x25f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x25f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe40000 Read this (4 bytes): BBBB 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:55:18 (1782928518) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00358169 s, 2.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:55:20 (1782928520) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134893 s, 77.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6aa lmm_fid: [0x200000404:0x6aa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6aa lmm_fid: [0x200000404:0x6aa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835626 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-01 13:55:21.000000000 -0400 Modify: 2026-07-01 13:55:21.000000000 -0400 Change: 2026-07-01 13:55:21.000000000 -0400 Birth: 2026-07-01 13:55:22.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:55:23 (1782928523) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:55:24 (1782928524) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.10679 s, 384 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0138312 s, 296 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 1 13:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6ac:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 1 13:55 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:55:26 (1782928526) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:55:29 (1782928529) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 790 sec ============ 13:55:31 (1782928531) === sanity-flr: start cleanup 13:55:32 (1782928532) === === sanity-flr: finish cleanup 13:55:33 (1782928533) ===