-----============= acceptance-small: sanity-flr ============----- Sun Jan 11 09:01:30 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg353-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:01:36 (1768140096) === oleg353-client.virtnet: executing check_config_client /mnt/lustre oleg353-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg353-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90d2456a2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90d2456a2800.idle_timeout=debug disable quota as required oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:01:46 (1768140106) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7579] == sanity-flr test 0a: lfs mirror create with -N option == 09:01:46 (1768140106) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:01:51 (1768140111) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:01:52 (1768140112) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:01:53 (1768140113) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:01:59 (1768140119) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:02:00 (1768140120) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:02:01 (1768140121) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90d2456a2800.stats=clear osc.lustre-OST0001-osc-ffff90d2456a2800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:02:03 (1768140123) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:02:06 (1768140126) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:02:09 (1768140129) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:02:10 (1768140130) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 09:02:12 (1768140132) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:02:15 (1768140135) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:02:16 (1768140136) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:02:18 (1768140138) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:02:21 (1768140141) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:02:23 (1768140143) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.398308 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340218 s, 30.8 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:02:26 (1768140146) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.126478 s, 124 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:02:29 (1768140149) Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:02:56 (1768140176) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.165241 s, 38.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (19s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:03:16 (1768140196) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (28s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:03:44 (1768140224) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.67145 s, 31.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 11 09:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.387s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.288s user 0m0.000s sys 0m0.148s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.494s user 0m0.000s sys 0m0.162s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90d2456a2800.ost_server_uuid in FULL state after 0 sec PASS 33b (36s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:04:20 (1768140260) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:04:21 (1768140261) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:04:23 (1768140263) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:04:25 (1768140265) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 09:04:29 (1768140269) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.197518 s, 63.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:04:34 (1768140274) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168756 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163696 s, 64.1 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:04:37 (1768140277) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:04:41 (1768140281) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000879911 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000788293 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00471871 s, 217 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000656976 s, 1.6 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 09:04:45 (1768140285) create test files with size 15730653 .. 78b29c727beed361b367249e83e00667 - 23fd0972ab0479ab9bce0385396ddd7b - 7b219ad5ef20520bbea3eb6a4340848a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 09:04:54 (1768140294) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:05:01 (1768140301) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:05:04 (1768140304) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663443 s, 15.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101301 s, 104 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:05:08 (1768140308) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.076955 s, 27.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.183945 s, 22.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0698406 s, 30.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 09:05:14 (1768140314) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:05:15 (1768140315) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:05:17 (1768140317) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189173 s, 55.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 09:05:20 (1768140320) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:05:21 (1768140321) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:05:26 (1768140326) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.15951 s, 65.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:05:43 (1768140343) llite.lustre-ffff90d246232000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0962988 s, 109 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff90d25fd29000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:05:47 (1768140347) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:06:02 (1768140362) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:06:15 (1768140375) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:06:16 (1768140376) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.105747 s, 29.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.070993 s, 44.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd7bdbb52 (stale) Mirror 2: 0xd7bdbb52 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 8f42702f8c009ee9d80c1ce16ff0fb25 - mirror 1 checksum: 02811b6a8248834256e6faf621c77f98 - mirror 2 checksum: 8f42702f8c009ee9d80c1ce16ff0fb25 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jan 11 09:06:18 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jan 11 09:06:18 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8f42702f8c009ee9d80c1ce16ff0fb25 - mirror 1 checksum: 02811b6a8248834256e6faf621c77f98 - mirror 2 checksum: 02811b6a8248834256e6faf621c77f98 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jan 11 09:06:18 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jan 11 09:06:18 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:06:21 (1768140381) Failing mds1 on oleg353-server Stopping /mnt/lustre-mds1 (opts:) on oleg353-server 09:06:23 (1768140383) shut down facet: mds1 facet_host: oleg353-server facet_failover_host: oleg353-server Failover mds1 to oleg353-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 09:06:38 (1768140398) targets are mounted 09:06:38 (1768140398) facet_failover done oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171558 s, 61.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:06:45 (1768140405) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:06:46 (1768140406) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:06:47 (1768140407) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00302509 s, 5.4 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:06:51 (1768140411) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:06:52 (1768140412) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:06:53 (1768140413) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:07:00 (1768140420) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:07:19 (1768140439) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:07:37 (1768140457) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:07:55 (1768140475) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:07:57 (1768140477) mirror create pid 52240 mirror split pid 52241 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52240 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52241 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 09:09:00 (1768140540) Seed set to 541 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1768140541.280480 write 0x016f9d thru 0x01b3d8 (0x443c bytes) 000004 1768140541.285244 MIRROR_EXTEND to 2 mirrors 000005 1768140541.579550 MIRROR_RESYNC 2 mirrors 000006 1768140541.731753 mapwrite 0x02029e thru 0x02e076 (0xddd9 bytes) 000007 1768140541.815702 write 0x01260d thru 0x01945c (0x6e50 bytes) 000008 1768140541.821962 read 0x01c3af thru 0x0214a8 (0x50fa bytes) truncating to largest ever: 0x7de 000010 1768140541.823105 trunc from 0x02e077 to 0x0007dd (0x2d899 bytes) 000012 1768140541.835768 read 0x00070e thru 0x0007dd (0xd0 bytes) 000013 1768140541.846948 mapwrite 0x005eee thru 0x0064ed (0x600 bytes) 000016 1768140541.875409 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x64ee): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x64ee): Mirror 2: 0xef8f8fe3 (stale) CRC-32 checksum value for chunk [0, 0x64ee): Mirror 1: 0x2701897c Mirror 3: 0x2701897c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x64ee: skipped 000017 1768140542.115458 read 0x0061f4 thru 0x0064ed (0x2fa bytes) 000018 1768140542.117527 write 0x028baf thru 0x0355d0 (0xca22 bytes) truncating to largest ever: 0x3c8b6 000021 1768140542.147065 trunc from 0x0355d1 to 0x03c8b5 (0x72e5 bytes) 000028 1768140542.161987 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8b6): Mirror 3: 0xae9f1ecd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8b6): Mirror 3: 0xae9f1ecd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8b6: skipped 000029 1768140542.259724 mapwrite 0x002455 thru 0x007991 (0x553d bytes) 000030 1768140542.285118 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000031 1768140542.321568 mapread 0x00aec0 thru 0x01a24f (0xf390 bytes) 000032 1768140542.328433 read 0x028cac thru 0x037266 (0xe5bb bytes) 000034 1768140542.331653 mapread 0x02829d thru 0x02f59c (0x7300 bytes) 000035 1768140542.332307 mapwrite 0x010fbf thru 0x017682 (0x66c4 bytes) 000039 1768140542.346413 mapwrite 0x001dcc thru 0x008ed8 (0x710d bytes) 000040 1768140542.362439 read 0x029592 thru 0x0372b1 (0xdd20 bytes) 000041 1768140542.363111 write 0x0009f9 thru 0x00be3d (0xb445 bytes) 000043 1768140542.368453 mapwrite 0x00f85d thru 0x01a72d (0xaed1 bytes) 000044 1768140542.385371 mapwrite 0x03ae30 thru 0x03ffff (0x51d0 bytes) truncating to largest ever: 0x3daa3 000045 1768140542.406312 trunc from 0x040000 to 0x03daa2 (0x255d bytes) 000047 1768140542.411894 read 0x014075 thru 0x02224d (0xe1d9 bytes) 000048 1768140542.412402 write 0x00aa92 thru 0x016174 (0xb6e3 bytes) 000049 1768140542.413133 mapwrite 0x01f5b0 thru 0x0219bf (0x2410 bytes) 000050 1768140542.432465 write 0x03abd0 thru 0x03ffff (0x5430 bytes) 000051 1768140542.433331 mapwrite 0x03d956 thru 0x03ffff (0x26aa bytes) 000055 1768140542.449587 trunc from 0x040000 to 0x00ad7c (0x35283 bytes) 000056 1768140542.455197 mapwrite 0x01faea thru 0x0200df (0x5f6 bytes) 000062 1768140542.480956 write 0x027cc7 thru 0x034e37 (0xd171 bytes) 000064 1768140542.482498 trunc from 0x034e38 to 0x01af81 (0x19eb6 bytes) 000068 1768140542.491687 trunc from 0x01af82 to 0x01185c (0x9725 bytes) 000069 1768140542.496885 mapwrite 0x00a37f thru 0x017dc7 (0xda49 bytes) 000070 1768140542.526548 mapread 0x0038ef thru 0x006995 (0x30a7 bytes) 000074 1768140542.529188 mapread 0x013b12 thru 0x017dc7 (0x42b6 bytes) 000075 1768140542.532146 MIRROR_EXTEND to 2 mirrors 000080 1768140542.718663 mapwrite 0x03155b thru 0x0383bf (0x6e65 bytes) 000081 1768140542.770081 read 0x020b65 thru 0x024a11 (0x3ead bytes) 000083 1768140542.774603 MIRROR_RESYNC 2 mirrors 000084 1768140542.965811 trunc from 0x0383c0 to 0x026d11 (0x116ae bytes) 000088 1768140542.990942 read 0x01401b thru 0x01707b (0x3061 bytes) 000089 1768140542.994508 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000090 1768140543.032639 write 0x014249 thru 0x023001 (0xedb9 bytes) 000093 1768140543.037296 write 0x021cb3 thru 0x02abb6 (0x8f04 bytes) 000094 1768140543.038249 read 0x006015 thru 0x00f894 (0x9880 bytes) 000097 1768140543.042800 mapread 0x008222 thru 0x017663 (0xf442 bytes) 000098 1768140543.048140 read 0x013dcd thru 0x018251 (0x4485 bytes) 000101 1768140543.049336 write 0x005bbb thru 0x00ba14 (0x5e5a bytes) 000105 1768140543.053434 read 0x01c321 thru 0x028227 (0xbf07 bytes) 000107 1768140543.054107 mapread 0x01291e thru 0x021f05 (0xf5e8 bytes) 000108 1768140543.054939 mapread 0x01b1c0 thru 0x022fda (0x7e1b bytes) 000109 1768140543.055496 write 0x0357ed thru 0x03ee67 (0x967b bytes) 000110 1768140543.056294 mapread 0x00d3db thru 0x01be32 (0xea58 bytes) 000115 1768140543.058430 write 0x035816 thru 0x039d51 (0x453c bytes) 000116 1768140543.059042 read 0x00d0b7 thru 0x01ca56 (0xf9a0 bytes) 000117 1768140543.059376 write 0x02bffa thru 0x02c822 (0x829 bytes) 000118 1768140543.062682 mapread 0x021f5c thru 0x022595 (0x63a bytes) 000120 1768140543.064014 mapread 0x02dac6 thru 0x0307a8 (0x2ce3 bytes) 000125 1768140543.066362 write 0x022451 thru 0x02c4be (0xa06e bytes) 000126 1768140543.067237 mapwrite 0x00ee74 thru 0x01dc96 (0xee23 bytes) 000130 1768140543.085329 trunc from 0x03ee68 to 0x00a20b (0x34c5c bytes) 000133 1768140543.090730 trunc from 0x00a20c to 0x01db26 (0x1391b bytes) 000139 1768140543.096435 mapwrite 0x00ebbf thru 0x013d66 (0x51a8 bytes) 000140 1768140543.124812 write 0x02ea7c thru 0x03e5ea (0xfb6f bytes) 000142 1768140543.126409 mapread 0x01e49c thru 0x02c7d9 (0xe33e bytes) 000144 1768140543.150736 mapread 0x00b884 thru 0x00e8a6 (0x3023 bytes) 000145 1768140543.155391 mapwrite 0x00e970 thru 0x01795d (0x8fee bytes) 000147 1768140543.180967 MIRROR_EXTEND to 2 mirrors 000150 1768140543.356251 mapread 0x01ccb3 thru 0x026520 (0x986e bytes) 000153 1768140543.362349 MIRROR_RESYNC 2 mirrors 000155 1768140543.463740 write 0x01a62c thru 0x0207cd (0x61a2 bytes) 000156 1768140543.486132 trunc from 0x03e5eb to 0x01d605 (0x20fe5 bytes) 000157 1768140543.496734 MIRROR_RESYNC 2 mirrors 000159 1768140543.666758 write 0x00243a thru 0x010b43 (0xe70a bytes) 000160 1768140543.685470 mapread 0x0103a5 thru 0x017e2a (0x7a86 bytes) 000161 1768140543.688478 mapread 0x01cd10 thru 0x01d605 (0x8f6 bytes) 000162 1768140543.690904 mapwrite 0x00a3c1 thru 0x012ecb (0x8b0b bytes) 000163 1768140543.708943 mapwrite 0x00e0ed thru 0x011020 (0x2f34 bytes) 000164 1768140543.720486 read 0x01340d thru 0x01c36a (0x8f5e bytes) 000165 1768140543.720979 write 0x029cd1 thru 0x02deab (0x41db bytes) 000166 1768140543.721687 write 0x0103e0 thru 0x0117ee (0x140f bytes) 000167 1768140543.722269 mapread 0x02670e thru 0x02deab (0x779e bytes) 000168 1768140543.726311 MIRROR_RESYNC 2 mirrors 000169 1768140543.884613 read 0x018c8f thru 0x020fa6 (0x8318 bytes) 000170 1768140543.887236 write 0x00b1e4 thru 0x012528 (0x7345 bytes) 000173 1768140543.913122 trunc from 0x02deac to 0x020e9a (0xd011 bytes) 000174 1768140543.930238 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20e9b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e9b): Mirror 2: 0x5c1e318c (stale) CRC-32 checksum value for chunk [0, 0x20e9b): Mirror 1: 0x6f795934 Mirror 3: 0x6f795934 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e9b: skipped 000175 1768140544.169505 mapwrite 0x03a841 thru 0x03ffff (0x57bf bytes) 000176 1768140544.208144 mapread 0x03f43b thru 0x03ffff (0xbc5 bytes) 000177 1768140544.209030 mapwrite 0x02a076 thru 0x02e620 (0x45ab bytes) 000179 1768140544.229384 mapwrite 0x02f79b thru 0x038880 (0x90e6 bytes) 000180 1768140544.247060 trunc from 0x040000 to 0x02e78b (0x11874 bytes) 000181 1768140544.253387 write 0x0375b8 thru 0x03ffff (0x8a48 bytes) 000184 1768140544.255478 mapread 0x02a0fe thru 0x02f1be (0x50c1 bytes) 000185 1768140544.258733 read 0x0365e5 thru 0x03ea58 (0x8474 bytes) 000187 1768140544.259390 trunc from 0x040000 to 0x010f78 (0x2f087 bytes) 000188 1768140544.267697 write 0x03dc5a thru 0x03ffff (0x23a6 bytes) 000191 1768140544.269643 trunc from 0x040000 to 0x011201 (0x2edfe bytes) 000192 1768140544.280313 read 0x00d85c thru 0x011201 (0x39a6 bytes) 000194 1768140544.285441 mapwrite 0x039647 thru 0x03db3a (0x44f4 bytes) 000198 1768140544.325726 mapread 0x03aade thru 0x03c1f2 (0x1715 bytes) 000199 1768140544.326556 trunc from 0x03db3b to 0x004000 (0x39b3a bytes) 000204 1768140544.335253 mapread 0x000a13 thru 0x003934 (0x2f22 bytes) 000205 1768140544.335955 read 0x0008e3 thru 0x004000 (0x371e bytes) 000207 1768140544.336605 trunc from 0x004001 to 0x00405a (0x5a bytes) 000208 1768140544.341501 write 0x00fe4b thru 0x015856 (0x5a0c bytes) 000209 1768140544.345434 trunc from 0x015857 to 0x027e61 (0x1260b bytes) 000212 1768140544.352141 MIRROR_RESYNC 3 mirrors 000215 1768140544.599779 mapwrite 0x0389b2 thru 0x03ffff (0x764e bytes) 000216 1768140544.677243 read 0x00d75b thru 0x01af82 (0xd828 bytes) 000217 1768140544.683822 MIRROR_RESYNC 3 mirrors 000218 1768140544.893233 read 0x03d8c6 thru 0x03ffff (0x273a bytes) 000219 1768140544.895577 write 0x02b166 thru 0x039eca (0xed65 bytes) 000222 1768140544.923578 write 0x03ed8f thru 0x03ffff (0x1271 bytes) 000224 1768140544.927889 trunc from 0x040000 to 0x026dfa (0x19205 bytes) 000225 1768140544.944977 read 0x0037c7 thru 0x01328d (0xfac7 bytes) 000226 1768140544.949556 mapread 0x0015f5 thru 0x0067b8 (0x51c4 bytes) 000228 1768140544.950883 mapwrite 0x010eb0 thru 0x0190f9 (0x824a bytes) 000229 1768140544.972805 trunc from 0x026dfb to 0x03c497 (0x1569d bytes) 000232 1768140544.985178 mapread 0x037d17 thru 0x039909 (0x1bf3 bytes) 000233 1768140544.989590 mapwrite 0x01424a thru 0x020b23 (0xc8da bytes) 000236 1768140545.019614 read 0x008b03 thru 0x00981d (0xd1b bytes) 000237 1768140545.020021 read 0x0144fc thru 0x01eddb (0xa8e0 bytes) 000239 1768140545.020618 write 0x03c862 thru 0x03ffff (0x379e bytes) 000241 1768140545.024172 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95b7002d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95b7002d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1768140545.135048 mapwrite 0x031c2d thru 0x03ffff (0xe3d3 bytes) 000245 1768140545.170343 mapread 0x00a975 thru 0x013997 (0x9023 bytes) 000246 1768140545.179509 mapread 0x00ff13 thru 0x015188 (0x5276 bytes) 000247 1768140545.180295 read 0x02afd5 thru 0x037229 (0xc255 bytes) 000248 1768140545.185116 trunc from 0x040000 to 0x0159c5 (0x2a63a bytes) 000251 1768140545.195059 write 0x00bf76 thru 0x0141be (0x8249 bytes) 000252 1768140545.199680 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x159c6): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x159c6): Mirror 3: 0xd2e44a25 (stale) CRC-32 checksum value for chunk [0, 0x159c6): Mirror 1: 0xefa5d5b Mirror 4: 0xefa5d5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x159c6: skipped 000253 1768140545.438727 mapread 0x00600b thru 0x00eace (0x8ac4 bytes) 000255 1768140545.445030 mapwrite 0x0208ed thru 0x02ce0b (0xc51f bytes) 000256 1768140545.504655 write 0x022ef4 thru 0x02a744 (0x7851 bytes) 000257 1768140545.505610 mapread 0x02650a thru 0x02c485 (0x5f7c bytes) 000259 1768140545.506527 write 0x0284d7 thru 0x037909 (0xf433 bytes) 000260 1768140545.507697 mapwrite 0x026340 thru 0x02ed56 (0x8a17 bytes) 000265 1768140545.527806 read 0x00fd7e thru 0x011c23 (0x1ea6 bytes) 000266 1768140545.531761 write 0x01831d thru 0x01ff16 (0x7bfa bytes) 000267 1768140545.535360 mapread 0x016118 thru 0x019b51 (0x3a3a bytes) 000268 1768140545.536002 mapread 0x003c78 thru 0x00f874 (0xbbfd bytes) 000269 1768140545.542255 MIRROR_RESYNC 3 mirrors 000270 1768140545.775137 mapread 0x033ce1 thru 0x037909 (0x3c29 bytes) 000271 1768140545.780215 write 0x031e5c thru 0x03ddfb (0xbfa0 bytes) 000273 1768140545.804703 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ddfc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddfc): Mirror 3: 0xdeb40448 (stale) Mirror 4: 0xdeb40448 (stale) CRC-32 checksum value for chunk [0, 0x3ddfc): Mirror 1: 0xcfc45822 Mirror 5: 0xcfc45822 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddfc: skipped 000274 1768140546.071510 write 0x037eca thru 0x03975d (0x1894 bytes) 000279 1768140546.100162 mapread 0x015819 thru 0x0201d7 (0xa9bf bytes) 000280 1768140546.109698 write 0x01af5a thru 0x02656d (0xb614 bytes) 000282 1768140546.110872 trunc from 0x03ddfc to 0x0331c4 (0xac37 bytes) 000284 1768140546.124357 mapread 0x00e69e thru 0x011189 (0x2aec bytes) 000287 1768140546.126099 mapread 0x01b73e thru 0x0254e5 (0x9da8 bytes) 000289 1768140546.127359 mapwrite 0x0084cc thru 0x00f39f (0x6ed4 bytes) 000291 1768140546.150826 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x331c5): Mirror 3: 0x624f9743 (stale) Mirror 4: 0x624f9743 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x331c5): Mirror 3: 0x624f9743 (stale) Mirror 4: 0x624f9743 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x331c5: skipped 000292 1768140546.246750 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x331c5): Mirror 3: 0x624f9743 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x331c5): Mirror 3: 0x624f9743 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x331c5: skipped 000296 1768140546.318262 read 0x01f4fd thru 0x02dd77 (0xe87b bytes) 000297 1768140546.323308 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000298 1768140546.359527 mapread 0x004ff2 thru 0x00ab8d (0x5b9c bytes) 000299 1768140546.365611 mapread 0x01b264 thru 0x0247e4 (0x9581 bytes) 000303 1768140546.372863 read 0x02bae3 thru 0x0331c4 (0x76e2 bytes) 000304 1768140546.376600 mapread 0x001329 thru 0x009554 (0x822c bytes) 000306 1768140546.377746 trunc from 0x0331c5 to 0x034ca6 (0x1ae2 bytes) 000308 1768140546.387966 read 0x027ca0 thru 0x03206d (0xa3ce bytes) 000309 1768140546.388373 read 0x004640 thru 0x005659 (0x101a bytes) 000312 1768140546.389458 write 0x00558a thru 0x006992 (0x1409 bytes) 000313 1768140546.390152 write 0x00882e thru 0x0118ce (0x90a1 bytes) 000317 1768140546.395054 write 0x0084dd thru 0x008725 (0x249 bytes) 000318 1768140546.395578 trunc from 0x034ca7 to 0x013022 (0x21c84 bytes) 000319 1768140546.403958 trunc from 0x013023 to 0x018938 (0x5916 bytes) 000322 1768140546.409024 mapwrite 0x02f1c6 thru 0x036cc4 (0x7aff bytes) 000323 1768140546.442455 write 0x00ce59 thru 0x01c91a (0xfac2 bytes) 000325 1768140546.444324 read 0x02eb3f thru 0x036cc4 (0x8186 bytes) 000329 1768140546.446358 mapread 0x02a376 thru 0x033303 (0x8f8e bytes) 000331 1768140546.447718 mapread 0x0321b3 thru 0x036cc4 (0x4b12 bytes) 000332 1768140546.448566 read 0x02fff5 thru 0x036cc4 (0x6cd0 bytes) 000333 1768140546.449271 trunc from 0x036cc5 to 0x01ee0e (0x17eb6 bytes) 000336 1768140546.457941 MIRROR_EXTEND to 2 mirrors 000338 1768140546.642184 read 0x010b8a thru 0x013d1b (0x3192 bytes) 000339 1768140546.644009 write 0x02e2f8 thru 0x0340ab (0x5db4 bytes) 000341 1768140546.666086 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x340ac): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340ac): Mirror 2: 0x5d3cb818 (stale) CRC-32 checksum value for chunk [0, 0x340ac): Mirror 1: 0x8c0def52 Mirror 3: 0x8c0def52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340ac: skipped 000342 1768140546.870343 read 0x02b01a thru 0x0340ab (0x9092 bytes) 000344 1768140546.873357 mapread 0x007bbc thru 0x0128f7 (0xad3c bytes) 000345 1768140546.881291 mapread 0x02bb24 thru 0x031077 (0x5554 bytes) 000347 1768140546.882212 write 0x01d427 thru 0x022e8d (0x5a67 bytes) 000349 1768140546.906177 mapread 0x0039ae thru 0x009d5e (0x63b1 bytes) 000350 1768140546.909134 trunc from 0x0340ac to 0x029a5b (0xa650 bytes) 000352 1768140546.920007 trunc from 0x029a5c to 0x01af9c (0xeabf bytes) 000354 1768140546.925607 trunc from 0x01af9d to 0x0148a8 (0x66f4 bytes) 000355 1768140546.930443 write 0x00f1f0 thru 0x01d401 (0xe212 bytes) 000356 1768140546.931829 write 0x009b3b thru 0x015948 (0xbe0e bytes) 000357 1768140546.933045 write 0x03fc9a thru 0x03ffff (0x366 bytes) 000362 1768140546.938016 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4149cdb0 (stale) Mirror 3: 0x4f8a0d9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ec76261 Mirror 4: 0x5ec76261 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000364 1768140547.177581 write 0x02e805 thru 0x03c047 (0xd843 bytes) 000365 1768140547.205163 read 0x00ca15 thru 0x013f18 (0x7504 bytes) 000366 1768140547.210026 mapwrite 0x03c281 thru 0x03ffff (0x3d7f bytes) 000367 1768140547.231006 trunc from 0x040000 to 0x01f5e9 (0x20a16 bytes) 000368 1768140547.238043 mapwrite 0x01f40f thru 0x024a93 (0x5685 bytes) 000369 1768140547.262398 mapwrite 0x002478 thru 0x004535 (0x20be bytes) 000370 1768140547.277972 trunc from 0x024a94 to 0x02ab06 (0x6073 bytes) truncating to largest ever: 0x3edff 000371 1768140547.283050 trunc from 0x02ab07 to 0x03edfe (0x142f8 bytes) 000372 1768140547.287309 mapread 0x00daec thru 0x017517 (0x9a2c bytes) 000374 1768140547.289974 write 0x03c1ea thru 0x03ffff (0x3e16 bytes) 000378 1768140547.293955 mapwrite 0x02243b thru 0x02cdec (0xa9b2 bytes) 000379 1768140547.312657 write 0x0388cb thru 0x03ffff (0x7735 bytes) 000381 1768140547.315399 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4f8a0d9a (stale) Mirror 4: 0x5ec76261 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4f8a0d9a (stale) Mirror 4: 0x5ec76261 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1768140547.407193 trunc from 0x040000 to 0x00e2c0 (0x31d3f bytes) 000384 1768140547.412028 mapread 0x0024b2 thru 0x0079fb (0x554a bytes) 000387 1768140547.417509 mapwrite 0x0116be thru 0x011bdb (0x51e bytes) 000388 1768140547.435835 write 0x01926e thru 0x0207d2 (0x7565 bytes) 000391 1768140547.437749 trunc from 0x0207d3 to 0x01902a (0x77a8 bytes) 000393 1768140547.443826 write 0x007aef thru 0x015fb8 (0xe4ca bytes) 000395 1768140547.447137 read 0x00b6a8 thru 0x01902a (0xd983 bytes) 000396 1768140547.449281 mapread 0x00a5c0 thru 0x018db3 (0xe7f4 bytes) 000398 1768140547.450539 read 0x003710 thru 0x0076e9 (0x3fda bytes) 000401 1768140547.451925 read 0x007a5b thru 0x0149f6 (0xcf9c bytes) 000402 1768140547.452410 mapwrite 0x02f293 thru 0x0374cf (0x823d bytes) 000405 1768140547.475174 trunc from 0x0374d0 to 0x036ea7 (0x628 bytes) 000406 1768140547.480229 read 0x025963 thru 0x027067 (0x1705 bytes) 000407 1768140547.480778 trunc from 0x036ea8 to 0x01b460 (0x1ba47 bytes) 000408 1768140547.485164 trunc from 0x01b461 to 0x02b660 (0x10200 bytes) 000409 1768140547.489365 read 0x024571 thru 0x025d33 (0x17c3 bytes) 000411 1768140547.493569 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b661): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b661): Mirror 3: 0x1e46a407 (stale) Mirror 4: 0xf8b95b26 (stale) CRC-32 checksum value for chunk [0, 0x2b661): Mirror 1: 0x74588555 Mirror 5: 0x74588555 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b661: skipped 000413 1768140547.755604 mapread 0x016f03 thru 0x01d396 (0x6494 bytes) 000414 1768140547.764027 mapread 0x011a74 thru 0x015360 (0x38ed bytes) 000415 1768140547.766992 mapread 0x026755 thru 0x02b660 (0x4f0c bytes) 000416 1768140547.772047 mapwrite 0x03cff4 thru 0x03ffff (0x300c bytes) 000417 1768140547.821594 trunc from 0x040000 to 0x03baf7 (0x4508 bytes) 000418 1768140547.829984 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baf8): Mirror 4: 0xed3136c3 (stale) Mirror 5: 0x19eafa77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baf8): Mirror 4: 0xed3136c3 (stale) Mirror 5: 0x19eafa77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baf8: skipped 000419 1768140547.949951 trunc from 0x03baf8 to 0x017bde (0x23f19 bytes) 000421 1768140547.955216 write 0x03fa51 thru 0x03ffff (0x5af bytes) 000423 1768140547.956536 read 0x016adc thru 0x01d542 (0x6a67 bytes) 000427 1768140547.960789 mapread 0x03195a thru 0x03a131 (0x87d8 bytes) 000428 1768140547.968083 write 0x000663 thru 0x001c07 (0x15a5 bytes) 000430 1768140547.972082 mapwrite 0x0298b5 thru 0x035351 (0xba9d bytes) 000431 1768140547.991478 mapread 0x01231d thru 0x01ef6f (0xcc53 bytes) 000433 1768140548.002301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ec76261 (stale) Mirror 5: 0xdb78d04f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7cbb9965 Mirror 6: 0x7cbb9965 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1768140548.375066 write 0x031876 thru 0x03bd82 (0xa50d bytes) 000436 1768140548.429686 trunc from 0x040000 to 0x03e16a (0x1e95 bytes) 000437 1768140548.442143 read 0x01623e thru 0x01de52 (0x7c15 bytes) 000441 1768140548.446201 mapwrite 0x01db7b thru 0x027cf9 (0xa17f bytes) 000442 1768140548.469393 read 0x01c248 thru 0x021cb0 (0x5a69 bytes) 000443 1768140548.469834 write 0x00ffd5 thru 0x010090 (0xbc bytes) 000444 1768140548.477605 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: 0xbe82294e (stale) Mirror 5: 0xc783d7ad (stale) Mirror 6: 0xcd037b65 (stale) CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 1: 0x29bde391 Mirror 7: 0x29bde391 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e16b: skipped 000445 1768140548.766229 write 0x012583 thru 0x014661 (0x20df bytes) 000446 1768140548.804014 read 0x013a74 thru 0x0227ae (0xed3b bytes) 000449 1768140548.808764 mapread 0x00df72 thru 0x015021 (0x70b0 bytes) 000451 1768140548.814670 mapwrite 0x02036b thru 0x021595 (0x122b bytes) 000452 1768140548.831300 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: 0xbe82294e (stale) Mirror 5: 0xc783d7ad (stale) Mirror 6: 0xcd037b65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: 0xbe82294e (stale) Mirror 5: 0xc783d7ad (stale) Mirror 6: 0xcd037b65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e16b: skipped 000456 1768140548.957824 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 4: 0xbe82294e (stale) Mirror 5: 0xc783d7ad (stale) Mirror 6: 0xcd037b65 (stale) CRC-32 checksum value for chunk [0, 0x3e16b): Mirror 1: 0x4374c75 Mirror 7: 0x4374c75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e16b: skipped 000457 1768140549.275990 read 0x0276aa thru 0x030320 (0x8c77 bytes) 000458 1768140549.287038 write 0x008897 thru 0x00e2f1 (0x5a5b bytes) 000459 1768140549.314356 mapread 0x002113 thru 0x00d9df (0xb8cd bytes) 000460 1768140549.317670 write 0x035b64 thru 0x03b89c (0x5d39 bytes) 000461 1768140549.321770 trunc from 0x03e16b to 0x02f317 (0xee53 bytes) 000462 1768140549.327986 mapread 0x001d97 thru 0x0028dc (0xb46 bytes) 000463 1768140549.330739 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f318): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f318): Mirror 4: 0xe04f9938 (stale) Mirror 5: 0xa92aa890 (stale) Mirror 6: 0x9d5f5f3d (stale) Mirror 7: 0x8bb68bd0 (stale) CRC-32 checksum value for chunk [0, 0x2f318): Mirror 1: 0x9350b939 Mirror 8: 0x9350b939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f318: skipped 000466 1768140549.578166 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f318): Mirror 4: 0xe04f9938 (stale) Mirror 5: 0xa92aa890 (stale) Mirror 7: 0x8bb68bd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f318): Mirror 4: 0xe04f9938 (stale) Mirror 5: 0xa92aa890 (stale) Mirror 7: 0x8bb68bd0 (stale) CRC-32 checksum value for chunk [0, 0x2f318): Mirror 1: 0x9350b939 Mirror 8: 0x9350b939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f318: skipped 000467 1768140549.770075 mapread 0x012483 thru 0x01df1d (0xba9b bytes) 000468 1768140549.791361 write 0x0330a5 thru 0x03cec1 (0x9e1d bytes) 000470 1768140549.828140 write 0x03d18b thru 0x03ffff (0x2e75 bytes) 000471 1768140549.829006 trunc from 0x040000 to 0x0048a8 (0x3b757 bytes) 000472 1768140549.849375 mapread 0x0044e4 thru 0x0048a8 (0x3c5 bytes) 000475 1768140549.852863 write 0x03460b thru 0x03ffff (0xb9f5 bytes) 000478 1768140549.854875 write 0x00a9a5 thru 0x013242 (0x889e bytes) 000479 1768140549.859551 mapread 0x009bb3 thru 0x014d9e (0xb1ec bytes) 000485 1768140549.865517 MIRROR_RESYNC 5 mirrors 000487 1768140550.181637 mapread 0x02e8b9 thru 0x038b64 (0xa2ac bytes) 000489 1768140550.187564 write 0x00897c thru 0x00b565 (0x2bea bytes) 000493 1768140550.208121 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xca89b321 (stale) Mirror 5: 0xca89b321 (stale) Mirror 7: 0xca89b321 (stale) Mirror 8: 0xca89b321 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x161dae55 Mirror 9: 0x161dae55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1768140550.479970 mapread 0x031f76 thru 0x039d28 (0x7db3 bytes) 000495 1768140550.486983 mapwrite 0x02bc2e thru 0x0378bd (0xbc90 bytes) 000496 1768140550.568378 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xca89b321 (stale) Mirror 7: 0xca89b321 (stale) Mirror 8: 0xca89b321 (stale) Mirror 9: 0x161dae55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xca89b321 (stale) Mirror 7: 0xca89b321 (stale) Mirror 8: 0xca89b321 (stale) Mirror 9: 0x161dae55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1768140550.680189 mapread 0x02f767 thru 0x0359b5 (0x624f bytes) 000499 1768140550.687052 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xca89b321 (stale) Mirror 8: 0xca89b321 (stale) Mirror 9: 0x161dae55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xca89b321 (stale) Mirror 8: 0xca89b321 (stale) Mirror 9: 0x161dae55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1768140550.823635 mapwrite 0x00a035 thru 0x00cae1 (0x2aad bytes) 000502 1768140550.846419 mapread 0x00dc41 thru 0x01d2aa (0xf66a bytes) 000507 1768140550.850769 write 0x024cad thru 0x031684 (0xc9d8 bytes) 000510 1768140550.855829 read 0x003ae7 thru 0x00d723 (0x9c3d bytes) 000511 1768140550.857763 write 0x024a40 thru 0x02e53a (0x9afb bytes) 000512 1768140550.858608 read 0x01baef thru 0x023526 (0x7a38 bytes) 000513 1768140550.859125 trunc from 0x040000 to 0x033cdd (0xc322 bytes) 000514 1768140550.864406 mapwrite 0x0385c0 thru 0x03a318 (0x1d59 bytes) 000518 1768140550.889494 read 0x00d9e2 thru 0x019d08 (0xc327 bytes) 000520 1768140550.892157 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: 0xb32099f6 (stale) Mirror 8: 0xb32099f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: 0xb32099f6 (stale) Mirror 8: 0xb32099f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a319: skipped 000522 1768140551.026761 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: 0xb32099f6 (stale) Mirror 8: 0xb32099f6 (stale) CRC-32 checksum value for chunk [0, 0x3a319): Mirror 1: 0x2a508d4f Mirror 9: 0x2a508d4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a319: skipped 000523 1768140551.349225 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: 0xb32099f6 (stale) Mirror 8: 0xb32099f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a319): Mirror 7: 0xb32099f6 (stale) Mirror 8: 0xb32099f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a319: skipped 000524 1768140551.484746 MIRROR_RESYNC 3 mirrors 000525 1768140551.794939 read 0x01f6b3 thru 0x02ba35 (0xc383 bytes) 000527 1768140551.801099 read 0x021351 thru 0x02d2be (0xbf6e bytes) 000528 1768140551.807520 write 0x03779e thru 0x03c6c1 (0x4f24 bytes) 000529 1768140551.844442 write 0x019225 thru 0x021427 (0x8203 bytes) 000534 1768140551.849534 read 0x02d12c thru 0x03bcc9 (0xeb9e bytes) 000537 1768140551.857631 read 0x01c9e0 thru 0x02372c (0x6d4d bytes) 000538 1768140551.858094 trunc from 0x03c6c2 to 0x0355ee (0x70d3 bytes) 000539 1768140551.867023 MIRROR_RESYNC 3 mirrors 000540 1768140552.099993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000541 1768140552.185341 write 0x02c2f6 thru 0x0397d4 (0xd4df bytes) 000542 1768140552.202308 mapread 0x028341 thru 0x036765 (0xe425 bytes) 000543 1768140552.206306 mapread 0x012b92 thru 0x019f30 (0x739f bytes) 000544 1768140552.209532 write 0x005200 thru 0x00f172 (0x9f73 bytes) 000545 1768140552.212529 read 0x011dc4 thru 0x016503 (0x4740 bytes) 000546 1768140552.214594 MIRROR_RESYNC 2 mirrors 000548 1768140552.377353 mapwrite 0x032c3f thru 0x03e9e6 (0xbda8 bytes) 000549 1768140552.428647 read 0x0168cb thru 0x019e3f (0x3575 bytes) 000552 1768140552.431138 mapwrite 0x01ef05 thru 0x026a5e (0x7b5a bytes) 000554 1768140552.450160 read 0x014ec5 thru 0x019d47 (0x4e83 bytes) 000555 1768140552.455388 MIRROR_RESYNC 2 mirrors 000556 1768140552.628952 write 0x017371 thru 0x02283d (0xb4cd bytes) 000559 1768140552.652748 mapread 0x031a0a thru 0x038701 (0x6cf8 bytes) 000560 1768140552.655697 trunc from 0x03e9e7 to 0x0249a6 (0x1a040 bytes) 000561 1768140552.666426 trunc from 0x0249a7 to 0x031c9b (0xd2f5 bytes) 000562 1768140552.673611 read 0x017047 thru 0x01a6aa (0x3664 bytes) 000564 1768140552.674740 mapwrite 0x036e9d thru 0x03ffff (0x9163 bytes) 000565 1768140552.710551 read 0x00f7cb thru 0x01aead (0xb6e3 bytes) 000568 1768140552.719940 mapwrite 0x0152ce thru 0x01ec1a (0x994d bytes) 000571 1768140552.738345 mapread 0x00f9b2 thru 0x018f41 (0x9590 bytes) 000573 1768140552.739517 write 0x000cc4 thru 0x0063a2 (0x56df bytes) 000574 1768140552.740490 trunc from 0x040000 to 0x0197db (0x26824 bytes) 000575 1768140552.746379 write 0x011822 thru 0x01eab2 (0xd291 bytes) 000576 1768140552.747670 read 0x01a2d1 thru 0x01eab2 (0x47e2 bytes) 000577 1768140552.748215 read 0x012700 thru 0x015b8d (0x348e bytes) 000578 1768140552.750406 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000580 1768140552.812450 read 0x004d0e thru 0x010b50 (0xbe43 bytes) 000582 1768140552.815438 trunc from 0x01eab3 to 0x02b923 (0xce71 bytes) 000584 1768140552.824173 mapread 0x017bb4 thru 0x020789 (0x8bd6 bytes) 000586 1768140552.831354 write 0x0206bb thru 0x026606 (0x5f4c bytes) 000588 1768140552.834217 mapread 0x01a1d2 thru 0x01e1ce (0x3ffd bytes) 000592 1768140552.835820 write 0x0274a6 thru 0x02dd9e (0x68f9 bytes) 000594 1768140552.840739 MIRROR_EXTEND to 2 mirrors 000595 1768140553.000286 read 0x00066b thru 0x001bd0 (0x1566 bytes) 000596 1768140553.003268 write 0x008c41 thru 0x00fc31 (0x6ff1 bytes) 000599 1768140553.029034 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dd9f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd9f): Mirror 2: 0x86e30c0d (stale) CRC-32 checksum value for chunk [0, 0x2dd9f): Mirror 1: 0x31c46ce3 Mirror 3: 0x31c46ce3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd9f: skipped 000601 1768140553.321896 trunc from 0x02dd9f to 0x017f3c (0x15e62 bytes) 000602 1768140553.343177 write 0x0181bf thru 0x018c49 (0xa8b bytes) 000603 1768140553.345381 MIRROR_RESYNC 3 mirrors 000604 1768140553.558461 write 0x0213f9 thru 0x0241e0 (0x2de8 bytes) 000606 1768140553.576297 trunc from 0x0241e1 to 0x00d071 (0x1716f bytes) 000607 1768140553.583087 trunc from 0x00d072 to 0x017abb (0xaa4a bytes) 000608 1768140553.588107 mapwrite 0x0313b6 thru 0x03dbab (0xc7f6 bytes) 000609 1768140553.616077 mapread 0x01878a thru 0x022125 (0x999c bytes) 000611 1768140553.622480 trunc from 0x03dbac to 0x01728a (0x26921 bytes) 000612 1768140553.628104 mapread 0x003b19 thru 0x00b9d5 (0x7ebd bytes) 000613 1768140553.631295 trunc from 0x01728b to 0x03530b (0x1e081 bytes) 000615 1768140553.635973 trunc from 0x03530c to 0x0087d8 (0x2cb33 bytes) 000616 1768140553.641324 mapwrite 0x01c484 thru 0x020a71 (0x45ee bytes) 000617 1768140553.661627 trunc from 0x020a72 to 0x013487 (0xd5ea bytes) 000620 1768140553.667218 write 0x031d62 thru 0x03783a (0x5ad9 bytes) 000621 1768140553.668298 write 0x033b15 thru 0x0354b5 (0x19a1 bytes) 000622 1768140553.669070 write 0x00baa5 thru 0x019000 (0xd55c bytes) 000623 1768140553.673888 mapwrite 0x00a540 thru 0x01748b (0xcf4c bytes) 000625 1768140553.691351 trunc from 0x03783b to 0x013a0f (0x23e2b bytes) 000626 1768140553.699243 mapwrite 0x02ab59 thru 0x03085a (0x5d02 bytes) 000627 1768140553.728036 trunc from 0x03085b to 0x022025 (0xe835 bytes) 000628 1768140553.732323 read 0x01c3a0 thru 0x022025 (0x5c86 bytes) 000629 1768140553.734787 mapread 0x001f7e thru 0x00a376 (0x83f9 bytes) 000631 1768140553.736117 mapwrite 0x016c41 thru 0x0177ca (0xb8a bytes) 000633 1768140553.750416 mapwrite 0x017b33 thru 0x0276a5 (0xfb73 bytes) 000634 1768140553.772606 mapread 0x002a69 thru 0x00fe39 (0xd3d1 bytes) 000636 1768140553.773961 mapwrite 0x03ed93 thru 0x03ffff (0x126d bytes) 000641 1768140553.797415 mapwrite 0x000d18 thru 0x002c89 (0x1f72 bytes) 000642 1768140553.815875 mapread 0x034313 thru 0x03ffff (0xbced bytes) 000643 1768140553.820724 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd09b413e (stale) Mirror 3: 0xd09b413e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4cbbba26 Mirror 4: 0x4cbbba26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1768140554.073432 write 0x0110c3 thru 0x0162a6 (0x51e4 bytes) 000647 1768140554.103262 trunc from 0x040000 to 0x015299 (0x2ad66 bytes) 000648 1768140554.118586 mapread 0x0096e7 thru 0x00e438 (0x4d52 bytes) 000651 1768140554.124798 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1529a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1529a): Mirror 2: 0x48fa5885 (stale) Mirror 3: 0x48fa5885 (stale) Mirror 4: 0xc0147e3a (stale) CRC-32 checksum value for chunk [0, 0x1529a): Mirror 1: 0xea646c0a Mirror 5: 0xea646c0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1529a: skipped 000652 1768140554.374771 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1529a): Mirror 3: 0x48fa5885 (stale) Mirror 4: 0xc0147e3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1529a): Mirror 3: 0x48fa5885 (stale) Mirror 4: 0xc0147e3a (stale) CRC-32 checksum value for chunk [0, 0x1529a): Mirror 1: 0xea646c0a Mirror 5: 0xea646c0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1529a: skipped 000654 1768140554.599615 mapread 0x005a00 thru 0x0102b3 (0xa8b4 bytes) 000655 1768140554.618395 read 0x002efc thru 0x0056df (0x27e4 bytes) 000657 1768140554.631003 mapread 0x0079ea thru 0x0103ac (0x89c3 bytes) 000659 1768140554.652040 read 0x00fd6b thru 0x010018 (0x2ae bytes) 000660 1768140554.660290 write 0x02a670 thru 0x034fb9 (0xa94a bytes) 000661 1768140554.700696 mapread 0x0340dc thru 0x034fb9 (0xede bytes) 000662 1768140554.702156 read 0x0074c9 thru 0x008481 (0xfb9 bytes) 000663 1768140554.706935 read 0x02c5d8 thru 0x034fb9 (0x89e2 bytes) 000664 1768140554.707508 read 0x02ca27 thru 0x02f0ae (0x2688 bytes) 000665 1768140554.707970 mapwrite 0x030845 thru 0x034d9e (0x455a bytes) 000667 1768140554.735449 read 0x024946 thru 0x02a391 (0x5a4c bytes) 000669 1768140554.736210 trunc from 0x034fba to 0x0071cd (0x2ddec bytes) 000670 1768140554.752149 trunc from 0x0071ce to 0x03572b (0x2e55e bytes) 000672 1768140554.757416 mapwrite 0x0103d0 thru 0x01e972 (0xe5a3 bytes) 000673 1768140554.810951 write 0x0062c9 thru 0x013fa0 (0xdcd8 bytes) 000675 1768140554.814959 mapwrite 0x025208 thru 0x030526 (0xb31f bytes) 000676 1768140554.856133 mapwrite 0x00bb14 thru 0x015421 (0x990e bytes) 000679 1768140554.887492 trunc from 0x03572c to 0x02b0ca (0xa661 bytes) 000680 1768140554.906351 read 0x01a0cd thru 0x0204fc (0x6430 bytes) 000681 1768140554.906726 write 0x003bcf thru 0x013a21 (0xfe53 bytes) 000682 1768140554.907672 write 0x02df16 thru 0x0301a2 (0x228d bytes) 000683 1768140554.908268 read 0x01a020 thru 0x01dee7 (0x3ec8 bytes) 000684 1768140554.908578 mapread 0x00fb79 thru 0x014533 (0x49bb bytes) 000687 1768140554.909770 mapread 0x00ae5e thru 0x00dd25 (0x2ec8 bytes) 000688 1768140554.912615 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301a3): Mirror 4: 0xc10e0c98 (stale) Mirror 5: 0xdbdb5b37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301a3): Mirror 4: 0xc10e0c98 (stale) Mirror 5: 0xdbdb5b37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301a3: skipped 000690 1768140555.058942 mapread 0x01c872 thru 0x021856 (0x4fe5 bytes) 000692 1768140555.075248 read 0x0222c8 thru 0x029203 (0x6f3c bytes) 000693 1768140555.079388 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301a3): Mirror 5: 0xdbdb5b37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301a3): Mirror 5: 0xdbdb5b37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301a3: skipped 000696 1768140555.178743 mapread 0x003789 thru 0x005b5b (0x23d3 bytes) 000697 1768140555.187463 write 0x00d346 thru 0x018ebd (0xbb78 bytes) 000698 1768140555.191069 mapwrite 0x020251 thru 0x029ce3 (0x9a93 bytes) 000700 1768140555.219389 mapwrite 0x0143f0 thru 0x02148a (0xd09b bytes) 000702 1768140555.237638 mapread 0x00f7fa thru 0x01575e (0x5f65 bytes) 000704 1768140555.238826 mapread 0x02ab8d thru 0x02c7a6 (0x1c1a bytes) 000708 1768140555.240914 write 0x000673 thru 0x00516c (0x4afa bytes) 000709 1768140555.244901 mapread 0x01bb86 thru 0x02b007 (0xf482 bytes) 000710 1768140555.245733 mapread 0x014d62 thru 0x015660 (0x8ff bytes) 000711 1768140555.248735 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x301a3): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301a3): Mirror 5: 0xdbdb5b37 (stale) CRC-32 checksum value for chunk [0, 0x301a3): Mirror 1: 0x390877ea Mirror 6: 0x390877ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301a3: skipped 000713 1768140555.530341 mapwrite 0x032839 thru 0x03ffff (0xd7c7 bytes) 000714 1768140555.598021 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9a373736 (stale) Mirror 6: 0xda090e63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1368bd7b Mirror 7: 0x1368bd7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000716 1768140555.861222 mapread 0x00cafa thru 0x00ff7f (0x3486 bytes) 000717 1768140555.873141 trunc from 0x040000 to 0x032851 (0xd7ae bytes) 000721 1768140555.930664 trunc from 0x032852 to 0x0021bd (0x30694 bytes) 000724 1768140555.943896 mapread 0x0003c0 thru 0x0021bd (0x1dfe bytes) 000725 1768140555.946432 mapread 0x001c69 thru 0x0021bd (0x555 bytes) 000726 1768140555.946986 write 0x007eae thru 0x016d47 (0xee9a bytes) 000728 1768140555.948061 write 0x00022a thru 0x001da1 (0x1b78 bytes) 000729 1768140555.948470 mapread 0x010efe thru 0x014684 (0x3787 bytes) 000730 1768140555.949219 mapread 0x015e4b thru 0x016d47 (0xefd bytes) 000731 1768140555.951689 MIRROR_RESYNC 4 mirrors 000732 1768140556.294762 mapread 0x0049f9 thru 0x00dff1 (0x95f9 bytes) 000734 1768140556.304459 mapwrite 0x01d055 thru 0x01e455 (0x1401 bytes) 000735 1768140556.382456 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e456): Mirror 6: 0xb6ab5ac1 (stale) Mirror 7: 0xb6ab5ac1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e456): Mirror 6: 0xb6ab5ac1 (stale) Mirror 7: 0xb6ab5ac1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e456: skipped 000736 1768140556.530188 mapread 0x019455 thru 0x01e455 (0x5001 bytes) 000737 1768140556.539278 mapwrite 0x02373f thru 0x02ffec (0xc8ae bytes) 000738 1768140556.581978 read 0x011bb3 thru 0x01756e (0x59bc bytes) 000741 1768140556.585524 mapwrite 0x038a51 thru 0x03cda8 (0x4358 bytes) 000745 1768140556.616113 trunc from 0x03cda9 to 0x03c7a6 (0x602 bytes) 000746 1768140556.620620 trunc from 0x03c7a7 to 0x0397a4 (0x3002 bytes) 000747 1768140556.625240 trunc from 0x0397a5 to 0x02f107 (0xa69d bytes) 000748 1768140556.630102 mapwrite 0x03a697 thru 0x03ffff (0x5969 bytes) 000749 1768140556.653417 trunc from 0x040000 to 0x00f01e (0x30fe1 bytes) 000752 1768140556.659782 read 0x000430 thru 0x00e557 (0xe128 bytes) 000753 1768140556.663599 mapwrite 0x031660 thru 0x03a096 (0x8a37 bytes) 000754 1768140556.709213 write 0x018a50 thru 0x025d8d (0xd33e bytes) 000755 1768140556.713161 MIRROR_RESYNC 3 mirrors 000756 1768140556.982342 read 0x021422 thru 0x02c567 (0xb146 bytes) 000757 1768140556.985154 read 0x02a90d thru 0x02b685 (0xd79 bytes) 000759 1768140556.985778 write 0x00479c thru 0x007680 (0x2ee5 bytes) 000761 1768140557.015005 trunc from 0x03a097 to 0x02f7aa (0xa8ec bytes) 000763 1768140557.032029 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f7ab): Mirror 7: 0xf6d79a03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f7ab): Mirror 7: 0xf6d79a03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f7ab: skipped 000764 1768140557.126833 read 0x003b0d thru 0x00d678 (0x9b6c bytes) 000765 1768140557.130400 trunc from 0x02f7ab to 0x008e7c (0x2692e bytes) 000766 1768140557.136206 read 0x004de5 thru 0x008e7c (0x4098 bytes) 000767 1768140557.136707 mapread 0x006998 thru 0x008e7c (0x24e5 bytes) 000770 1768140557.138209 mapread 0x0066a5 thru 0x008e7c (0x27d8 bytes) 000771 1768140557.139117 read 0x001a1c thru 0x003827 (0x1e0c bytes) 000772 1768140557.148642 mapwrite 0x0205c6 thru 0x02cb08 (0xc543 bytes) 000773 1768140557.178329 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000776 1768140557.220475 write 0x0378ed thru 0x03ffff (0x8713 bytes) 000778 1768140557.222180 mapwrite 0x0151f9 thru 0x01865a (0x3462 bytes) 000779 1768140557.250263 write 0x03b6a1 thru 0x03d79e (0x20fe bytes) 000780 1768140557.251040 read 0x02e143 thru 0x02ee6e (0xd2c bytes) 000781 1768140557.254872 trunc from 0x040000 to 0x03ded2 (0x212d bytes) 000782 1768140557.260512 trunc from 0x03ded3 to 0x03a0ee (0x3de4 bytes) 000785 1768140557.267650 mapwrite 0x006bac thru 0x008fb2 (0x2407 bytes) 000788 1768140557.293235 trunc from 0x03a0ef to 0x009f3b (0x301b3 bytes) 000794 1768140557.301188 read 0x000c5b thru 0x009f3b (0x92e1 bytes) 000795 1768140557.303390 mapread 0x009641 thru 0x009f3b (0x8fb bytes) 000796 1768140557.304016 read 0x003ded thru 0x009f3b (0x614f bytes) 000799 1768140557.306548 MIRROR_EXTEND to 2 mirrors 000800 1768140557.512428 mapwrite 0x02d9cf thru 0x031901 (0x3f33 bytes) 000801 1768140557.604487 trunc from 0x031902 to 0x03dc63 (0xc362 bytes) 000802 1768140557.612689 mapread 0x02d92c thru 0x036b53 (0x9228 bytes) 000803 1768140557.620913 mapread 0x021bcb thru 0x02edda (0xd210 bytes) 000806 1768140557.630214 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000807 1768140557.684743 trunc from 0x03dc64 to 0x02c351 (0x11912 bytes) 000810 1768140557.691618 mapread 0x00b461 thru 0x010b2a (0x56ca bytes) 000811 1768140557.697675 mapread 0x0212c9 thru 0x02c351 (0xb089 bytes) 000812 1768140557.702580 write 0x03d228 thru 0x03ffff (0x2dd8 bytes) 000813 1768140557.703259 write 0x017ba4 thru 0x01a18f (0x25ec bytes) 000814 1768140557.705909 read 0x01d635 thru 0x0236c2 (0x608e bytes) 000815 1768140557.706464 mapwrite 0x01e26d thru 0x02c34e (0xe0e2 bytes) 000817 1768140557.730016 trunc from 0x040000 to 0x007d71 (0x3828e bytes) 000818 1768140557.735319 mapwrite 0x013e1c thru 0x020ecf (0xd0b4 bytes) 000820 1768140557.764044 mapread 0x006a92 thru 0x00cfea (0x6559 bytes) 000821 1768140557.767698 read 0x01adb4 thru 0x020ecf (0x611c bytes) 000829 1768140557.771493 write 0x0242b0 thru 0x0257f6 (0x1547 bytes) 000832 1768140557.773461 write 0x02dd5a thru 0x03d7b0 (0xfa57 bytes) 000834 1768140557.774837 read 0x00ab8b thru 0x00c90a (0x1d80 bytes) 000836 1768140557.775463 read 0x020ee4 thru 0x0268a2 (0x59bf bytes) 000837 1768140557.779027 write 0x028fbe thru 0x02cc86 (0x3cc9 bytes) 000840 1768140557.780960 read 0x010c60 thru 0x01ae72 (0xa213 bytes) 000841 1768140557.784808 MIRROR_EXTEND to 2 mirrors 000845 1768140558.056756 write 0x033793 thru 0x0368da (0x3148 bytes) 000847 1768140558.111216 trunc from 0x03d7b1 to 0x02968b (0x14125 bytes) 000848 1768140558.133968 mapread 0x0190bb thru 0x027baa (0xeaf0 bytes) 000851 1768140558.141495 read 0x01f429 thru 0x02910b (0x9ce3 bytes) 000854 1768140558.143011 read 0x0180e6 thru 0x01b91d (0x3838 bytes) 000855 1768140558.143679 write 0x022d71 thru 0x023c32 (0xec2 bytes) 000860 1768140558.151036 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2968c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2968c): Mirror 2: 0x1012a91f (stale) CRC-32 checksum value for chunk [0, 0x2968c): Mirror 1: 0x647adb69 Mirror 3: 0x647adb69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2968c: skipped 000861 1768140558.450028 read 0x016d97 thru 0x01ba51 (0x4cbb bytes) 000862 1768140558.454891 MIRROR_RESYNC 3 mirrors 000863 1768140558.806794 mapread 0x018730 thru 0x01ccd4 (0x45a5 bytes) 000864 1768140558.821767 read 0x00f746 thru 0x0197cb (0xa086 bytes) 000867 1768140558.827455 write 0x031c18 thru 0x0388e2 (0x6ccb bytes) 000869 1768140558.872786 mapread 0x01d1bd thru 0x02c39f (0xf1e3 bytes) 000872 1768140558.881343 read 0x007a38 thru 0x00b0f3 (0x36bc bytes) 000875 1768140558.887597 mapwrite 0x02b694 thru 0x0349a5 (0x9312 bytes) 000877 1768140558.910590 trunc from 0x0388e3 to 0x016f50 (0x21992 bytes) 000878 1768140558.934919 read 0x009e83 thru 0x010e10 (0x6f8e bytes) 000879 1768140558.935553 mapwrite 0x02476a thru 0x02b7bf (0x7056 bytes) 000880 1768140558.985440 write 0x020fb1 thru 0x02cb2f (0xbb7f bytes) 000881 1768140558.986260 write 0x02ed16 thru 0x0339e8 (0x4cd3 bytes) 000883 1768140558.987179 trunc from 0x0339e9 to 0x027293 (0xc755 bytes) 000885 1768140558.996284 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27294): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27294): Mirror 2: 0x86a7b381 (stale) Mirror 3: 0x86a7b381 (stale) CRC-32 checksum value for chunk [0, 0x27294): Mirror 1: 0x4d5c8405 Mirror 4: 0x4d5c8405 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27294: skipped 000886 1768140559.324643 MIRROR_RESYNC 4 mirrors 000887 1768140559.584195 write 0x01f643 thru 0x01fe9c (0x85a bytes) 000888 1768140559.630086 mapwrite 0x01ecc7 thru 0x02a941 (0xbc7b bytes) 000889 1768140559.679903 trunc from 0x02a942 to 0x02d522 (0x2be1 bytes) 000892 1768140559.688154 write 0x02a791 thru 0x032c6e (0x84de bytes) 000895 1768140559.696216 MIRROR_RESYNC 4 mirrors 000896 1768140560.021082 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) 000897 1768140560.137249 MIRROR_EXTEND to 4 mirrors 000898 1768140560.417346 mapread 0x00c6ed thru 0x01b1a4 (0xeab8 bytes) 000899 1768140560.431291 mapwrite 0x024a93 thru 0x034777 (0xfce5 bytes) 000900 1768140560.490831 trunc from 0x034778 to 0x006037 (0x2e740 bytes) 000902 1768140560.498446 mapwrite 0x015857 thru 0x023b3e (0xe2e8 bytes) 000903 1768140560.532563 mapwrite 0x007f84 thru 0x017e05 (0xfe82 bytes) 000904 1768140560.556742 write 0x024668 thru 0x02c518 (0x7eb1 bytes) 000906 1768140560.558107 mapwrite 0x012e06 thru 0x015e8c (0x3087 bytes) 000909 1768140560.579976 mapread 0x014d6c thru 0x01cb60 (0x7df5 bytes) 000913 1768140560.582075 write 0x03495e thru 0x03ac38 (0x62db bytes) 000915 1768140560.583721 mapread 0x02e3f7 thru 0x02f6fa (0x1304 bytes) 000918 1768140560.588424 mapread 0x01a8ba thru 0x029390 (0xead7 bytes) 000922 1768140560.590620 mapwrite 0x02ca7f thru 0x02f0ac (0x262e bytes) 000923 1768140560.612891 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: 0x34915314 (stale) Mirror 4: 0x34915314 (stale) Mirror 5: 0x34915314 (stale) CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 1: 0xef968c96 Mirror 6: 0xef968c96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac39: skipped 000924 1768140560.890603 mapread 0x01dc2f thru 0x025e50 (0x8222 bytes) 000925 1768140560.902544 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: 0x34915314 (stale) Mirror 4: 0x34915314 (stale) Mirror 5: 0x34915314 (stale) CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 1: 0xef968c96 Mirror 6: 0xef968c96 Mirror 7: 0xef968c96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac39: skipped 000927 1768140561.272046 read 0x0244fd thru 0x02e03f (0x9b43 bytes) 000928 1768140561.282396 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: 0x34915314 (stale) Mirror 4: 0x34915314 (stale) Mirror 5: 0x34915314 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 2: 0x34915314 (stale) Mirror 4: 0x34915314 (stale) Mirror 5: 0x34915314 (stale) CRC-32 checksum value for chunk [0, 0x3ac39): Mirror 1: 0xef968c96 Mirror 6: 0xef968c96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac39: skipped 000929 1768140561.510405 mapwrite 0x031d20 thru 0x038f52 (0x7233 bytes) 000930 1768140561.603041 read 0x02125f thru 0x02d5e7 (0xc389 bytes) 000931 1768140561.607803 write 0x00576a thru 0x006111 (0x9a8 bytes) 000933 1768140561.614123 read 0x02612d thru 0x02ffa9 (0x9e7d bytes) 000935 1768140561.615025 mapwrite 0x00b75f thru 0x011c43 (0x64e5 bytes) 000936 1768140561.639429 trunc from 0x03ac39 to 0x00771f (0x33519 bytes) 000937 1768140561.671237 mapwrite 0x011d1f thru 0x012594 (0x876 bytes) 000938 1768140561.701620 write 0x00c2ea thru 0x014bbf (0x88d6 bytes) 000942 1768140561.707430 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14bc0): Mirror 2: 0xa313cd01 (stale) Mirror 4: 0xa313cd01 (stale) Mirror 6: 0x6e90b269 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14bc0): Mirror 2: 0xa313cd01 (stale) Mirror 4: 0xa313cd01 (stale) Mirror 6: 0x6e90b269 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14bc0: skipped 000945 1768140561.868284 mapread 0x013375 thru 0x014bbf (0x184b bytes) 000946 1768140561.877095 read 0x003082 thru 0x005f1c (0x2e9b bytes) 000948 1768140561.880338 read 0x0106c2 thru 0x014bbf (0x44fe bytes) 000950 1768140561.885201 mapwrite 0x0366d8 thru 0x03e0e0 (0x7a09 bytes) 000951 1768140561.922603 trunc from 0x03e0e1 to 0x02a563 (0x13b7d bytes) 000953 1768140561.935778 mapwrite 0x00a9f5 thru 0x017df2 (0xd3fe bytes) 000955 1768140561.966542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a564): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a564): Mirror 2: 0x543a6aea (stale) Mirror 4: 0x543a6aea (stale) Mirror 6: 0x9dc9cbb6 (stale) CRC-32 checksum value for chunk [0, 0x2a564): Mirror 1: 0x1e7d93d7 Mirror 7: 0x1e7d93d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a564: skipped 000956 1768140562.318631 mapwrite 0x0230a9 thru 0x023873 (0x7cb bytes) 000958 1768140562.404247 mapwrite 0x0378b5 thru 0x03ffff (0x874b bytes) 000959 1768140562.432498 mapwrite 0x035da1 thru 0x03ffff (0xa25f bytes) 000961 1768140562.446345 read 0x03f1b1 thru 0x03ffff (0xe4f bytes) 000962 1768140562.446716 trunc from 0x040000 to 0x016a91 (0x2956e bytes) 000963 1768140562.452532 trunc from 0x016a92 to 0x03039f (0x1990e bytes) 000966 1768140562.459495 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x303a0): Mirror 2: 0x60dbe094 (stale) Mirror 4: 0x60dbe094 (stale) Mirror 6: 0x1b3d79d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303a0): Mirror 2: 0x60dbe094 (stale) Mirror 4: 0x60dbe094 (stale) Mirror 6: 0x1b3d79d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303a0: skipped 000967 1768140562.594942 write 0x00d3b8 thru 0x017847 (0xa490 bytes) 000968 1768140562.600071 trunc from 0x0303a0 to 0x020faf (0xf3f0 bytes) 000970 1768140562.611680 read 0x01fdc3 thru 0x020faf (0x11ed bytes) 000971 1768140562.617339 trunc from 0x020fb0 to 0x0179b8 (0x95f7 bytes) 000974 1768140562.626646 trunc from 0x0179b9 to 0x02d5a1 (0x15be9 bytes) 000975 1768140562.633730 write 0x008341 thru 0x015523 (0xd1e3 bytes) 000978 1768140562.639146 mapwrite 0x03cd56 thru 0x03ffff (0x32aa bytes) 000979 1768140562.666619 write 0x0279ee thru 0x031c8e (0xa2a1 bytes) 000980 1768140562.671204 mapread 0x02d96d thru 0x03cd6a (0xf3fe bytes) 000983 1768140562.674627 mapread 0x01b3cb thru 0x02a78c (0xf3c2 bytes) 000984 1768140562.677729 trunc from 0x040000 to 0x03b5dc (0x4a23 bytes) 000986 1768140562.683223 read 0x010731 thru 0x01f72d (0xeffd bytes) 000988 1768140562.685762 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b5dd): Mirror 4: 0x45ed2ff3 (stale) Mirror 6: 0x3c194bde (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5dd): Mirror 4: 0x45ed2ff3 (stale) Mirror 6: 0x3c194bde (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5dd: skipped 000990 1768140562.799993 mapread 0x038dd6 thru 0x03b5dc (0x2807 bytes) 000991 1768140562.806361 MIRROR_RESYNC 3 mirrors 000992 1768140563.068109 mapread 0x038ec4 thru 0x03b5dc (0x2719 bytes) 000995 1768140563.086338 read 0x01c371 thru 0x026f64 (0xabf4 bytes) 000996 1768140563.092031 mapread 0x0017a4 thru 0x00d78b (0xbfe8 bytes) 000998 1768140563.097620 read 0x022523 thru 0x02725b (0x4d39 bytes) All operations completed A-OK! PASS 70a (24s) == sanity-flr test 71: check mirror extend parent fid ==== 09:09:24 (1768140564) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 09:09:26 (1768140566) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x754:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0924664 s, 22.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x759:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x754:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550904 s, 19.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x759:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x755:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x754:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x759:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x755:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x754:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 09:09:29 (1768140569) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4120832 .. Extending file size to 1169440 .. done Extending file size to 1765472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7597504 .. Extending file size to 8016160 .. Extending file size to 5263616 .. Extending file size to 9050816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5944192 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7357024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9210976 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8202336 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8292896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9204192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9284480 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57690 57691 57693 57695 57696 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:10:34 (1768140634) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8445088 .. Extending file size to 6897408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7597440 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8696896 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8746496 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9041760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9046432 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9128768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61135 61136 61138 61140 61142 done 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:11:40 (1768140700) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0434902 s, 48.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120872 s, 17.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5a7d9f25 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:11:47 (1768140707) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0339248 s, 61.8 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 09:11:49 (1768140709) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x579:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x76f:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:11:51 (1768140711) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0473709 s, 44.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x770:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x771:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:11:54 (1768140714) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0549906 s, 38.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x772:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x772:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x773:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:11:57 (1768140717) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0472658 s, 44.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x770:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x774:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x770:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x774:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x775:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:12:21 (1768140741) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229985 s, 45.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x771:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x777:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x776:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x771:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x777:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x776:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x772:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:13:03 (1768140783) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x773:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x774:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423831 s, 49.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x773:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x774:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x778:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x773:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x774:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x778:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:13:48 (1768140828) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. Creating new pool test_204f oleg353-server: Pool lustre.test_204f created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0386654 s, 54.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x775:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x779:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x775:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x779:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg353-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:14:41 (1768140881) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:14:43 (1768140883) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:14:46 (1768140886) oleg353-server: Pool lustre.test_206 created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg353-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:15:02 (1768140902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287903 s, 36.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120276 s, 87.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb00fff53 Mirror 3: 0xb00fff53 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:15:05 (1768140905) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:15:06 (1768140906) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:15:08 (1768140908) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0205189 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1fb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x940000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x940000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x940000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1fe0000 Read this (4 bytes): BBBB 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:15:23 (1768140923) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00294918 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:15:25 (1768140925) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140094 s, 74.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x59e lmm_fid: [0x200000404:0x59e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x59e lmm_fid: [0x200000404:0x59e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x784:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835358 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-11 09:15:25.000000000 -0500 Modify: 2026-01-11 09:15:25.000000000 -0500 Change: 2026-01-11 09:15:26.000000000 -0500 Birth: 2026-01-11 09:15:26.000000000 -0500 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:15:27 (1768140927) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:15:28 (1768140928) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0979545 s, 418 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108955 s, 376 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 11 09:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5a0:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 11 09:15 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:15:30 (1768140930) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:15:32 (1768140932) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 844 sec ============ 09:15:35 (1768140935) === sanity-flr: start cleanup 09:15:35 (1768140935) === === sanity-flr: finish cleanup 09:15:36 (1768140936) ===