-----============= acceptance-small: sanity-flr ============----- Sun Jan 11 09:01:45 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg411-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:01:51 (1768140111) === oleg411-client.virtnet: executing check_config_client /mnt/lustre oleg411-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg411-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d6c84dd9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d6c84dd9000.idle_timeout=debug disable quota as required oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:02:03 (1768140123) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8214] == sanity-flr test 0a: lfs mirror create with -N option == 09:02:03 (1768140123) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:02:09 (1768140129) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:02:10 (1768140130) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:02:11 (1768140131) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:02:17 (1768140137) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:02:18 (1768140138) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:02:19 (1768140139) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d6c84dd9000.stats=clear osc.lustre-OST0001-osc-ffff8d6c84dd9000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:02:22 (1768140142) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:02:25 (1768140145) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:02:27 (1768140147) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:02:28 (1768140148) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 09:02:31 (1768140151) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:02:33 (1768140153) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:02:35 (1768140155) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:02:37 (1768140157) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:02:40 (1768140160) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:02:42 (1768140162) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.140088 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212916 s, 49.2 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:02:44 (1768140164) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0605564 s, 104 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:02:47 (1768140167) Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:03:14 (1768140194) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0591835 s, 53.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:03:25 (1768140205) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:03:51 (1768140231) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.806987 s, 65.0 MB/s -rw-r--r-- 1 root root 52428800 Jan 11 09:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.249s user 0m0.000s sys 0m0.140s Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.233s user 0m0.001s sys 0m0.138s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.297s user 0m0.001s sys 0m0.138s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6c84dd9000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:04:18 (1768140258) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:04:19 (1768140259) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:04:20 (1768140260) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:04:21 (1768140261) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 09:04:23 (1768140263) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.141151 s, 89.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:04:26 (1768140266) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135513 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133098 s, 78.8 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:04:28 (1768140268) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:04:31 (1768140271) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000636015 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000454391 s, 2.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000476788 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000625975 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 09:04:34 (1768140274) create test files with size 15741245 .. bd3b3e81e817039d97200bd6da615e0e - 78749b21ea0a6af9883e6b2c69e82115 - d562e466bfc01555aaed366113a7e417 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 09:04:39 (1768140279) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:04:43 (1768140283) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:04:45 (1768140285) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302104 s, 34.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011604 s, 90.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 09:04:47 (1768140287) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0266774 s, 78.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0892554 s, 47.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0376881 s, 55.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 09:04:51 (1768140291) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:04:52 (1768140292) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:04:53 (1768140293) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914646 s, 115 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 09:04:55 (1768140295) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0854641 s, 36.8 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:04:58 (1768140298) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:05:00 (1768140300) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0932555 s, 112 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:05:21 (1768140321) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8d6c9fd51800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.083861 s, 125 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8d6ca0cb9800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:05:25 (1768140325) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (10s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:05:35 (1768140335) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:05:46 (1768140346) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:05:47 (1768140347) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0763488 s, 41.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0459127 s, 68.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf911062e (stale) Mirror 2: 0xf911062e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 248e9727285abe8b9d965fb369fab279 - mirror 1 checksum: 1d3e4ccd1f809dc1f7a753aab4c17bbd - mirror 2 checksum: 248e9727285abe8b9d965fb369fab279 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jan 11 09:05:48 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jan 11 09:05:48 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 248e9727285abe8b9d965fb369fab279 - mirror 1 checksum: 1d3e4ccd1f809dc1f7a753aab4c17bbd - mirror 2 checksum: 1d3e4ccd1f809dc1f7a753aab4c17bbd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jan 11 09:05:48 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jan 11 09:05:48 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:05:50 (1768140350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108312 s, 96.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:05:52 (1768140352) Failing mds1 on oleg411-server Stopping /mnt/lustre-mds1 (opts:) on oleg411-server 09:06:00 (1768140360) shut down facet: mds1 facet_host: oleg411-server facet_failover_host: oleg411-server Failover mds1 to oleg411-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 09:06:14 (1768140374) targets are mounted 09:06:14 (1768140374) facet_failover done oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914945 s, 115 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x59:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:06:21 (1768140381) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00791853 s, 2.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00448583 s, 4.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:06:24 (1768140384) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00804768 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00315614 s, 6.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:06:27 (1768140387) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00197936 s, 8.3 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:06:29 (1768140389) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00929368 s, 2.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:06:31 (1768140391) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00183578 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00934221 s, 438 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:06:33 (1768140393) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:06:38 (1768140398) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:06:56 (1768140416) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:07:14 (1768140434) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:07:31 (1768140451) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:07:33 (1768140453) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 55896 mirror split pid 55897 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55896 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55897 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 09:08:35 (1768140515) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 516 000002 1768140516.087490 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x1c872 000003 1768140516.183764 fallocate 0x0159e7 thru 0x01c871 (0x6e8b bytes) fallocating to largest ever: 0x2685f 000004 1768140516.192666 fallocate 0x0248e5 thru 0x02685e (0x1f7a bytes) 000005 1768140516.194308 write 0x03a5ea thru 0x03ffff (0x5a16 bytes) 000006 1768140516.196343 punch from 0x00ab2e to 0x00e966 (0x3e39 bytes) 000007 1768140516.197956 mapread 0x017271 thru 0x017a64 (0x7f4 bytes) 000008 1768140516.200173 mapwrite 0x01d620 thru 0x02008e (0x2a6f bytes) fallocating to largest ever: 0x40000 000009 1768140516.210916 fallocate 0x03e1d3 thru 0x03ffff (0x1e2d bytes) 000010 1768140516.212775 fallocate 0x026fd1 thru 0x034561 (0xd591 bytes) 000011 1768140516.215444 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000012 1768140516.247923 punch from 0x00fae1 to 0x01f306 (0xf826 bytes) 000013 1768140516.249783 mapwrite 0x022abd thru 0x022e5e (0x3a2 bytes) 000017 1768140516.256240 write 0x00e017 thru 0x017eac (0x9e96 bytes) 000022 1768140516.262530 fallocate 0x03f2bf thru 0x03ffff (0xd41 bytes) 000023 1768140516.264457 read 0x0306ed thru 0x03d51b (0xce2f bytes) 000024 1768140516.267576 trunc from 0x040000 to 0x00082b (0x3f7d4 bytes) 000025 1768140516.276326 mapwrite 0x002613 thru 0x004dbd (0x27ab bytes) 000026 1768140516.285920 read 0x003565 thru 0x004dbd (0x1859 bytes) 000028 1768140516.286642 write 0x03fe72 thru 0x03ffff (0x18e bytes) 000029 1768140516.287605 trunc from 0x03e64d to 0x03ffff (0x19b3 bytes) 000030 1768140516.290565 mapwrite 0x032c56 thru 0x0375a5 (0x4950 bytes) 000031 1768140516.299502 mapwrite 0x004a83 thru 0x012b8b (0xe109 bytes) 000032 1768140516.306514 trunc from 0x01681e to 0x018eea (0x26cd bytes) 000033 1768140516.308205 fallocate 0x010837 thru 0x014036 (0x3800 bytes) 000034 1768140516.309840 mapwrite 0x00bb65 thru 0x00f3d5 (0x3871 bytes) 000035 1768140516.314728 trunc from 0x020f2e to 0x02a783 (0x9856 bytes) 000036 1768140516.316198 fallocate 0x01130a thru 0x018cce (0x79c5 bytes) 000037 1768140516.317957 mapread 0x024412 thru 0x02926e (0x4e5d bytes) 000038 1768140516.325053 trunc from 0x040000 to 0x036fb1 (0x904e bytes) 000039 1768140516.329312 mapread 0x012a0a thru 0x01fe14 (0xd40b bytes) 000040 1768140516.333310 read 0x005357 thru 0x008aa7 (0x3751 bytes) 000041 1768140516.333826 trunc from 0x02a2dc to 0x02ab9b (0x8c0 bytes) 000042 1768140516.336575 MIRROR_EXTEND to 2 mirrors 000043 1768140516.493999 punch from 0x0175ff to 0x01abcf (0x35d1 bytes) 000044 1768140516.503948 punch from 0x00edfa to 0x01de3b (0xf042 bytes) 000046 1768140516.505795 punch from 0x01168d to 0x01748c (0x5e00 bytes) 000047 1768140516.507060 mapwrite 0x0328eb thru 0x0333df (0xaf5 bytes) 000048 1768140516.513635 trunc from 0x01adeb to 0x0208de (0x5af4 bytes) 000049 1768140516.515382 read 0x013256 thru 0x022710 (0xf4bb bytes) 000050 1768140516.518004 read 0x02844b thru 0x02883a (0x3f0 bytes) 000051 1768140516.520828 punch from 0x02bca7 to 0x036fb1 (0xb30b bytes) 000052 1768140516.522398 mapwrite 0x028b9a thru 0x034227 (0xb68e bytes) 000053 1768140516.538604 trunc from 0x0223f9 to 0x0308a5 (0xe4ad bytes) 000054 1768140516.541126 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36fb2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fb2): Mirror 2: 0x20d18635 (stale) CRC-32 checksum value for chunk [0, 0x36fb2): Mirror 1: 0xaa98a9a3 Mirror 3: 0xaa98a9a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fb2: skipped 000055 1768140516.727702 punch from 0x022e0a to 0x02feaf (0xd0a6 bytes) 000056 1768140516.736944 write 0x018f39 thru 0x02775d (0xe825 bytes) 000057 1768140516.741180 mapwrite 0x006d23 thru 0x014e84 (0xe162 bytes) 000058 1768140516.753224 write 0x01820f thru 0x023405 (0xb1f7 bytes) 000060 1768140516.754256 mapread 0x01a9e5 thru 0x026318 (0xb934 bytes) 000061 1768140516.754796 trunc from 0x036fb2 to 0x03adf0 (0x3e3f bytes) 000062 1768140516.764085 trunc from 0x036679 to 0x03adf0 (0x4778 bytes) 000063 1768140516.765757 punch from 0x0396b0 to 0x03adf0 (0x1741 bytes) 000064 1768140516.767247 write 0x01692b thru 0x026524 (0xfbfa bytes) 000065 1768140516.769912 MIRROR_RESYNC 3 mirrors 000066 1768140516.981146 trunc from 0x004bba to 0x009cd2 (0x5119 bytes) 000067 1768140516.988869 write 0x00493c thru 0x007646 (0x2d0b bytes) 000068 1768140516.991389 fallocate 0x030f0b thru 0x032b85 (0x1c7b bytes) 000069 1768140516.992712 trunc from 0x00cca2 to 0x011c78 (0x4fd7 bytes) 000070 1768140516.993896 mapread 0x03588a thru 0x03adf0 (0x5567 bytes) 000071 1768140516.996112 punch from 0x025b70 to 0x033082 (0xd513 bytes) 000072 1768140516.997476 mapread 0x029ed1 thru 0x037bd1 (0xdd01 bytes) 000076 1768140517.001597 mapread 0x02533b thru 0x0301f4 (0xaeba bytes) 000077 1768140517.003289 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3adf1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adf1): Mirror 2: 0x764b777d (stale) Mirror 3: 0x764b777d (stale) CRC-32 checksum value for chunk [0, 0x3adf1): Mirror 1: 0xe22d9175 Mirror 4: 0xe22d9175 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adf1: skipped 000078 1768140517.218993 read 0x03095a thru 0x0363af (0x5a56 bytes) 000079 1768140517.222556 trunc from 0x03adf1 to 0x024883 (0x1656d bytes) 000080 1768140517.233114 mapread 0x00f207 thru 0x01c1a4 (0xcf9e bytes) 000082 1768140517.239597 trunc from 0x024884 to 0x016f48 (0xd93b bytes) 000083 1768140517.244200 mapread 0x011b70 thru 0x0160be (0x454f bytes) 000085 1768140517.245148 trunc from 0x016f49 to 0x017518 (0x5d0 bytes) 000086 1768140517.248563 mapread 0x015d07 thru 0x017518 (0x1812 bytes) 000087 1768140517.250217 write 0x02a65a thru 0x036978 (0xc31f bytes) 000088 1768140517.252062 MIRROR_RESYNC 4 mirrors 000089 1768140517.478576 punch from 0x00f110 to 0x013a7a (0x496b bytes) 000090 1768140517.486610 trunc from 0x01081e to 0x016a54 (0x6237 bytes) 000091 1768140517.487908 trunc from 0x036979 to 0x000ff9 (0x3597f bytes) 000092 1768140517.495745 fallocate 0x031ea7 thru 0x03d4f5 (0xb64f bytes) 000093 1768140517.497322 read 0x022086 thru 0x02405a (0x1fd5 bytes) 000094 1768140517.498935 trunc from 0x03d4f6 to 0x0169aa (0x26b4b bytes) 000095 1768140517.503394 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x169ab): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169ab): Mirror 2: 0xd2062cc5 (stale) Mirror 3: 0xd2062cc5 (stale) Mirror 4: 0xd2062cc5 (stale) CRC-32 checksum value for chunk [0, 0x169ab): Mirror 1: 0x60946e03 Mirror 5: 0x60946e03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169ab: skipped 000096 1768140517.678463 mapread 0x004f11 thru 0x00b3ed (0x64dd bytes) 000097 1768140517.685485 trunc from 0x015450 to 0x0169aa (0x155b bytes) 000098 1768140517.693788 read 0x002b6a thru 0x00f4ab (0xc942 bytes) 000099 1768140517.696030 write 0x01d8ef thru 0x01ff49 (0x265b bytes) 000100 1768140517.697455 trunc from 0x00d15c to 0x015fa8 (0x8e4d bytes) 000101 1768140517.699014 write 0x011a99 thru 0x017c43 (0x61ab bytes) 000102 1768140517.701668 mapread 0x0052ae thru 0x0124f8 (0xd24b bytes) 000103 1768140517.702220 mapread 0x018a73 thru 0x01ff49 (0x74d7 bytes) 000104 1768140517.704890 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ff4a): Mirror 2: 0x44ca2c4f (stale) Mirror 4: 0x44ca2c4f (stale) Mirror 5: 0xc26f1843 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff4a): Mirror 2: 0x44ca2c4f (stale) Mirror 4: 0x44ca2c4f (stale) Mirror 5: 0xc26f1843 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff4a: skipped 000105 1768140517.780654 fallocate 0x000592 thru 0x00c043 (0xbab2 bytes) 000108 1768140517.782952 trunc from 0x011828 to 0x01ed8c (0xd565 bytes) 000109 1768140517.784459 fallocate 0x019895 thru 0x01f3f6 (0x5b62 bytes) 000110 1768140517.785881 punch from 0x004620 to 0x006ed9 (0x28ba bytes) 000111 1768140517.787135 punch from 0x000879 to 0x00d386 (0xcb0e bytes) 000112 1768140517.788377 mapwrite 0x03b52c thru 0x03ffff (0x4ad4 bytes) 000113 1768140517.805398 mapwrite 0x035c3a thru 0x03d833 (0x7bfa bytes) 000114 1768140517.812379 trunc from 0x037acf to 0x03ffff (0x8531 bytes) 000115 1768140517.814064 trunc from 0x005856 to 0x00b294 (0x5a3f bytes) 000116 1768140517.815560 fallocate 0x00468d thru 0x00f272 (0xabe6 bytes) 000117 1768140517.817234 write 0x02cd28 thru 0x03aabc (0xdd95 bytes) 000118 1768140517.821356 trunc from 0x040000 to 0x01b4ef (0x24b10 bytes) 000119 1768140517.826353 MIRROR_RESYNC 4 mirrors 000120 1768140517.994693 read 0x004a98 thru 0x005505 (0xa6e bytes) 000121 1768140517.996129 read 0x011f6a thru 0x01b4ef (0x9586 bytes) 000122 1768140517.998490 trunc from 0x01b4f0 to 0x01097f (0xab70 bytes) 000123 1768140518.010885 mapwrite 0x021fdc thru 0x0308ff (0xe924 bytes) 000124 1768140518.023684 MIRROR_RESYNC 4 mirrors 000125 1768140518.167564 fallocate 0x0009af thru 0x00a3e8 (0x9a3a bytes) 000126 1768140518.175665 mapwrite 0x0308bb thru 0x0398c5 (0x900b bytes) 000127 1768140518.191811 trunc from 0x02776e to 0x02f014 (0x78a7 bytes) 000128 1768140518.195418 read 0x02993a thru 0x032747 (0x8e0e bytes) 000129 1768140518.197507 trunc from 0x02bd88 to 0x0314e4 (0x575d bytes) 000130 1768140518.198902 read 0x0177d2 thru 0x022ceb (0xb51a bytes) 000131 1768140518.202587 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x398c6): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398c6): Mirror 2: 0x1713d4f5 (stale) Mirror 4: 0x1713d4f5 (stale) Mirror 5: 0x1713d4f5 (stale) CRC-32 checksum value for chunk [0, 0x398c6): Mirror 1: 0x7774b126 Mirror 6: 0x7774b126 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398c6: skipped 000132 1768140518.383683 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398c6): Mirror 4: 0x1713d4f5 (stale) Mirror 5: 0x1713d4f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398c6): Mirror 4: 0x1713d4f5 (stale) Mirror 5: 0x1713d4f5 (stale) CRC-32 checksum value for chunk [0, 0x398c6): Mirror 1: 0x7774b126 Mirror 6: 0x7774b126 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398c6: skipped 000133 1768140518.534722 read 0x003e6d thru 0x00f715 (0xb8a9 bytes) 000134 1768140518.539928 read 0x031b6c thru 0x032163 (0x5f8 bytes) 000135 1768140518.545039 trunc from 0x0000e8 to 0x00b85f (0xb778 bytes) 000137 1768140518.554129 write 0x012de6 thru 0x01f1db (0xc3f6 bytes) 000138 1768140518.557577 fallocate 0x016537 thru 0x0191fe (0x2cc8 bytes) 000142 1768140518.560660 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398c6): Mirror 5: 0x1713d4f5 (stale) Mirror 6: 0x7774b126 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398c6): Mirror 5: 0x1713d4f5 (stale) Mirror 6: 0x7774b126 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398c6: skipped 000143 1768140518.632466 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x398c6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398c6): Mirror 5: 0x1713d4f5 (stale) Mirror 6: 0x7774b126 (stale) CRC-32 checksum value for chunk [0, 0x398c6): Mirror 1: 0xed295cd5 Mirror 7: 0xed295cd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398c6: skipped 000144 1768140518.817388 mapwrite 0x03c6e6 thru 0x03ffff (0x391a bytes) 000145 1768140518.838623 trunc from 0x040000 to 0x0088f6 (0x37709 bytes) 000146 1768140518.842971 fallocate 0x021ca2 thru 0x02bc51 (0x9fb0 bytes) 000147 1768140518.844393 trunc from 0x0088f7 to 0x025c40 (0x1d34a bytes) 000149 1768140518.848049 mapread 0x001cfa thru 0x006122 (0x4429 bytes) 000150 1768140518.850717 trunc from 0x000dcf to 0x00bb5b (0xad8d bytes) 000152 1768140518.852644 fallocate 0x035edd thru 0x03ffff (0xa123 bytes) 000153 1768140518.854940 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25c41): Mirror 5: 0x612d8c42 (stale) Mirror 7: 0xb9f2a48f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c41): Mirror 5: 0x612d8c42 (stale) Mirror 7: 0xb9f2a48f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c41: skipped 000154 1768140518.921965 punch from 0x013e07 to 0x015f6b (0x2165 bytes) 000155 1768140518.923452 punch from 0x01f488 to 0x02463f (0x51b8 bytes) 000156 1768140518.924834 mapwrite 0x00e28d thru 0x01310b (0x4e7f bytes) 000157 1768140518.934611 MIRROR_RESYNC 3 mirrors 000158 1768140519.090399 trunc from 0x000a9f to 0x00b4b0 (0xaa12 bytes) 000159 1768140519.097343 trunc from 0x023aa9 to 0x025c40 (0x2198 bytes) 000160 1768140519.098634 fallocate 0x031b70 thru 0x03ffff (0xe490 bytes) 000161 1768140519.100031 write 0x02d19b thru 0x0360c1 (0x8f27 bytes) 000162 1768140519.100909 read 0x01757e thru 0x01bb85 (0x4608 bytes) 000163 1768140519.103193 trunc from 0x014049 to 0x0148b2 (0x86a bytes) 000164 1768140519.104521 mapwrite 0x0225a2 thru 0x023bad (0x160c bytes) 000165 1768140519.111444 mapwrite 0x0306e8 thru 0x03ed80 (0xe699 bytes) 000166 1768140519.124534 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed81): Mirror 7: 0x93f2f44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed81): Mirror 7: 0x93f2f44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed81: skipped 000168 1768140519.186071 mapread 0x02aec3 thru 0x033d5a (0x8e98 bytes) 000169 1768140519.191349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000170 1768140519.217993 mapread 0x006f70 thru 0x013b6a (0xcbfb bytes) 000171 1768140519.223682 mapread 0x00197a thru 0x005dec (0x4473 bytes) 000172 1768140519.224162 trunc from 0x00c6a5 to 0x0188c7 (0xc223 bytes) 000173 1768140519.226583 MIRROR_EXTEND to 2 mirrors 000174 1768140519.359532 read 0x03c70d thru 0x03ed80 (0x2674 bytes) 000175 1768140519.362584 trunc from 0x02930a to 0x02eb95 (0x588c bytes) 000176 1768140519.371896 fallocate 0x002a40 thru 0x00c1ea (0x97ab bytes) 000178 1768140519.373653 write 0x03fbe5 thru 0x03ffff (0x41b bytes) 000180 1768140519.375970 mapread 0x027c1f thru 0x029136 (0x1518 bytes) 000181 1768140519.380516 mapread 0x02f4ff thru 0x02f709 (0x20b bytes) 000182 1768140519.380963 read 0x0003a6 thru 0x00c614 (0xc26f bytes) 000183 1768140519.385113 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x361566e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x275ed771 Mirror 3: 0x275ed771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000184 1768140519.566898 trunc from 0x040000 to 0x026cd3 (0x1932c bytes) 000185 1768140519.580726 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26cd4): Mirror 3: 0x4d64b344 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26cd4): Mirror 3: 0x4d64b344 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26cd4: skipped 000186 1768140519.641097 trunc from 0x026cd4 to 0x01876d (0xe566 bytes) 000187 1768140519.645436 read 0x002282 thru 0x00b1e7 (0x8f66 bytes) 000188 1768140519.647589 punch from 0x007c2b to 0x0090d8 (0x14ae bytes) 000189 1768140519.648843 punch from 0x014e6c to 0x01876d (0x3902 bytes) 000190 1768140519.651322 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1876e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1876e): Mirror 3: 0x4f588df1 (stale) CRC-32 checksum value for chunk [0, 0x1876e): Mirror 1: 0x4f588df1 Mirror 4: 0x4f588df1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1876e: skipped 000191 1768140519.820186 punch from 0x00de20 to 0x0160b0 (0x8291 bytes) 000193 1768140519.830572 mapwrite 0x02fad6 thru 0x038043 (0x856e bytes) 000194 1768140519.850794 read 0x0334bb thru 0x0360ed (0x2c33 bytes) 000195 1768140519.851312 write 0x03599c thru 0x03ffff (0xa664 bytes) 000197 1768140519.852479 trunc from 0x040000 to 0x010513 (0x2faec bytes) 000198 1768140519.856407 trunc from 0x0100e7 to 0x010513 (0x42d bytes) 000199 1768140519.857640 fallocate 0x007318 thru 0x00c75f (0x5448 bytes) 000200 1768140519.858941 mapread 0x00d0a2 thru 0x010513 (0x3472 bytes) 000201 1768140519.861381 trunc from 0x008a01 to 0x00c370 (0x3970 bytes) 000204 1768140519.863138 write 0x0234f1 thru 0x02ecc3 (0xb7d3 bytes) 000205 1768140519.864026 write 0x015277 thru 0x0247f5 (0xf57f bytes) 000206 1768140519.866917 trunc from 0x003d18 to 0x003d31 (0x1a bytes) 000207 1768140519.868305 fallocate 0x032513 thru 0x03960c (0x70fa bytes) 000209 1768140519.870289 read 0x00d13b thru 0x014281 (0x7147 bytes) 000210 1768140519.870640 fallocate 0x017c71 thru 0x017eb1 (0x241 bytes) 000211 1768140519.873499 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3960d): Mirror 4: 0xda9affdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3960d): Mirror 4: 0xda9affdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3960d: skipped 000212 1768140519.936579 mapread 0x038630 thru 0x038b88 (0x559 bytes) 000213 1768140519.938160 trunc from 0x0294ba to 0x02d85a (0x43a1 bytes) 000214 1768140519.939548 write 0x01e85a thru 0x01f22c (0x9d3 bytes) 000215 1768140519.943689 trunc from 0x02e29c to 0x035ddc (0x7b41 bytes) 000216 1768140519.945165 trunc from 0x028342 to 0x0325b4 (0xa273 bytes) 000217 1768140519.947989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000220 1768140519.981590 fallocate 0x01df55 thru 0x023ce9 (0x5d95 bytes) 000223 1768140519.984528 trunc from 0x03960d to 0x03f8cd (0x62c1 bytes) 000224 1768140519.988807 mapread 0x0347f8 thru 0x036ab2 (0x22bb bytes) 000225 1768140519.993754 mapwrite 0x01946c thru 0x022274 (0x8e09 bytes) 000226 1768140520.004525 trunc from 0x00d4e6 to 0x018d9c (0xb8b7 bytes) 000227 1768140520.006025 write 0x02838a thru 0x036c20 (0xe897 bytes) 000228 1768140520.007100 trunc from 0x03f8ce to 0x005238 (0x3a695 bytes) 000229 1768140520.010886 fallocate 0x001011 thru 0x007222 (0x6212 bytes) 000230 1768140520.013416 MIRROR_EXTEND to 2 mirrors 000232 1768140520.156620 trunc from 0x0023bc to 0x005238 (0x2e7d bytes) 000233 1768140520.164078 trunc from 0x005239 to 0x00097a (0x48be bytes) 000235 1768140520.169291 trunc from 0x00097b to 0x00b1a3 (0xa829 bytes) 000236 1768140520.172816 mapwrite 0x035df8 thru 0x03726c (0x1475 bytes) 000237 1768140520.184917 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3726d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3726d): Mirror 2: 0xad3d713d (stale) CRC-32 checksum value for chunk [0, 0x3726d): Mirror 1: 0xd8a4f66 Mirror 3: 0xd8a4f66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3726d: skipped 000238 1768140520.358089 read 0x034f45 thru 0x03726c (0x2328 bytes) 000239 1768140520.361028 read 0x015f4a thru 0x01af42 (0x4ff9 bytes) 000240 1768140520.366042 MIRROR_RESYNC 3 mirrors 000241 1768140520.494354 trunc from 0x00acbb to 0x012345 (0x768b bytes) 000242 1768140520.503006 MIRROR_RESYNC 3 mirrors 000243 1768140520.639730 punch from 0x033cf4 to 0x03726c (0x3579 bytes) 000244 1768140520.647181 mapread 0x015afa thru 0x018007 (0x250e bytes) 000245 1768140520.651863 trunc from 0x008ab7 to 0x00b4e9 (0x2a33 bytes) 000246 1768140520.653453 trunc from 0x03726d to 0x00ddc7 (0x294a5 bytes) 000247 1768140520.662584 punch from 0x004c93 to 0x0088d0 (0x3c3e bytes) 000248 1768140520.663790 trunc from 0x003df8 to 0x00ddc7 (0x9fd0 bytes) 000249 1768140520.666308 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xddc8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xddc8): Mirror 2: 0x749f4592 (stale) Mirror 3: 0x749f4592 (stale) CRC-32 checksum value for chunk [0, 0xddc8): Mirror 1: 0x749f4592 Mirror 4: 0x749f4592 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xddc8: skipped 000250 1768140520.847506 write 0x00b6d2 thru 0x01073d (0x506c bytes) 000251 1768140520.857292 punch from 0x001644 to 0x00b282 (0x9c3f bytes) 000252 1768140520.860482 trunc from 0x01073e to 0x015a1d (0x52e0 bytes) 000253 1768140520.868427 mapread 0x00ae87 thru 0x00ba04 (0xb7e bytes) 000254 1768140520.872076 mapread 0x00571a thru 0x00d69c (0x7f83 bytes) 000255 1768140520.872942 mapread 0x007cc0 thru 0x011c50 (0x9f91 bytes) 000257 1768140520.873895 trunc from 0x015a1e to 0x004611 (0x1140c bytes) 000258 1768140520.877959 write 0x01eacb thru 0x027bc3 (0x90f9 bytes) 000259 1768140520.878799 write 0x00eff8 thru 0x01c9ba (0xd9c3 bytes) 000260 1768140520.882472 mapwrite 0x0154a0 thru 0x01667c (0x11dd bytes) 000261 1768140520.888224 read 0x00fee5 thru 0x011141 (0x125d bytes) 000262 1768140520.888633 trunc from 0x027bc4 to 0x004f5a (0x22c69 bytes) 000263 1768140520.894534 MIRROR_RESYNC 4 mirrors 000264 1768140521.043948 fallocate 0x009fbd thru 0x014fad (0xaff1 bytes) 000265 1768140521.052490 fallocate 0x017fc4 thru 0x024ca0 (0xccdd bytes) 000266 1768140521.053863 trunc from 0x00f1ce to 0x014fad (0x5de0 bytes) 000267 1768140521.055103 mapwrite 0x03858d thru 0x038cab (0x71f bytes) 000268 1768140521.068135 trunc from 0x035c3d to 0x038cab (0x306f bytes) 000269 1768140521.069502 read 0x00aaf6 thru 0x00fb1d (0x5028 bytes) 000270 1768140521.071552 trunc from 0x038cac to 0x022ef1 (0x15dba bytes) 000271 1768140521.075499 trunc from 0x022ef2 to 0x00484a (0x1e6a7 bytes) 000272 1768140521.079247 fallocate 0x036f6b thru 0x03ffff (0x9095 bytes) 000273 1768140521.080960 read 0x037bc2 thru 0x039088 (0x14c7 bytes) 000274 1768140521.083968 trunc from 0x031baa to 0x03f259 (0xd6b0 bytes) 000275 1768140521.085264 mapread 0x0192de thru 0x0220f7 (0x8e1a bytes) 000276 1768140521.085839 write 0x03d943 thru 0x03ffff (0x26bd bytes) 000277 1768140521.087558 mapread 0x037ce6 thru 0x03b791 (0x3aac bytes) 000278 1768140521.093075 trunc from 0x040000 to 0x01fc26 (0x203d9 bytes) 000279 1768140521.097267 write 0x013c5f thru 0x014d2a (0x10cc bytes) 000280 1768140521.098033 trunc from 0x010575 to 0x017a96 (0x7522 bytes) 000281 1768140521.100790 punch from 0x0141a4 to 0x017e8d (0x3cea bytes) 000282 1768140521.102289 trunc from 0x01536b to 0x01fc26 (0xa8bc bytes) 000284 1768140521.103959 punch from 0x00859f to 0x013c3b (0xb69d bytes) 000285 1768140521.105342 mapwrite 0x03cabe thru 0x03f28d (0x27d0 bytes) 000286 1768140521.117034 punch from 0x02e255 to 0x030ae0 (0x288c bytes) 000287 1768140521.118319 read 0x00bc13 thru 0x00cf27 (0x1315 bytes) 000288 1768140521.123033 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f28e): Mirror 2: 0x64609099 (stale) Mirror 3: 0x64609099 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f28e): Mirror 2: 0x64609099 (stale) Mirror 3: 0x64609099 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f28e: skipped 000289 1768140521.192489 trunc from 0x03f28e to 0x01b45c (0x23e31 bytes) 000290 1768140521.197346 mapread 0x00367d thru 0x00ae7a (0x77fe bytes) 000291 1768140521.202476 trunc from 0x0009fa to 0x004d63 (0x436a bytes) 000292 1768140521.203901 trunc from 0x004706 to 0x007ad0 (0x33cb bytes) 000293 1768140521.206509 MIRROR_RESYNC 3 mirrors 000294 1768140521.339063 MIRROR_EXTEND to 4 mirrors 000295 1768140521.472416 MIRROR_RESYNC 4 mirrors 000296 1768140521.569374 trunc from 0x01b45d to 0x003bc0 (0x1789c bytes) 000297 1768140521.582909 mapread 0x002eea thru 0x003bc0 (0xcd7 bytes) 000298 1768140521.585575 read 0x00174e thru 0x003bc0 (0x2473 bytes) 000299 1768140521.587018 mapread 0x0002ec thru 0x001567 (0x127c bytes) 000300 1768140521.588460 read 0x0034e6 thru 0x003bc0 (0x6db bytes) 000302 1768140521.589051 read 0x001e70 thru 0x003bc0 (0x1d51 bytes) 000304 1768140521.589618 read 0x00249a thru 0x003bc0 (0x1727 bytes) 000305 1768140521.589937 mapwrite 0x039570 thru 0x03ffff (0x6a90 bytes) 000306 1768140521.601254 fallocate 0x0050ef thru 0x013163 (0xe075 bytes) 000307 1768140521.602702 trunc from 0x040000 to 0x024855 (0x1b7aa bytes) 000309 1768140521.606580 write 0x036104 thru 0x03ffff (0x9efc bytes) 000310 1768140521.607379 fallocate 0x01dd39 thru 0x01e4f3 (0x7bb bytes) 000311 1768140521.609935 MIRROR_RESYNC 4 mirrors 000312 1768140521.768774 trunc from 0x0384a1 to 0x03de62 (0x59c2 bytes) 000313 1768140521.776901 trunc from 0x040000 to 0x03167d (0xe982 bytes) 000315 1768140521.786684 write 0x0327a1 thru 0x0350b0 (0x2910 bytes) 000316 1768140521.787730 mapwrite 0x025839 thru 0x026f92 (0x175a bytes) 000317 1768140521.794673 punch from 0x0158b0 to 0x021478 (0xbbc9 bytes) 000318 1768140521.796209 trunc from 0x0350b1 to 0x013176 (0x21f3a bytes) 000319 1768140521.800221 read 0x007830 thru 0x00f011 (0x77e2 bytes) 000320 1768140521.802502 mapread 0x00991d thru 0x013176 (0x985a bytes) 000321 1768140521.804217 trunc from 0x013177 to 0x036026 (0x22eb0 bytes) 000322 1768140521.807720 fallocate 0x007c64 thru 0x010589 (0x8926 bytes) 000323 1768140521.808993 punch from 0x02d08c to 0x031871 (0x47e6 bytes) 000325 1768140521.810417 mapread 0x012c8a thru 0x0146ee (0x1a65 bytes) 000327 1768140521.812868 trunc from 0x00822c to 0x011e3e (0x9c13 bytes) 000328 1768140521.814064 fallocate 0x005b1c thru 0x007b33 (0x2018 bytes) 000329 1768140521.815434 mapwrite 0x022fef thru 0x032168 (0xf17a bytes) 000330 1768140521.825141 trunc from 0x036027 to 0x01a5db (0x1ba4b bytes) 000331 1768140521.828764 mapwrite 0x005e19 thru 0x010ff2 (0xb1da bytes) 000332 1768140521.837538 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a5dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a5dc): Mirror 2: 0x805649ee (stale) Mirror 3: 0x805649ee (stale) Mirror 4: 0x805649ee (stale) CRC-32 checksum value for chunk [0, 0x1a5dc): Mirror 1: 0x9bacfa7a Mirror 5: 0x9bacfa7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a5dc: skipped 000333 1768140522.024490 mapwrite 0x023fae thru 0x02a6c4 (0x6717 bytes) 000334 1768140522.050042 write 0x00f2f2 thru 0x016603 (0x7312 bytes) 000335 1768140522.052615 mapwrite 0x007e77 thru 0x012cb7 (0xae41 bytes) 000336 1768140522.060639 trunc from 0x0270fc to 0x02a6c4 (0x35c9 bytes) 000337 1768140522.061998 trunc from 0x00654b to 0x006fe7 (0xa9d bytes) 000338 1768140522.063344 trunc from 0x021f75 to 0x026155 (0x41e1 bytes) 000339 1768140522.064862 write 0x01354f thru 0x01eb56 (0xb608 bytes) 000340 1768140522.065891 punch from 0x01cc6f to 0x02a6c4 (0xda56 bytes) 000341 1768140522.070385 MIRROR_RESYNC 5 mirrors 000342 1768140522.276543 fallocate 0x03ab0b thru 0x03ffff (0x54f5 bytes) 000343 1768140522.284886 fallocate 0x00574a thru 0x00c6a0 (0x6f57 bytes) 000344 1768140522.287494 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d89a0d2 (stale) Mirror 4: 0x1d89a0d2 (stale) Mirror 5: 0x1d89a0d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d89a0d2 (stale) Mirror 4: 0x1d89a0d2 (stale) Mirror 5: 0x1d89a0d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000345 1768140522.369261 trunc from 0x040000 to 0x018b65 (0x2749a bytes) 000346 1768140522.377947 mapwrite 0x01edf4 thru 0x01ffc2 (0x11cf bytes) 000347 1768140522.387492 mapread 0x01fd46 thru 0x01ffc2 (0x27d bytes) 000348 1768140522.389135 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ffc3): Mirror 3: 0xda100663 (stale) Mirror 4: 0xda100663 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ffc3): Mirror 3: 0xda100663 (stale) Mirror 4: 0xda100663 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ffc3: skipped 000349 1768140522.460290 fallocate 0x00266e thru 0x00287c (0x20f bytes) 000350 1768140522.462099 mapread 0x014bb4 thru 0x01a877 (0x5cc4 bytes) 000351 1768140522.467060 write 0x012bf3 thru 0x019012 (0x6420 bytes) 000352 1768140522.469512 mapwrite 0x0370a6 thru 0x038580 (0x14db bytes) 000354 1768140522.481023 write 0x035e64 thru 0x03d082 (0x721f bytes) 000355 1768140522.482280 trunc from 0x01941f to 0x023553 (0xa135 bytes) 000356 1768140522.484344 read 0x0044b5 thru 0x0122a4 (0xddf0 bytes) 000357 1768140522.487417 punch from 0x00700d to 0x008563 (0x1557 bytes) 000358 1768140522.490630 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d083): Mirror 3: 0x41cdf4a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d083): Mirror 3: 0x41cdf4a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d083: skipped 000360 1768140522.559598 trunc from 0x03d083 to 0x010846 (0x2c83c bytes) 000361 1768140522.564329 fallocate 0x008573 thru 0x00c1c5 (0x3c53 bytes) 000362 1768140522.566675 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10847): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10847): Mirror 3: 0x895aecc9 (stale) CRC-32 checksum value for chunk [0, 0x10847): Mirror 1: 0x7336095a Mirror 4: 0x7336095a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10847: skipped 000363 1768140522.730551 trunc from 0x010847 to 0x028909 (0x180c3 bytes) 000364 1768140522.746546 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2890a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2890a): Mirror 3: 0x4fcbfe44 (stale) Mirror 4: 0xb444810f (stale) CRC-32 checksum value for chunk [0, 0x2890a): Mirror 1: 0xb444810f Mirror 5: 0xb444810f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2890a: skipped 000365 1768140522.925311 trunc from 0x02890a to 0x00e9d2 (0x19f37 bytes) 000366 1768140522.939553 mapwrite 0x037959 thru 0x03e8f6 (0x6f9e bytes) 000367 1768140522.952708 mapread 0x02879b thru 0x034593 (0xbdf9 bytes) 000368 1768140522.956474 trunc from 0x03e8f7 to 0x032757 (0xc19f bytes) 000369 1768140522.961466 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32758): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32758): Mirror 3: 0xe607c908 (stale) Mirror 4: 0x6c0784ce (stale) Mirror 5: 0x6c0784ce (stale) CRC-32 checksum value for chunk [0, 0x32758): Mirror 1: 0x528af209 Mirror 6: 0x528af209 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32758: skipped 000370 1768140523.162444 write 0x02ec1b thru 0x02ffd2 (0x13b8 bytes) 000371 1768140523.173624 read 0x0091fd thru 0x00dacd (0x48d1 bytes) 000372 1768140523.175756 fallocate 0x003104 thru 0x003bb6 (0xab3 bytes) 000373 1768140523.177109 mapread 0x001f8b thru 0x00d369 (0xb3df bytes) 000374 1768140523.179388 punch from 0x000609 to 0x009057 (0x8a4f bytes) 000375 1768140523.180853 trunc from 0x032758 to 0x032c72 (0x51b bytes) 000376 1768140523.185108 trunc from 0x032c73 to 0x011c03 (0x2106f bytes) 000377 1768140523.188721 trunc from 0x00fb91 to 0x011c03 (0x2073 bytes) 000378 1768140523.191352 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11c04): Mirror 4: 0x6778f84f (stale) Mirror 5: 0x6778f84f (stale) Mirror 6: 0x8527181c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c04): Mirror 4: 0x6778f84f (stale) Mirror 5: 0x6778f84f (stale) Mirror 6: 0x8527181c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c04: skipped 000379 1768140523.259632 mapread 0x005db7 thru 0x011c03 (0xbe4d bytes) 000380 1768140523.265258 fallocate 0x006eb2 thru 0x00cb36 (0x5c85 bytes) 000381 1768140523.268450 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11c04): Mirror 4: 0x6778f84f (stale) Mirror 5: 0x6778f84f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c04): Mirror 4: 0x6778f84f (stale) Mirror 5: 0x6778f84f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c04: skipped 000382 1768140523.337630 mapwrite 0x030bfe thru 0x03af4b (0xa34e bytes) 000383 1768140523.350485 mapwrite 0x022790 thru 0x02896b (0x61dc bytes) 000384 1768140523.356670 fallocate 0x02f1ef thru 0x03a216 (0xb028 bytes) 000385 1768140523.358135 mapread 0x02c6a5 thru 0x02c6d9 (0x35 bytes) 000386 1768140523.358593 read 0x01099b thru 0x01c875 (0xbedb bytes) 000387 1768140523.360679 trunc from 0x00e317 to 0x01c908 (0xe5f2 bytes) 000388 1768140523.362309 punch from 0x01fca6 to 0x02c251 (0xc5ac bytes) 000389 1768140523.363704 trunc from 0x0102a0 to 0x0183b8 (0x8119 bytes) 000390 1768140523.364957 fallocate 0x02da65 thru 0x03d55a (0xfaf6 bytes) 000391 1768140523.366546 punch from 0x01d9d7 to 0x0297d1 (0xbdfb bytes) 000392 1768140523.367868 mapwrite 0x03e38a thru 0x03ffff (0x1c76 bytes) 000393 1768140523.376557 fallocate 0x01f679 thru 0x0282eb (0x8c73 bytes) 000394 1768140523.378117 punch from 0x0306bf to 0x031a9d (0x13df bytes) 000395 1768140523.380607 MIRROR_RESYNC 3 mirrors 000396 1768140523.559960 punch from 0x018b8b to 0x021e44 (0x92ba bytes) 000397 1768140523.567850 MIRROR_RESYNC 3 mirrors 000398 1768140523.744414 read 0x022d89 thru 0x0249f8 (0x1c70 bytes) 000399 1768140523.745805 trunc from 0x02069d to 0x024468 (0x3dcc bytes) 000400 1768140523.753760 trunc from 0x040000 to 0x03bce6 (0x4319 bytes) 000402 1768140523.762101 trunc from 0x03bce7 to 0x0097bd (0x32529 bytes) 000403 1768140523.765811 read 0x004c17 thru 0x0097bd (0x4ba7 bytes) 000404 1768140523.768014 fallocate 0x00fa3e thru 0x01b755 (0xbd18 bytes) 000405 1768140523.770497 MIRROR_RESYNC 3 mirrors 000406 1768140523.911939 write 0x010d87 thru 0x0191f5 (0x846f bytes) 000407 1768140523.920110 punch from 0x0067b0 to 0x00c26d (0x5abe bytes) 000408 1768140523.921466 mapwrite 0x003087 thru 0x0104b7 (0xd431 bytes) 000409 1768140523.929568 mapwrite 0x00006d thru 0x00f7be (0xf752 bytes) 000410 1768140523.937395 MIRROR_RESYNC 3 mirrors 000411 1768140524.083553 mapwrite 0x022266 thru 0x022b78 (0x913 bytes) 000412 1768140524.095921 mapread 0x0227bf thru 0x022b78 (0x3ba bytes) 000413 1768140524.096463 write 0x010c10 thru 0x012f4e (0x233f bytes) 000414 1768140524.099609 trunc from 0x022b79 to 0x0181be (0xa9ba bytes) 000415 1768140524.104165 punch from 0x00353a to 0x00ba50 (0x8517 bytes) 000416 1768140524.105628 read 0x0116bf thru 0x017e72 (0x67b4 bytes) 000417 1768140524.105962 mapread 0x001113 thru 0x007d85 (0x6c73 bytes) 000419 1768140524.108507 mapwrite 0x0131de thru 0x01bd45 (0x8b68 bytes) 000420 1768140524.118246 fallocate 0x015267 thru 0x021d72 (0xcb0c bytes) 000421 1768140524.119822 fallocate 0x011f8a thru 0x01b19d (0x9214 bytes) 000422 1768140524.121332 fallocate 0x03bac0 thru 0x03ffff (0x4540 bytes) 000423 1768140524.122701 trunc from 0x040000 to 0x013aee (0x2c511 bytes) 000424 1768140524.126419 trunc from 0x013aef to 0x018894 (0x4da6 bytes) 000425 1768140524.129921 read 0x006556 thru 0x0092d0 (0x2d7b bytes) 000426 1768140524.130353 mapread 0x00e4ac thru 0x018894 (0xa3e9 bytes) 000427 1768140524.133937 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18895): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18895): Mirror 4: 0x2e15eb29 (stale) Mirror 5: 0x2e15eb29 (stale) CRC-32 checksum value for chunk [0, 0x18895): Mirror 1: 0x9e688228 Mirror 6: 0x9e688228 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18895: skipped 000428 1768140524.315616 write 0x012b20 thru 0x014856 (0x1d37 bytes) 000429 1768140524.327023 trunc from 0x0143ab to 0x018894 (0x44ea bytes) 000430 1768140524.329791 fallocate 0x02a36f thru 0x031be4 (0x7876 bytes) 000431 1768140524.332490 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31be5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31be5): Mirror 4: 0x14112a1c (stale) Mirror 5: 0x14112a1c (stale) Mirror 6: 0xa1c1b400 (stale) CRC-32 checksum value for chunk [0, 0x31be5): Mirror 1: 0x690cc339 Mirror 7: 0x690cc339 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31be5: skipped 000432 1768140524.529651 write 0x035cb7 thru 0x03ffff (0xa349 bytes) 000433 1768140524.539276 read 0x007345 thru 0x009593 (0x224f bytes) 000436 1768140524.541608 write 0x00e33f thru 0x0174b5 (0x9177 bytes) 000437 1768140524.544429 trunc from 0x026c3f to 0x034c3b (0xdffd bytes) 000438 1768140524.546921 MIRROR_RESYNC 5 mirrors 000440 1768140524.768250 write 0x036f6b thru 0x03ffff (0x9095 bytes) 000441 1768140524.776438 punch from 0x003828 to 0x00ff6c (0xc745 bytes) 000442 1768140524.777968 mapwrite 0x020d12 thru 0x02241d (0x170c bytes) 000443 1768140524.784823 punch from 0x030712 to 0x0343c0 (0x3caf bytes) 000445 1768140524.786504 fallocate 0x01aab6 thru 0x020ce6 (0x6231 bytes) 000446 1768140524.788021 fallocate 0x03ae1f thru 0x03e1e5 (0x33c7 bytes) 000448 1768140524.789687 trunc from 0x02a504 to 0x02d483 (0x2f80 bytes) 000449 1768140524.790968 write 0x01faad thru 0x025fbe (0x6512 bytes) 000450 1768140524.791621 punch from 0x02bdff to 0x038a77 (0xcc79 bytes) 000451 1768140524.792974 mapread 0x014ee9 thru 0x01c8d4 (0x79ec bytes) 000452 1768140524.795380 trunc from 0x040000 to 0x028614 (0x179eb bytes) 000453 1768140524.803770 mapwrite 0x001f74 thru 0x0029a6 (0xa33 bytes) 000454 1768140524.810013 trunc from 0x01580b to 0x02165f (0xbe55 bytes) 000456 1768140524.812092 read 0x026d6a thru 0x028614 (0x18ab bytes) 000457 1768140524.812668 fallocate 0x0057b5 thru 0x006bfe (0x144a bytes) 000458 1768140524.815455 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28615): Mirror 4: 0xecfa3076 (stale) Mirror 5: 0xecfa3076 (stale) Mirror 6: 0xecfa3076 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28615): Mirror 4: 0xecfa3076 (stale) Mirror 5: 0xecfa3076 (stale) Mirror 6: 0xecfa3076 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28615: skipped 000459 1768140524.892075 trunc from 0x028615 to 0x00c2f5 (0x1c31f bytes) 000460 1768140524.896840 mapwrite 0x03a22c thru 0x03ffff (0x5dd4 bytes) 000461 1768140524.908343 mapwrite 0x001391 thru 0x0054ac (0x411c bytes) 000463 1768140524.914689 write 0x0204ee thru 0x027696 (0x71a9 bytes) 000464 1768140524.917192 mapwrite 0x007c75 thru 0x00a548 (0x28d4 bytes) 000465 1768140524.921880 fallocate 0x037d2e thru 0x03f5a3 (0x7876 bytes) 000466 1768140524.923719 punch from 0x02854d to 0x029e39 (0x18ed bytes) 000467 1768140524.924943 punch from 0x00656e to 0x00a2e5 (0x3d78 bytes) 000468 1768140524.926689 mapread 0x0166ed thru 0x021510 (0xae24 bytes) 000469 1768140524.929417 punch from 0x00d2fc to 0x0177fb (0xa500 bytes) 000470 1768140524.930790 fallocate 0x00b5eb thru 0x00cd6e (0x1784 bytes) 000471 1768140524.932185 fallocate 0x028eea thru 0x033637 (0xa74e bytes) 000472 1768140524.935208 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe0dcbc4f (stale) Mirror 6: 0xe0dcbc4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe0dcbc4f (stale) Mirror 6: 0xe0dcbc4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1768140525.010010 mapread 0x01dc83 thru 0x029db5 (0xc133 bytes) 000474 1768140525.017262 trunc from 0x0194d0 to 0x029494 (0xffc5 bytes) 000475 1768140525.020516 MIRROR_RESYNC 3 mirrors 000477 1768140525.226524 mapwrite 0x00025b thru 0x00fd10 (0xfab6 bytes) 000478 1768140525.262097 write 0x01e7ae thru 0x0221c9 (0x3a1c bytes) 000479 1768140525.265336 trunc from 0x01dfe2 to 0x02c500 (0xe51f bytes) 000480 1768140525.269500 trunc from 0x036c27 to 0x03c048 (0x5422 bytes) 000482 1768140525.271461 read 0x02bd25 thru 0x033104 (0x73e0 bytes) 000483 1768140525.274144 fallocate 0x03c97d thru 0x03f7d0 (0x2e54 bytes) 000484 1768140525.275848 mapread 0x0129c2 thru 0x021824 (0xee63 bytes) 000485 1768140525.281505 punch from 0x0275da to 0x036109 (0xeb30 bytes) 000486 1768140525.284983 MIRROR_RESYNC 3 mirrors 000487 1768140525.471086 trunc from 0x040000 to 0x008204 (0x37dfb bytes) 000488 1768140525.485942 write 0x01acd0 thru 0x02043c (0x576d bytes) 000489 1768140525.487242 read 0x01bff1 thru 0x02043c (0x444c bytes) 000490 1768140525.487812 punch from 0x0032f6 to 0x01098a (0xd695 bytes) 000492 1768140525.490002 mapread 0x000c14 thru 0x001862 (0xc4f bytes) 000493 1768140525.493022 trunc from 0x016a23 to 0x01edd8 (0x83b6 bytes) 000494 1768140525.496038 trunc from 0x016b6b to 0x02043c (0x98d2 bytes) 000495 1768140525.497244 mapread 0x01d6b1 thru 0x02043c (0x2d8c bytes) 000496 1768140525.500288 fallocate 0x001ab6 thru 0x010477 (0xe9c2 bytes) 000497 1768140525.501818 fallocate 0x01c2d7 thru 0x020fc8 (0x4cf2 bytes) 000498 1768140525.504437 MIRROR_RESYNC 3 mirrors 000499 1768140525.668831 trunc from 0x004e6f to 0x008689 (0x381b bytes) 000501 1768140525.679098 write 0x0292e8 thru 0x029884 (0x59d bytes) 000503 1768140525.681315 MIRROR_RESYNC 3 mirrors 000504 1768140525.853304 mapwrite 0x03f705 thru 0x03ffff (0x8fb bytes) 000505 1768140525.875830 punch from 0x028946 to 0x02c908 (0x3fc3 bytes) 000506 1768140525.877136 write 0x029934 thru 0x02d34e (0x3a1b bytes) 000507 1768140525.880475 trunc from 0x040000 to 0x024700 (0x1b8ff bytes) 000508 1768140525.885597 punch from 0x01b90d to 0x02213e (0x6832 bytes) 000511 1768140525.888032 read 0x01a46c thru 0x024700 (0xa295 bytes) 000513 1768140525.891310 write 0x0098db thru 0x013f05 (0xa62b bytes) 000514 1768140525.895596 fallocate 0x03f37f thru 0x03ffff (0xc81 bytes) 000515 1768140525.897350 punch from 0x0091ca to 0x01089c (0x76d3 bytes) 000516 1768140525.900851 read 0x001fa1 thru 0x00df6f (0xbfcf bytes) 000517 1768140525.904339 mapwrite 0x03e083 thru 0x03ffff (0x1f7d bytes) 000518 1768140525.908180 fallocate 0x00eef4 thru 0x019248 (0xa355 bytes) 000519 1768140525.909639 mapwrite 0x01c67d thru 0x0215cf (0x4f53 bytes) 000520 1768140525.914130 mapwrite 0x00b137 thru 0x00cff9 (0x1ec3 bytes) 000521 1768140525.917785 write 0x032ad9 thru 0x03ffff (0xd527 bytes) 000522 1768140525.918708 fallocate 0x03e220 thru 0x03ffff (0x1de0 bytes) 000523 1768140525.920400 punch from 0x020b8c to 0x02729a (0x670f bytes) 000524 1768140525.922033 mapread 0x020da7 thru 0x028975 (0x7bcf bytes) 000525 1768140525.932091 read 0x0064dd thru 0x0092ce (0x2df2 bytes) 000526 1768140525.932501 write 0x02707d thru 0x02d74c (0x66d0 bytes) 000527 1768140525.933160 trunc from 0x040000 to 0x022ae2 (0x1d51d bytes) 000528 1768140525.937614 mapwrite 0x022842 thru 0x02b8db (0x909a bytes) 000529 1768140525.947702 mapwrite 0x03886b thru 0x03ffff (0x7795 bytes) 000531 1768140525.959463 MIRROR_RESYNC 3 mirrors 000532 1768140526.220622 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000533 1768140526.300980 trunc from 0x003760 to 0x00704a (0x38eb bytes) 000534 1768140526.308530 write 0x030c14 thru 0x034388 (0x3775 bytes) 000535 1768140526.311774 trunc from 0x014ad9 to 0x01a6d8 (0x5c00 bytes) 000536 1768140526.313255 fallocate 0x0240a9 thru 0x025736 (0x168e bytes) 000537 1768140526.314649 punch from 0x01f221 to 0x027857 (0x8637 bytes) 000538 1768140526.315970 write 0x008b73 thru 0x018ab8 (0xff46 bytes) 000539 1768140526.319577 mapwrite 0x02c4d8 thru 0x035497 (0x8fc0 bytes) 000540 1768140526.328086 write 0x012e74 thru 0x021523 (0xe6b0 bytes) 000541 1768140526.330896 MIRROR_RESYNC 2 mirrors 000542 1768140526.499395 MIRROR_RESYNC 2 mirrors 000543 1768140526.594939 trunc from 0x040000 to 0x01f426 (0x20bd9 bytes) 000544 1768140526.608044 trunc from 0x01f427 to 0x034be7 (0x157c1 bytes) 000545 1768140526.612236 trunc from 0x00c2fc to 0x0145f6 (0x82fb bytes) 000548 1768140526.614497 mapread 0x02b07c thru 0x034be7 (0x9b6c bytes) 000549 1768140526.619999 MIRROR_RESYNC 2 mirrors 000552 1768140526.774256 read 0x00c4d0 thru 0x019281 (0xcdb2 bytes) 000553 1768140526.776795 trunc from 0x034be8 to 0x031f78 (0x2c6f bytes) 000554 1768140526.790487 fallocate 0x032bd9 thru 0x03ffff (0xd427 bytes) 000555 1768140526.793385 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31f79): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f79): Mirror 4: 0x402670a (stale) CRC-32 checksum value for chunk [0, 0x31f79): Mirror 1: 0x402670a Mirror 5: 0x402670a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f79: skipped 000556 1768140526.985932 trunc from 0x00eae9 to 0x00fdb8 (0x12d0 bytes) 000557 1768140526.994739 trunc from 0x031f79 to 0x013a1a (0x1e55e bytes) 000558 1768140527.004501 read 0x005450 thru 0x00e40e (0x8fbf bytes) 000559 1768140527.006900 punch from 0x002b66 to 0x010522 (0xd9bd bytes) 000560 1768140527.008420 punch from 0x011b7b to 0x0136b3 (0x1b39 bytes) 000561 1768140527.009644 write 0x02e37c thru 0x037572 (0x91f7 bytes) 000562 1768140527.010476 write 0x021b2d thru 0x02a938 (0x8e0c bytes) 000564 1768140527.013137 read 0x030d78 thru 0x031c04 (0xe8d bytes) 000565 1768140527.013525 fallocate 0x004a80 thru 0x01232b (0xd8ac bytes) 000566 1768140527.014980 fallocate 0x036374 thru 0x03a1a3 (0x3e30 bytes) 000567 1768140527.016405 read 0x02dcdd thru 0x036ace (0x8df2 bytes) 000568 1768140527.019262 mapwrite 0x002c0a thru 0x00b16e (0x8565 bytes) 000569 1768140527.025167 fallocate 0x00c72c thru 0x01a69e (0xdf73 bytes) 000570 1768140527.027063 mapwrite 0x010c60 thru 0x012a82 (0x1e23 bytes) 000571 1768140527.031300 trunc from 0x03a1a4 to 0x02f980 (0xa823 bytes) 000572 1768140527.036395 MIRROR_RESYNC 3 mirrors 000573 1768140527.228487 read 0x01c240 thru 0x02269f (0x6460 bytes) 000574 1768140527.230490 read 0x015267 thru 0x023a52 (0xe7ec bytes) 000576 1768140527.233706 MIRROR_EXTEND to 4 mirrors 000577 1768140527.385976 mapwrite 0x038c27 thru 0x03bfa0 (0x337a bytes) 000579 1768140527.410537 write 0x03ba9c thru 0x03ffff (0x4564 bytes) 000580 1768140527.411620 read 0x01a163 thru 0x01f046 (0x4ee4 bytes) 000581 1768140527.414549 mapwrite 0x01357c thru 0x015b81 (0x2606 bytes) 000582 1768140527.422260 read 0x009149 thru 0x00f157 (0x600f bytes) 000583 1768140527.424967 write 0x0376ee thru 0x03d609 (0x5f1c bytes) 000585 1768140527.426187 trunc from 0x002e1c to 0x00af18 (0x80fd bytes) 000586 1768140527.427858 mapwrite 0x03de3d thru 0x03ffff (0x21c3 bytes) 000587 1768140527.433224 mapread 0x02e887 thru 0x03c54d (0xdcc7 bytes) 000589 1768140527.434086 fallocate 0x01c83f thru 0x025fd9 (0x979b bytes) 000590 1768140527.435849 trunc from 0x040000 to 0x01831c (0x27ce3 bytes) 000591 1768140527.441562 fallocate 0x02959a thru 0x036abf (0xd526 bytes) 000592 1768140527.444339 MIRROR_RESYNC 4 mirrors 000593 1768140527.656444 read 0x017aa3 thru 0x0200fc (0x865a bytes) 000594 1768140527.658678 fallocate 0x01b815 thru 0x020744 (0x4f30 bytes) 000595 1768140527.667938 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36ac0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ac0): Mirror 4: 0x5bbb1551 (stale) Mirror 5: 0x5bbb1551 (stale) Mirror 6: 0x5bbb1551 (stale) CRC-32 checksum value for chunk [0, 0x36ac0): Mirror 1: 0x5bbb1551 Mirror 7: 0x5bbb1551 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ac0: skipped 000596 1768140527.877514 trunc from 0x00f7eb to 0x011083 (0x1899 bytes) 000597 1768140527.887121 trunc from 0x036ac0 to 0x003b4e (0x32f71 bytes) 000598 1768140527.896426 mapwrite 0x009a98 thru 0x01191c (0x7e85 bytes) 000599 1768140527.908709 trunc from 0x00c0f0 to 0x00c555 (0x466 bytes) 000600 1768140527.910186 punch from 0x00b9f2 to 0x01191c (0x5f2b bytes) 000601 1768140527.911589 trunc from 0x01191d to 0x02f62a (0x1dd0e bytes) 000602 1768140527.915467 trunc from 0x003256 to 0x00ab8a (0x7935 bytes) 000603 1768140527.917212 trunc from 0x01a025 to 0x027fa6 (0xdf82 bytes) 000604 1768140527.918862 trunc from 0x029b81 to 0x02f62a (0x5aaa bytes) 000605 1768140527.920443 punch from 0x02b1d6 to 0x02f62a (0x4455 bytes) 000606 1768140527.922025 read 0x0075fe thru 0x01161f (0xa022 bytes) 000607 1768140527.924886 write 0x01036e thru 0x010af6 (0x789 bytes) 000608 1768140527.925707 trunc from 0x02f62b to 0x02f06a (0x5c0 bytes) 000609 1768140527.929647 trunc from 0x02e185 to 0x02f06a (0xee6 bytes) 000610 1768140527.931227 mapread 0x018061 thru 0x026995 (0xe935 bytes) 000611 1768140527.935863 write 0x036c4e thru 0x036dbb (0x16e bytes) 000612 1768140527.936431 punch from 0x02ba42 to 0x030a0a (0x4fc9 bytes) 000613 1768140527.938121 punch from 0x00936c to 0x010ece (0x7b63 bytes) 000614 1768140527.942249 write 0x019f7b thru 0x01b4d4 (0x155a bytes) 000615 1768140527.943052 mapread 0x033ad4 thru 0x036dbb (0x32e8 bytes) 000616 1768140527.945586 punch from 0x01dd19 to 0x02abac (0xce94 bytes) 000617 1768140527.947234 mapwrite 0x02b4c5 thru 0x0377df (0xc31b bytes) 000618 1768140527.961482 trunc from 0x0377e0 to 0x03941f (0x1c40 bytes) 000619 1768140527.964911 mapwrite 0x03586b thru 0x037ff1 (0x2787 bytes) 000620 1768140527.968184 mapwrite 0x009262 thru 0x00eeb7 (0x5c56 bytes) 000621 1768140527.973985 write 0x023e0d thru 0x032629 (0xe81d bytes) 000622 1768140527.976454 punch from 0x034e8f to 0x03941f (0x4591 bytes) 000623 1768140527.977963 trunc from 0x039420 to 0x0377a7 (0x1c78 bytes) 000624 1768140527.981288 mapread 0x024601 thru 0x030fe4 (0xc9e4 bytes) 000625 1768140527.981809 trunc from 0x019c6e to 0x025c88 (0xc01b bytes) 000626 1768140527.985180 write 0x010cf9 thru 0x0116bd (0x9c5 bytes) 000627 1768140527.987154 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x377a8): Mirror 4: 0xf37b33c5 (stale) Mirror 6: 0xf37b33c5 (stale) Mirror 7: 0xf37b33c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377a8): Mirror 4: 0xf37b33c5 (stale) Mirror 6: 0xf37b33c5 (stale) Mirror 7: 0xf37b33c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377a8: skipped 000628 1768140528.068145 trunc from 0x0377a8 to 0x032d50 (0x4a57 bytes) 000629 1768140528.073403 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32d51): Mirror 6: 0xb82cf64f (stale) Mirror 7: 0xb82cf64f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d51): Mirror 6: 0xb82cf64f (stale) Mirror 7: 0xb82cf64f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d51: skipped 000630 1768140528.148987 mapwrite 0x03333d thru 0x03ba37 (0x86fb bytes) 000631 1768140528.161772 read 0x0068ca thru 0x0108ef (0xa026 bytes) 000632 1768140528.163766 write 0x01ca8f thru 0x02c52e (0xfaa0 bytes) 000633 1768140528.167159 trunc from 0x03871f to 0x03ba37 (0x3319 bytes) 000634 1768140528.168625 read 0x02a5b2 thru 0x031381 (0x6dd0 bytes) 000635 1768140528.170607 punch from 0x0190ae to 0x023c81 (0xabd4 bytes) 000636 1768140528.173806 punch from 0x01e459 to 0x02a8bd (0xc465 bytes) 000637 1768140528.175133 trunc from 0x0069f0 to 0x006ce3 (0x2f4 bytes) 000638 1768140528.176473 trunc from 0x03ba38 to 0x009d9e (0x31c99 bytes) 000639 1768140528.180390 read 0x002100 thru 0x009d9e (0x7c9f bytes) 000640 1768140528.181846 trunc from 0x009d9f to 0x038d1d (0x2ef7f bytes) 000641 1768140528.185281 mapwrite 0x012503 thru 0x01c4dd (0x9fdb bytes) 000642 1768140528.193111 trunc from 0x038d1e to 0x00b4d7 (0x2d846 bytes) 000643 1768140528.197280 read 0x006611 thru 0x007b0a (0x14fa bytes) 000644 1768140528.197660 mapwrite 0x017804 thru 0x01e6b1 (0x6eae bytes) 000645 1768140528.208760 mapwrite 0x03106c thru 0x03183b (0x7d0 bytes) 000646 1768140528.218324 mapread 0x01a1e3 thru 0x01ecfa (0x4b18 bytes) 000647 1768140528.218874 mapwrite 0x028946 thru 0x02eee9 (0x65a4 bytes) 000648 1768140528.224054 read 0x0137aa thru 0x01b430 (0x7c87 bytes) 000649 1768140528.224565 read 0x007b7b thru 0x00febd (0x8343 bytes) 000650 1768140528.224897 read 0x020195 thru 0x02ca8e (0xc8fa bytes) 000651 1768140528.225321 punch from 0x00ba7e to 0x013d02 (0x8285 bytes) 000652 1768140528.228338 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3183c): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3183c): Mirror 6: 0x3c7fc470 (stale) Mirror 7: 0x3c7fc470 (stale) CRC-32 checksum value for chunk [0, 0x3183c): Mirror 1: 0x46f036c6 Mirror 8: 0x46f036c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3183c: skipped 000653 1768140528.436014 read 0x01c420 thru 0x01de6e (0x1a4f bytes) 000654 1768140528.439514 mapwrite 0x03c758 thru 0x03ffff (0x38a8 bytes) 000655 1768140528.462794 punch from 0x011163 to 0x015a89 (0x4927 bytes) 000656 1768140528.464100 mapread 0x035583 thru 0x03777f (0x21fd bytes) 000657 1768140528.466689 mapwrite 0x030682 thru 0x03bc76 (0xb5f5 bytes) 000658 1768140528.473591 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x55fc1dd6 (stale) Mirror 8: 0x7acc1fca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x55fc1dd6 (stale) Mirror 8: 0x7acc1fca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000659 1768140528.545647 read 0x019313 thru 0x021425 (0x8113 bytes) 000660 1768140528.549044 MIRROR_RESYNC 3 mirrors 000661 1768140528.760569 MIRROR_EXTEND to 4 mirrors 000662 1768140528.923987 read 0x038146 thru 0x03ffff (0x7eba bytes) 000663 1768140528.926491 read 0x02c5e1 thru 0x03a677 (0xe097 bytes) 000664 1768140528.930129 punch from 0x032313 to 0x032beb (0x8d9 bytes) 000665 1768140528.940340 trunc from 0x0280ff to 0x031470 (0x9372 bytes) 000666 1768140528.942169 mapread 0x03065e thru 0x0323dc (0x1d7f bytes) 000667 1768140528.947628 MIRROR_RESYNC 4 mirrors 000668 1768140529.208799 trunc from 0x02daf0 to 0x03399a (0x5eab bytes) 000669 1768140529.216339 fallocate 0x019020 thru 0x020689 (0x766a bytes) 000670 1768140529.217873 read 0x02e37e thru 0x034e8b (0x6b0e bytes) 000671 1768140529.221316 MIRROR_RESYNC 4 mirrors 000672 1768140529.478205 MIRROR_EXTEND to 5 mirrors 000673 1768140529.638883 write 0x031b7d thru 0x039b96 (0x801a bytes) 000674 1768140529.649046 fallocate 0x02ab6d thru 0x030397 (0x582b bytes) 000676 1768140529.650981 trunc from 0x03a562 to 0x03ffff (0x5a9e bytes) 000677 1768140529.652336 fallocate 0x004d26 thru 0x01094b (0xbc26 bytes) 000678 1768140529.653777 trunc from 0x040000 to 0x036a7d (0x9582 bytes) 000679 1768140529.658725 trunc from 0x00a762 to 0x0138a1 (0x9140 bytes) 000680 1768140529.660187 punch from 0x003668 to 0x0059ac (0x2345 bytes) 000682 1768140529.661736 trunc from 0x020caa to 0x020ec6 (0x21d bytes) 000683 1768140529.663175 write 0x027c2f thru 0x02f68e (0x7a60 bytes) 000684 1768140529.666056 punch from 0x02539c to 0x025c80 (0x8e5 bytes) 000685 1768140529.667447 mapread 0x020124 thru 0x022fae (0x2e8b bytes) 000688 1768140529.669138 trunc from 0x01e755 to 0x026b9a (0x8446 bytes) 000689 1768140529.672248 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a7e): Mirror 7: 0x90ec211b (stale) Mirror 8: 0x90ec211b (stale) Mirror 9: 0x90ec211b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a7e): Mirror 7: 0x90ec211b (stale) Mirror 8: 0x90ec211b (stale) Mirror 9: 0x90ec211b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a7e: skipped 000690 1768140529.766815 write 0x0282b0 thru 0x0341bf (0xbf10 bytes) 000691 1768140529.770597 fallocate 0x033d4d thru 0x03ffff (0xc2b3 bytes) 000692 1768140529.772119 trunc from 0x036a7e to 0x02ace5 (0xbd98 bytes) 000693 1768140529.776512 write 0x02cc8a thru 0x03bbb6 (0xef2d bytes) 000694 1768140529.777568 punch from 0x039d02 to 0x03bbb6 (0x1eb5 bytes) 000696 1768140529.781576 read 0x03ab8d thru 0x03bbb6 (0x102a bytes) 000697 1768140529.783093 trunc from 0x03bbb7 to 0x02abdf (0x10fd7 bytes) 000698 1768140529.786961 punch from 0x00027b to 0x00f244 (0xefca bytes) 000699 1768140529.788463 read 0x0032b6 thru 0x00e9ab (0xb6f6 bytes) 000701 1768140529.791869 trunc from 0x01c512 to 0x021d5a (0x5849 bytes) 000702 1768140529.793647 mapread 0x005b39 thru 0x00fc5d (0xa125 bytes) 000704 1768140529.794898 punch from 0x026efb to 0x02abdf (0x3ce5 bytes) 000705 1768140529.796177 read 0x024557 thru 0x02abdf (0x6689 bytes) 000706 1768140529.798385 trunc from 0x02abe0 to 0x02dc5b (0x307c bytes) 000707 1768140529.801759 mapwrite 0x029c74 thru 0x035205 (0xb592 bytes) 000708 1768140529.812729 punch from 0x000496 to 0x002b0b (0x2676 bytes) 000709 1768140529.814280 write 0x01a6f8 thru 0x028ce9 (0xe5f2 bytes) 000710 1768140529.815567 fallocate 0x031507 thru 0x03fd85 (0xe87f bytes) 000711 1768140529.817165 trunc from 0x03fd86 to 0x028b8a (0x171fb bytes) 000712 1768140529.820770 mapwrite 0x02756f thru 0x035273 (0xdd05 bytes) 000713 1768140529.842117 write 0x00d9b4 thru 0x013210 (0x585d bytes) 000714 1768140529.843177 fallocate 0x023b78 thru 0x02eb00 (0xaf89 bytes) 000715 1768140529.844929 mapread 0x019c4d thru 0x01e908 (0x4cbc bytes) 000716 1768140529.845430 trunc from 0x01e842 to 0x023fec (0x57ab bytes) 000717 1768140529.846767 punch from 0x034fed to 0x035273 (0x287 bytes) 000718 1768140529.848057 punch from 0x02d630 to 0x02d8b4 (0x285 bytes) 000719 1768140529.849344 mapwrite 0x00e425 thru 0x01a1e9 (0xbdc5 bytes) 000720 1768140529.856511 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35274): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35274): Mirror 7: 0xbaac16ed (stale) Mirror 8: 0xbaac16ed (stale) Mirror 9: 0xbaac16ed (stale) CRC-32 checksum value for chunk [0, 0x35274): Mirror 1: 0x1cbdf435 Mirror 10: 0x1cbdf435 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35274: skipped 000721 1768140530.064783 trunc from 0x035274 to 0x009b19 (0x2b75a bytes) 000722 1768140530.080057 trunc from 0x009b1a to 0x0139cc (0x9eb3 bytes) 000723 1768140530.086185 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x139cd): Mirror 8: 0xf59149fa (stale) Mirror 9: 0xf59149fa (stale) Mirror 10: 0x6271ffc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x139cd): Mirror 8: 0xf59149fa (stale) Mirror 9: 0xf59149fa (stale) Mirror 10: 0x6271ffc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x139cd: skipped 000724 1768140530.171696 trunc from 0x0139cd to 0x00ea75 (0x4f57 bytes) 000725 1768140530.176824 mapread 0x00cc76 thru 0x00ea75 (0x1e00 bytes) 000726 1768140530.182026 trunc from 0x00ea76 to 0x02c9e7 (0x1df72 bytes) 000727 1768140530.186112 mapwrite 0x01e6c8 thru 0x0291ce (0xab07 bytes) 000728 1768140530.195589 fallocate 0x00bb92 thru 0x00c2ad (0x71c bytes) 000729 1768140530.197100 mapread 0x012c91 thru 0x0214d0 (0xe840 bytes) 000730 1768140530.199646 punch from 0x024d53 to 0x025ce6 (0xf94 bytes) 000731 1768140530.200940 write 0x0368ef thru 0x0389a4 (0x20b6 bytes) 000732 1768140530.202840 MIRROR_RESYNC 4 mirrors 000733 1768140530.407689 mapwrite 0x018144 thru 0x01e281 (0x613e bytes) 000735 1768140530.441816 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x389a5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389a5): Mirror 8: 0xf76429e4 (stale) Mirror 9: 0xf76429e4 (stale) Mirror 10: 0xf76429e4 (stale) CRC-32 checksum value for chunk [0, 0x389a5): Mirror 1: 0x280f86c Mirror 11: 0x280f86c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389a5: skipped 000736 1768140530.664978 trunc from 0x0389a5 to 0x03131b (0x7689 bytes) 000737 1768140530.681145 mapwrite 0x03f52e thru 0x03ffff (0xad2 bytes) 000738 1768140530.691193 mapwrite 0x03ee19 thru 0x03ffff (0x11e7 bytes) 000739 1768140530.697458 trunc from 0x028729 to 0x036f29 (0xe801 bytes) 000740 1768140530.699343 write 0x03ce95 thru 0x03ffff (0x316b bytes) 000741 1768140530.702432 trunc from 0x040000 to 0x027442 (0x18bbd bytes) 000742 1768140530.708460 MIRROR_RESYNC 5 mirrors 000743 1768140530.906389 punch from 0x024185 to 0x027442 (0x32be bytes) 000744 1768140530.916152 mapread 0x00f6f7 thru 0x0141e4 (0x4aee bytes) 000745 1768140530.921195 punch from 0x00a625 to 0x01948d (0xee69 bytes) 000746 1768140530.923028 fallocate 0x024537 thru 0x028fb7 (0x4a81 bytes) 000747 1768140530.924771 trunc from 0x028fb8 to 0x03756d (0xe5b6 bytes) 000748 1768140530.935284 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3756e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3756e): Mirror 8: 0x8da03b59 (stale) Mirror 9: 0x8da03b59 (stale) Mirror 10: 0x8da03b59 (stale) Mirror 11: 0x8da03b59 (stale) CRC-32 checksum value for chunk [0, 0x3756e): Mirror 1: 0xde4bd5dd Mirror 12: 0xde4bd5dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3756e: skipped 000749 1768140531.168179 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3756e): Mirror 8: 0x8da03b59 (stale) Mirror 9: 0x8da03b59 (stale) Mirror 10: 0x8da03b59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3756e): Mirror 8: 0x8da03b59 (stale) Mirror 9: 0x8da03b59 (stale) Mirror 10: 0x8da03b59 (stale) CRC-32 checksum value for chunk [0, 0x3756e): Mirror 1: 0xde4bd5dd Mirror 12: 0xde4bd5dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3756e: skipped 000750 1768140531.354613 trunc from 0x03756e to 0x018ddc (0x1e791 bytes) 000751 1768140531.367450 fallocate 0x007b8b thru 0x00b769 (0x3bdf bytes) 000752 1768140531.369054 mapread 0x00eb14 thru 0x015ea3 (0x7390 bytes) 000753 1768140531.374071 trunc from 0x0001bf to 0x004dd3 (0x4c15 bytes) 000754 1768140531.375933 trunc from 0x018ddd to 0x004a93 (0x14349 bytes) 000755 1768140531.381627 trunc from 0x002a9e to 0x004a93 (0x1ff6 bytes) 000756 1768140531.384386 MIRROR_RESYNC 5 mirrors 000757 1768140531.568464 trunc from 0x002c12 to 0x004a93 (0x1e82 bytes) 000758 1768140531.575127 punch from 0x0010fd to 0x004a93 (0x3997 bytes) 000759 1768140531.576854 read 0x0024fa thru 0x004a93 (0x259a bytes) 000760 1768140531.578891 read 0x000d14 thru 0x004a93 (0x3d80 bytes) 000761 1768140531.580572 mapwrite 0x034642 thru 0x03f2c6 (0xac85 bytes) 000762 1768140531.595318 punch from 0x016e91 to 0x026293 (0xf403 bytes) 000763 1768140531.600583 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 9: 0xad7cc07b (stale) Mirror 10: 0xad7cc07b (stale) Mirror 12: 0xad7cc07b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 9: 0xad7cc07b (stale) Mirror 10: 0xad7cc07b (stale) Mirror 12: 0xad7cc07b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c7: skipped 000764 1768140531.690831 mapread 0x02e262 thru 0x02ed2a (0xac9 bytes) 000765 1768140531.692523 fallocate 0x016c01 thru 0x0225f9 (0xb9f9 bytes) 000766 1768140531.694285 fallocate 0x020099 thru 0x0243b1 (0x4319 bytes) 000767 1768140531.695758 mapread 0x009391 thru 0x00d852 (0x44c2 bytes) 000768 1768140531.699250 read 0x00eaaa thru 0x00f9a4 (0xefb bytes) 000769 1768140531.699647 mapwrite 0x0186e1 thru 0x02396c (0xb28c bytes) 000770 1768140531.709675 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 9: 0xad7cc07b (stale) Mirror 10: 0xad7cc07b (stale) Mirror 12: 0xad7cc07b (stale) CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 1: 0x850471f Mirror 13: 0x850471f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c7: skipped 000771 1768140531.923572 read 0x0108b4 thru 0x014d0c (0x4459 bytes) 000772 1768140531.927129 write 0x00d4b1 thru 0x015204 (0x7d54 bytes) 000773 1768140531.939347 write 0x0107b5 thru 0x010ab9 (0x305 bytes) 000774 1768140531.941103 MIRROR_RESYNC 5 mirrors 000775 1768140532.221062 punch from 0x00b472 to 0x0126a1 (0x7230 bytes) 000776 1768140532.228638 mapwrite 0x0359db thru 0x03c689 (0x6caf bytes) 000777 1768140532.237833 trunc from 0x03f2c7 to 0x016f6b (0x2835b bytes) 000778 1768140532.242783 write 0x005661 thru 0x00e94f (0x92ef bytes) 000779 1768140532.246074 mapwrite 0x00286e thru 0x008c35 (0x63c8 bytes) 000780 1768140532.251888 mapread 0x00730c thru 0x0074af (0x1a4 bytes) 000781 1768140532.252688 trunc from 0x01032f to 0x016f6b (0x6c3d bytes) 000783 1768140532.254548 write 0x035505 thru 0x03b64f (0x614b bytes) 000784 1768140532.255246 mapread 0x0360df thru 0x03b64f (0x5571 bytes) 000785 1768140532.255871 fallocate 0x0335d4 thru 0x03f609 (0xc036 bytes) 000786 1768140532.257916 mapread 0x02fce8 thru 0x039610 (0x9929 bytes) 000787 1768140532.260735 mapread 0x015d44 thru 0x01e975 (0x8c32 bytes) 000788 1768140532.266503 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b650): Mirror 9: 0x42103e25 (stale) Mirror 10: 0x42103e25 (stale) Mirror 12: 0x42103e25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b650): Mirror 9: 0x42103e25 (stale) Mirror 10: 0x42103e25 (stale) Mirror 12: 0x42103e25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b650: skipped 000789 1768140532.349046 mapwrite 0x013260 thru 0x01da12 (0xa7b3 bytes) 000790 1768140532.360090 mapwrite 0x02ad4e thru 0x038932 (0xdbe5 bytes) 000791 1768140532.372113 trunc from 0x03b650 to 0x0370fd (0x4552 bytes) 000792 1768140532.377713 fallocate 0x02a48a thru 0x034199 (0x9d10 bytes) 000793 1768140532.379465 trunc from 0x033a13 to 0x0370fd (0x36eb bytes) 000794 1768140532.380911 mapwrite 0x02379c thru 0x02c569 (0x8dce bytes) 000795 1768140532.387864 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x370fe): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370fe): Mirror 9: 0x4e26620 (stale) Mirror 10: 0x4e26620 (stale) Mirror 12: 0x4e26620 (stale) CRC-32 checksum value for chunk [0, 0x370fe): Mirror 1: 0x13c37868 Mirror 13: 0x13c37868 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370fe: skipped 000796 1768140532.603115 write 0x015594 thru 0x01a4ff (0x4f6c bytes) 000799 1768140532.616525 mapwrite 0x002787 thru 0x0063a2 (0x3c1c bytes) 000800 1768140532.623709 punch from 0x0116e0 to 0x013407 (0x1d28 bytes) 000801 1768140532.625118 trunc from 0x0370fe to 0x000d24 (0x363d9 bytes) 000804 1768140532.637115 mapread 0x0003fd thru 0x000d24 (0x928 bytes) 000805 1768140532.639001 mapread 0x000c07 thru 0x000d24 (0x11e bytes) 000806 1768140532.639544 trunc from 0x000d25 to 0x02645c (0x25738 bytes) 000807 1768140532.643503 write 0x02faac thru 0x03363a (0x3b8f bytes) 000809 1768140532.644531 mapread 0x02fa92 thru 0x03363a (0x3ba9 bytes) 000810 1768140532.645045 punch from 0x02f1b8 to 0x03363a (0x4483 bytes) 000811 1768140532.649038 trunc from 0x03363b to 0x026a22 (0xcc18 bytes) 000812 1768140532.653410 mapread 0x01141e thru 0x019e97 (0x8a7a bytes) 000813 1768140532.657706 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a23): Mirror 9: 0xf0de7520 (stale) Mirror 12: 0xf0de7520 (stale) Mirror 13: 0xe3079c31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a23): Mirror 9: 0xf0de7520 (stale) Mirror 12: 0xf0de7520 (stale) Mirror 13: 0xe3079c31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a23: skipped 000814 1768140532.738146 punch from 0x017220 to 0x01955a (0x233b bytes) 000815 1768140532.739635 trunc from 0x009ff1 to 0x018d29 (0xed39 bytes) 000816 1768140532.741154 trunc from 0x001c9e to 0x00e73c (0xca9f bytes) 000817 1768140532.742638 read 0x024ca5 thru 0x024fa4 (0x300 bytes) 000818 1768140532.744310 fallocate 0x00aa9b thru 0x0161e6 (0xb74c bytes) 000819 1768140532.745777 punch from 0x005b65 to 0x00a3b8 (0x4854 bytes) 000820 1768140532.748392 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a23): Mirror 12: 0xf0de7520 (stale) Mirror 13: 0xe3079c31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a23): Mirror 12: 0xf0de7520 (stale) Mirror 13: 0xe3079c31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a23: skipped 000821 1768140532.815783 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a23): Mirror 12: 0xf0de7520 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a23): Mirror 12: 0xf0de7520 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a23: skipped 000822 1768140532.882942 fallocate 0x008331 thru 0x00d8e2 (0x55b2 bytes) 000823 1768140532.884829 fallocate 0x0080d5 thru 0x0113b3 (0x92df bytes) 000824 1768140532.886426 write 0x002f7b thru 0x0059a8 (0x2a2e bytes) 000825 1768140532.892922 mapread 0x02190c thru 0x026a22 (0x5117 bytes) 000826 1768140532.897158 MIRROR_RESYNC 2 mirrors 000827 1768140533.051562 mapread 0x004533 thru 0x00ccfc (0x87ca bytes) 000828 1768140533.055865 fallocate 0x02d7b9 thru 0x02e743 (0xf8b bytes) 000829 1768140533.063920 mapread 0x012791 thru 0x016e22 (0x4692 bytes) 000830 1768140533.066671 fallocate 0x00f47d thru 0x019a26 (0xa5aa bytes) 000831 1768140533.068011 fallocate 0x02851f thru 0x02e4b3 (0x5f95 bytes) 000832 1768140533.069469 write 0x00ef9d thru 0x017484 (0x84e8 bytes) 000833 1768140533.070429 write 0x02ac65 thru 0x02c896 (0x1c32 bytes) 000835 1768140533.071652 punch from 0x016c7c to 0x01f5a5 (0x892a bytes) 000836 1768140533.075094 mapwrite 0x0203d6 thru 0x02a311 (0x9f3c bytes) 000837 1768140533.084317 read 0x01e334 thru 0x02493e (0x660b bytes) 000838 1768140533.084913 mapread 0x0256c1 thru 0x025dc6 (0x706 bytes) 000839 1768140533.085782 fallocate 0x01df72 thru 0x02980d (0xb89c bytes) 000840 1768140533.087501 trunc from 0x00bbde to 0x0104c8 (0x48eb bytes) 000841 1768140533.089109 mapwrite 0x021f4a thru 0x03159a (0xf651 bytes) 000842 1768140533.106574 fallocate 0x02a83d thru 0x0340e4 (0x98a8 bytes) 000843 1768140533.107979 fallocate 0x0114a3 thru 0x0199da (0x8538 bytes) 000844 1768140533.109325 mapwrite 0x00b766 thru 0x00f97b (0x4216 bytes) 000845 1768140533.115902 fallocate 0x02842a thru 0x02c4b1 (0x4088 bytes) 000846 1768140533.117428 mapwrite 0x011c0d thru 0x020c33 (0xf027 bytes) 000847 1768140533.124946 trunc from 0x03159b to 0x03723e (0x5ca4 bytes) 000848 1768140533.128879 read 0x00470b thru 0x00edaf (0xa6a5 bytes) 000849 1768140533.129268 mapread 0x03204c thru 0x03723e (0x51f3 bytes) 000850 1768140533.131656 trunc from 0x01dd57 to 0x022426 (0x46d0 bytes) 000851 1768140533.133083 punch from 0x031ff9 to 0x03723e (0x5246 bytes) 000852 1768140533.134639 fallocate 0x02ce03 thru 0x03363a (0x6838 bytes) 000854 1768140533.136490 mapread 0x022779 thru 0x02c469 (0x9cf1 bytes) 000855 1768140533.138055 read 0x004d36 thru 0x0073e3 (0x26ae bytes) 000856 1768140533.138455 fallocate 0x034f4f thru 0x03da7b (0x8b2d bytes) 000857 1768140533.140183 punch from 0x0292a4 to 0x029f0c (0xc69 bytes) 000858 1768140533.141716 fallocate 0x01bca1 thru 0x01e83a (0x2b9a bytes) 000859 1768140533.144389 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3723f): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3723f): Mirror 12: 0xbbb1eb58 (stale) CRC-32 checksum value for chunk [0, 0x3723f): Mirror 1: 0xbd08d787 Mirror 13: 0xbd08d787 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3723f: skipped 000860 1768140533.361352 mapwrite 0x02fe34 thru 0x03dd65 (0xdf32 bytes) 000861 1768140533.388933 trunc from 0x03dd66 to 0x0082d4 (0x35a91 bytes) 000862 1768140533.394215 mapread 0x003880 thru 0x004266 (0x9e7 bytes) 000864 1768140533.397389 trunc from 0x002c6d to 0x0082d4 (0x5668 bytes) 000865 1768140533.399292 mapwrite 0x034c35 thru 0x03ffff (0xb3cb bytes) 000866 1768140533.412581 fallocate 0x008b9e thru 0x009ce9 (0x114c bytes) 000867 1768140533.414160 trunc from 0x026ac7 to 0x02e43d (0x7977 bytes) 000868 1768140533.415772 punch from 0x027797 to 0x030ed8 (0x9742 bytes) 000869 1768140533.417378 mapwrite 0x022d98 thru 0x02bfdd (0x9246 bytes) 000870 1768140533.426894 mapread 0x00e7c1 thru 0x017221 (0x8a61 bytes) 000871 1768140533.431196 fallocate 0x00f31c thru 0x01ea61 (0xf746 bytes) 000872 1768140533.432890 mapread 0x02d52d thru 0x035e3a (0x890e bytes) 000873 1768140533.433580 trunc from 0x040000 to 0x01611b (0x29ee4 bytes) 000874 1768140533.437295 mapread 0x0126b7 thru 0x01611b (0x3a65 bytes) 000875 1768140533.437796 trunc from 0x007ed9 to 0x010b16 (0x8c3e bytes) 000876 1768140533.440320 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1611c): Mirror 13: 0xa80f6b2d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1611c): Mirror 13: 0xa80f6b2d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1611c: skipped 000877 1768140533.506913 MIRROR_RESYNC 2 mirrors 000878 1768140533.642264 read 0x00a673 thru 0x00fead (0x583b bytes) 000879 1768140533.644247 mapwrite 0x0013fa thru 0x00a300 (0x8f07 bytes) 000880 1768140533.674585 write 0x02644d thru 0x03358a (0xd13e bytes) 000881 1768140533.675560 mapwrite 0x017da0 thru 0x01d81f (0x5a80 bytes) 000882 1768140533.682979 punch from 0x006eb4 to 0x013bba (0xcd07 bytes) 000883 1768140533.685074 punch from 0x024a1c to 0x02714b (0x2730 bytes) 000884 1768140533.686623 mapwrite 0x0353ce thru 0x03c36b (0x6f9e bytes) 000885 1768140533.706161 mapwrite 0x00c271 thru 0x018305 (0xc095 bytes) 000887 1768140533.714992 trunc from 0x011fec to 0x01ef18 (0xcf2d bytes) 000888 1768140533.717091 mapread 0x002a98 thru 0x008d96 (0x62ff bytes) 000889 1768140533.719481 MIRROR_RESYNC 2 mirrors 000890 1768140533.879077 write 0x0040f3 thru 0x00a7ed (0x66fb bytes) 000892 1768140533.889644 trunc from 0x03c36c to 0x00b70a (0x30c61 bytes) 000893 1768140533.899564 punch from 0x006949 to 0x00b6cd (0x4d85 bytes) 000894 1768140533.903169 mapread 0x0023ac thru 0x00b70a (0x935f bytes) 000895 1768140533.906102 fallocate 0x00c73f thru 0x0135c9 (0x6e8b bytes) 000896 1768140533.907906 mapread 0x007ec9 thru 0x011c1d (0x9d55 bytes) 000898 1768140533.911170 mapread 0x00c274 thru 0x01179b (0x5528 bytes) 000900 1768140533.912299 mapread 0x00ade9 thru 0x0135c9 (0x87e1 bytes) 000902 1768140533.913250 mapread 0x003e85 thru 0x011779 (0xd8f5 bytes) 000903 1768140533.913828 read 0x01359f thru 0x0135c9 (0x2b bytes) 000904 1768140533.914282 fallocate 0x03f1f0 thru 0x03ffff (0xe10 bytes) 000905 1768140533.916304 fallocate 0x01d981 thru 0x01fdbc (0x243c bytes) 000906 1768140533.918120 trunc from 0x0135ca to 0x0252f0 (0x11d27 bytes) 000907 1768140533.922045 mapwrite 0x018db1 thru 0x01d24f (0x449f bytes) 000908 1768140533.931311 MIRROR_RESYNC 2 mirrors 000909 1768140534.083724 mapread 0x000b7b thru 0x008b62 (0x7fe8 bytes) 000910 1768140534.090713 MIRROR_RESYNC 2 mirrors 000911 1768140534.184713 mapwrite 0x00d0b4 thru 0x00d965 (0x8b2 bytes) 000912 1768140534.208789 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x252f1): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x252f1): Mirror 13: 0xba139d8c (stale) CRC-32 checksum value for chunk [0, 0x252f1): Mirror 1: 0x8dab822a Mirror 14: 0x8dab822a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x252f1: skipped 000913 1768140534.429690 punch from 0x004645 to 0x00928e (0x4c4a bytes) 000914 1768140534.438969 mapread 0x006358 thru 0x006fc4 (0xc6d bytes) 000915 1768140534.442328 MIRROR_RESYNC 3 mirrors 000916 1768140534.620053 trunc from 0x0252f1 to 0x03a403 (0x15113 bytes) 000918 1768140534.632879 punch from 0x0129ec to 0x01e867 (0xbe7c bytes) 000919 1768140534.634304 trunc from 0x01cfd5 to 0x0212f8 (0x4324 bytes) 000920 1768140534.635641 trunc from 0x03a404 to 0x02e337 (0xc0cc bytes) 000921 1768140534.639772 write 0x022631 thru 0x024dae (0x277e bytes) 000922 1768140534.642561 trunc from 0x02e338 to 0x02a095 (0x42a2 bytes) 000923 1768140534.646331 read 0x028bd4 thru 0x02a095 (0x14c2 bytes) 000924 1768140534.648010 write 0x0392b5 thru 0x03ffff (0x6d4b bytes) 000925 1768140534.648720 write 0x00b920 thru 0x01b5c8 (0xfca9 bytes) 000926 1768140534.652670 punch from 0x018211 to 0x021d01 (0x9af1 bytes) 000927 1768140534.656217 read 0x02ba51 thru 0x02bad1 (0x81 bytes) 000928 1768140534.659062 fallocate 0x007324 thru 0x0140bc (0xcd99 bytes) 000930 1768140534.661473 read 0x028bdf thru 0x0361a7 (0xd5c9 bytes) 000931 1768140534.662077 mapwrite 0x0195c0 thru 0x0280db (0xeb1c bytes) 000932 1768140534.670918 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x5364c92a (stale) Mirror 14: 0x5364c92a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x78a9ad5 Mirror 15: 0x78a9ad5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000933 1768140534.867390 MIRROR_RESYNC 4 mirrors 000935 1768140535.037826 read 0x02e859 thru 0x0305a7 (0x1d4f bytes) 000936 1768140535.039511 mapwrite 0x02de55 thru 0x02efb7 (0x1163 bytes) 000937 1768140535.067641 mapwrite 0x013f70 thru 0x0165d1 (0x2662 bytes) 000938 1768140535.073381 write 0x0239ce thru 0x02dd0e (0xa341 bytes) 000939 1768140535.075951 write 0x000ed9 thru 0x004a84 (0x3bac bytes) 000940 1768140535.079426 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x78a9ad5 (stale) Mirror 14: 0x78a9ad5 (stale) Mirror 15: 0x78a9ad5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b91a4f4 Mirror 16: 0x1b91a4f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000941 1768140535.281451 trunc from 0x040000 to 0x02fc3e (0x103c1 bytes) 000942 1768140535.295821 trunc from 0x02fc3f to 0x00a700 (0x2553e bytes) 000943 1768140535.300531 read 0x0096c8 thru 0x009986 (0x2bf bytes) 000944 1768140535.303114 MIRROR_RESYNC 5 mirrors 000945 1768140535.487567 trunc from 0x008a99 to 0x0097fb (0xd63 bytes) 000946 1768140535.495866 mapread 0x00678c thru 0x00a493 (0x3d08 bytes) 000947 1768140535.500189 read 0x004244 thru 0x00a087 (0x5e44 bytes) 000948 1768140535.500568 punch from 0x001db3 to 0x00a700 (0x894e bytes) 000949 1768140535.502168 trunc from 0x004f26 to 0x00a700 (0x57db bytes) 000950 1768140535.503562 fallocate 0x033c1b thru 0x035860 (0x1c46 bytes) 000951 1768140535.504976 mapread 0x008fd4 thru 0x00a700 (0x172d bytes) 000952 1768140535.507285 fallocate 0x007f93 thru 0x010024 (0x8092 bytes) 000953 1768140535.508732 write 0x02874a thru 0x02d677 (0x4f2e bytes) 000955 1768140535.509634 read 0x02ba6c thru 0x02d677 (0x1c0c bytes) 000956 1768140535.509940 fallocate 0x03dc06 thru 0x03ffff (0x23fa bytes) 000957 1768140535.511280 write 0x037a0a thru 0x03ffff (0x85f6 bytes) 000958 1768140535.515499 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd2e081d (stale) Mirror 14: 0xd2e081d (stale) Mirror 15: 0xd2e081d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd2e081d (stale) Mirror 14: 0xd2e081d (stale) Mirror 15: 0xd2e081d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1768140535.596078 read 0x03d112 thru 0x03ffff (0x2eee bytes) 000961 1768140535.598213 trunc from 0x03de2d to 0x03ffff (0x21d3 bytes) 000962 1768140535.599744 read 0x024c3e thru 0x02ab05 (0x5ec8 bytes) 000963 1768140535.602946 mapread 0x00a1d7 thru 0x017974 (0xd79e bytes) 000965 1768140535.603772 fallocate 0x01a745 thru 0x028bac (0xe468 bytes) 000966 1768140535.605333 trunc from 0x009aa7 to 0x00c07b (0x25d5 bytes) 000968 1768140535.607199 mapread 0x01c170 thru 0x027992 (0xb823 bytes) 000969 1768140535.607728 fallocate 0x01f414 thru 0x02eab4 (0xf6a1 bytes) 000970 1768140535.609126 punch from 0x019f97 to 0x01b193 (0x11fd bytes) 000971 1768140535.610676 read 0x019b2a thru 0x021405 (0x78dc bytes) 000972 1768140535.613020 trunc from 0x0178a6 to 0x01abab (0x3306 bytes) 000973 1768140535.614332 punch from 0x02e381 to 0x033096 (0x4d16 bytes) 000974 1768140535.615837 trunc from 0x01e9ae to 0x0287b9 (0x9e0c bytes) 000975 1768140535.617188 read 0x00056f thru 0x0036b8 (0x314a bytes) 000976 1768140535.617611 mapwrite 0x020bb8 thru 0x02bb1f (0xaf68 bytes) 000977 1768140535.626649 mapread 0x0015d8 thru 0x003797 (0x21c0 bytes) 000978 1768140535.627197 write 0x0137fc thru 0x0200ca (0xc8cf bytes) 000979 1768140535.629091 trunc from 0x040000 to 0x028d58 (0x172a7 bytes) 000980 1768140535.636898 punch from 0x001bcb to 0x010b0c (0xef42 bytes) 000981 1768140535.638353 mapwrite 0x025b3b thru 0x032e8f (0xd355 bytes) 000982 1768140535.658868 trunc from 0x024324 to 0x0316f5 (0xd3d2 bytes) 000983 1768140535.660523 punch from 0x00602c to 0x011d91 (0xbd66 bytes) 000984 1768140535.661802 write 0x03e0f0 thru 0x03ffff (0x1f10 bytes) 000985 1768140535.662406 trunc from 0x026256 to 0x02d876 (0x7621 bytes) 000986 1768140535.663652 punch from 0x026a18 to 0x03388f (0xce78 bytes) 000987 1768140535.665041 mapread 0x02fc38 thru 0x03a25f (0xa628 bytes) 000988 1768140535.680253 fallocate 0x0110e7 thru 0x01edea (0xdd04 bytes) 000989 1768140535.682114 mapread 0x024953 thru 0x027e74 (0x3522 bytes) 000990 1768140535.688851 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xd2e081d (stale) Mirror 15: 0xd2e081d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xd2e081d (stale) Mirror 15: 0xd2e081d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1768140535.767298 mapwrite 0x0197aa thru 0x021a93 (0x82ea bytes) 000992 1768140535.777455 read 0x03cd85 thru 0x03ffff (0x327b bytes) 000993 1768140535.778999 punch from 0x0088c1 to 0x00950f (0xc4f bytes) 000994 1768140535.780403 write 0x0103f8 thru 0x0132cd (0x2ed6 bytes) 000995 1768140535.783849 fallocate 0x032994 thru 0x03df0c (0xb579 bytes) 000996 1768140535.786731 MIRROR_RESYNC 3 mirrors 000997 1768140535.956221 trunc from 0x040000 to 0x03915b (0x6ea4 bytes) 000998 1768140535.971834 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3915c): Mirror 15: 0x392cd4be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3915c): Mirror 15: 0x392cd4be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3915c: skipped 001000 1768140536.041135 write 0x01c701 thru 0x01d945 (0x1245 bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 71: check mirror extend parent fid ==== 09:08:57 (1768140537) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x892:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:09:00 (1768140540) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0566883 s, 37.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd44:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279532 s, 37.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd44:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd47:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd44:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd47:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd45:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 09:09:03 (1768140543) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6462112 .. Extending file size to 3847872 .. failed Extending file size to 6904320 .. Extending file size to 7817440 .. Extending file size to 8002816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8842336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8436352 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9138528 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9036096 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9038464 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 65164 65165 65167 65169 65171 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) 192.168.204.111@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:10:08 (1768140608) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 4303232 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2688640 .. Extending file size to 6319488 .. Extending file size to 9130400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7715392 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8978656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9282048 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9170848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9174688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 69273 69274 69276 69278 69280 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) 192.168.204.111@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:11:11 (1768140671) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3720 1267584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1269288 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26020 3581000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17936 3589084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43956 7170084 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3720 1267584 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2016 1269288 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26020 3581000 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17936 3589084 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43956 7170084 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0412599 s, 50.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521135 s, 40.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcf2691bd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:11:17 (1768140677) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.032319 s, 64.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 09:11:19 (1768140679) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xd5a:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:11:21 (1768140681) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0202798 s, 103 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:11:23 (1768140683) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0194302 s, 108 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd60:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd60:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5c:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:11:25 (1768140685) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0221885 s, 94.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5e:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:11:49 (1768140709) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173493 s, 60.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:12:23 (1768140743) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0183015 s, 115 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd60:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd60:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:12:56 (1768140776) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg411-server: Pool lustre.test_204f created Adding targets to pool oleg411-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg411-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0231997 s, 90.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd63:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd63:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg411-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg411-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg411-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:13:48 (1768140828) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:13:50 (1768140830) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:13:52 (1768140832) oleg411-server: Pool lustre.test_206 created Adding targets to pool oleg411-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg411-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg411-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg411-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg411-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:14:08 (1768140848) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254136 s, 41.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107431 s, 97.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x99e381b9 Mirror 3: 0x99e381b9 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:14:11 (1768140851) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:14:12 (1768140852) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:14:13 (1768140853) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.00943935 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1710000 Read this (4 bytes): BBBB 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:14:28 (1768140868) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00287807 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:14:30 (1768140870) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127071 s, 82.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8c2 lmm_fid: [0x200000405:0x8c2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8c2 lmm_fid: [0x200000405:0x8c2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613378 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-11 09:14:31.000000000 -0500 Modify: 2026-01-11 09:14:31.000000000 -0500 Change: 2026-01-11 09:14:31.000000000 -0500 Birth: 2026-01-11 09:14:31.000000000 -0500 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:14:33 (1768140873) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140931 s, 74.4 MB/s oleg411-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg411-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg411-server: O/280000401/d12/d6c: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:14:47 (1768140887) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0286606 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00424057 s, 966 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 11 09:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8c7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 11 09:14 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:14:49 (1768140889) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:14:51 (1768140891) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 789 sec ============ 09:14:54 (1768140894) === sanity-flr: start cleanup 09:14:54 (1768140894) === === sanity-flr: finish cleanup 09:14:56 (1768140896) ===