-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 19:24:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg307-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:24:43 (1787613883) === oleg307-client.virtnet: executing check_config_client /mnt/lustre oleg307-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg307-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97b650a15800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97b650a15800.idle_timeout=debug disable quota as required oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:25:19 (1787613919) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8250] == sanity-flr test 0a: lfs mirror create with -N option == 19:25:22 (1787613922) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:25:40 (1787613940) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:25:44 (1787613944) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:25:48 (1787613948) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:26:11 (1787613971) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:26:14 (1787613974) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:26:17 (1787613977) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97b650a15800.stats=clear osc.lustre-OST0001-osc-ffff97b650a15800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:26:25 (1787613985) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:26:36 (1787613996) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:26:44 (1787614004) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:26:48 (1787614008) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:26:56 (1787614016) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 19:27:07 (1787614027) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:27:14 (1787614034) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:27:22 (1787614042) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:27:31 (1787614051) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:27:43 (1787614063) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:27:50 (1787614070) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.57254 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107933 s, 9.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:28:00 (1787614080) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0882827 s, 23.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:28:08 (1787614088) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:28:18 (1787614098) Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:29:33 (1787614173) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.297666 s, 14.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:30:15 (1787614215) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:31:38 (1787614298) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.19021 s, 16.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 19:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.040s user 0m0.003s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.370s user 0m0.000s sys 0m0.188s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.337s user 0m0.000s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg307-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b650a15800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:33:14 (1787614394) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:33:18 (1787614398) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:33:21 (1787614401) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:33:25 (1787614405) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 19:33:34 (1787614414) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.772872 s, 16.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:33:49 (1787614429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216446 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326875 s, 32.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:33:56 (1787614436) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:34:04 (1787614444) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000862578 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000784385 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113479 s, 902 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116595 s, 878 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 19:34:12 (1787614452) create test files with size 15759979 .. 0c3503ddaffb4bbe321be71f436035a1 - 9d133279a38d0f72b5bc0e85987912b4 - 2f886f207aaa54efacc6fc76abd44226 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 19:34:33 (1787614473) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:34:46 (1787614486) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:34:53 (1787614493) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907317 s, 11.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223367 s, 46.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 19:35:01 (1787614501) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0587239 s, 35.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.330708 s, 12.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142055 s, 14.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 19:35:15 (1787614515) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 19:35:18 (1787614518) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:35:20 (1787614520) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406741 s, 25.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 19:35:28 (1787614528) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.375607 s, 8.4 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:35:37 (1787614537) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:35:46 (1787614546) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.251545 s, 41.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:35:56 (1787614556) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff97b643fc2000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324057 s, 32.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff97b660645800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:36:10 (1787614570) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:36:45 (1787614605) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:37:17 (1787614637) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:37:20 (1787614640) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223222 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108192 s, 29.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbe51d4b8 (nosync,stale) Mirror 2: 0xbe51d4b8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0403e763efb10098c98b9e059242ea08 - mirror 1 checksum: b1b3ca3255bf3d820c9514bd2d6b8362 - mirror 2 checksum: 0403e763efb10098c98b9e059242ea08 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T19:37:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T19:37:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0403e763efb10098c98b9e059242ea08 - mirror 1 checksum: b1b3ca3255bf3d820c9514bd2d6b8362 - mirror 2 checksum: b1b3ca3255bf3d820c9514bd2d6b8362 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T19:37:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T19:37:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:37:28 (1787614648) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297462 s, 35.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:37:35 (1787614655) Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server 19:37:45 (1787614665) shut down facet: mds1 facet_host: oleg307-server facet_failover_host: oleg307-server Failover mds1 to oleg307-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 19:38:08 (1787614688) targets are mounted 19:38:08 (1787614688) facet_failover done oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361461 s, 29.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:38:23 (1787614703) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0405355 s, 505 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00981011 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:38:32 (1787614712) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0303039 s, 676 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00599933 s, 3.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:38:42 (1787614722) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0113468 s, 1.4 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:38:48 (1787614728) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0596135 s, 344 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:38:55 (1787614735) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00678102 s, 604 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0318996 s, 128 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:39:01 (1787614741) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:39:19 (1787614759) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:39:45 (1787614785) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:40:09 (1787614809) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:40:32 (1787614832) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:40:39 (1787614839) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57582 mirror split pid 57583 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57582 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57583 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 19:41:46 (1787614906) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4908 skipping zero size read skipping zero size read skipping zero length punch hole truncating to largest ever: 0x21dbf 000004 1787614908.846531 trunc from 00000000 to 0x021dbe (0x21dbf bytes) 000005 1787614908.867650 write 0x01632a thru 0x0232ae (0xcf85 bytes) fallocating to largest ever: 0x32e4c 000007 1787614908.875916 fallocate 0x0242a5 thru 0x032e4b (0xeba7 bytes) 000009 1787614908.883970 mapwrite 0x02503c thru 0x02938f (0x4354 bytes) 000010 1787614908.954730 trunc from 0x00cedc to 0x01b0e8 (0xe20d bytes) 000011 1787614908.962740 mapread 0x01cff9 thru 0x02938f (0xc397 bytes) 000013 1787614908.967063 trunc from 0x01ecf7 to 0x026967 (0x7c71 bytes) truncating to largest ever: 0x34669 000014 1787614908.972893 trunc from 0x029390 to 0x034668 (0xb2d9 bytes) 000015 1787614908.992476 write 0x035fe4 thru 0x03ffff (0xa01c bytes) 000016 1787614908.996471 write 0x03d078 thru 0x03ffff (0x2f88 bytes) fallocating to largest ever: 0x40000 000017 1787614908.999744 fallocate 0x038489 thru 0x03ffff (0x7b77 bytes) 000018 1787614909.011975 fallocate 0x0120e1 thru 0x0129b7 (0x8d7 bytes) 000019 1787614909.028928 fallocate 0x030af3 thru 0x03ff16 (0xf424 bytes) 000021 1787614909.039797 punch from 0x03a039 to 0x03ffff (0x5fc7 bytes) 000022 1787614909.056296 trunc from 0x040000 to 0x0154cc (0x2ab33 bytes) 000024 1787614909.084638 punch from 0x0010d7 to 0x00b2cc (0xa1f6 bytes) 000025 1787614909.090626 mapread 0x00dd0b thru 0x0154cc (0x77c2 bytes) 000028 1787614909.106923 punch from 0x001104 to 0x010d82 (0xfc7f bytes) 000029 1787614909.112634 read 0x004a91 thru 0x00c00c (0x757c bytes) 000030 1787614909.124391 write 0x005a85 thru 0x01206f (0xc5eb bytes) 000031 1787614909.126057 read 0x00bcd9 thru 0x00ed7e (0x30a6 bytes) 000032 1787614909.127782 trunc from 0x000f4b to 0x00b731 (0xa7e7 bytes) 000033 1787614909.150302 fallocate 0x0392a2 thru 0x03ffff (0x6d5e bytes) 000034 1787614909.156163 trunc from 0x0046e1 to 0x00a38a (0x5caa bytes) 000035 1787614909.161572 mapwrite 0x0148fd thru 0x0183ca (0x3ace bytes) 000036 1787614909.205926 trunc from 0x014a57 to 0x0183ca (0x3974 bytes) 000037 1787614909.217104 MIRROR_EXTEND to 2 mirrors 000038 1787614909.764918 write 0x0112b6 thru 0x015569 (0x42b4 bytes) 000039 1787614909.819307 punch from 0x003a8e to 0x00966f (0x5be2 bytes) 000040 1787614909.823884 trunc from 0x0183cb to 0x02c485 (0x140bb bytes) 000041 1787614909.869693 read 0x017946 thru 0x024373 (0xca2e bytes) 000042 1787614909.879266 fallocate 0x02dbc7 thru 0x039b16 (0xbf50 bytes) 000043 1787614909.884003 trunc from 0x0039f0 to 0x00e7bf (0xadd0 bytes) 000044 1787614909.890469 punch from 0x0164f2 to 0x01927d (0x2d8c bytes) 000045 1787614909.895281 trunc from 0x039b17 to 0x028494 (0x11682 bytes) 000046 1787614909.911596 trunc from 0x02534d to 0x028494 (0x3148 bytes) 000047 1787614909.916965 fallocate 0x025c25 thru 0x02ccc5 (0x70a1 bytes) 000048 1787614909.921849 fallocate 0x02f085 thru 0x036ed1 (0x7e4d bytes) 000049 1787614909.926923 trunc from 0x028495 to 0x0154dd (0x12fb7 bytes) 000050 1787614909.940533 fallocate 0x0133b9 thru 0x0174ac (0x40f4 bytes) 000051 1787614909.946223 punch from 0x008389 to 0x0088f2 (0x56a bytes) 000052 1787614909.950152 trunc from 0x0154de to 0x012f86 (0x2557 bytes) 000053 1787614909.968638 punch from 0x000a8f to 0x002fd5 (0x2547 bytes) 000054 1787614909.973324 trunc from 0x012f87 to 0x02e1e6 (0x1b260 bytes) 000055 1787614909.986515 read 0x002ba8 thru 0x010206 (0xd65f bytes) 000057 1787614910.004631 mapwrite 0x03f234 thru 0x03ffff (0xdcc bytes) 000058 1787614910.042785 read 0x035088 thru 0x03ffff (0xaf78 bytes) 000059 1787614910.043605 fallocate 0x03c6eb thru 0x03ffff (0x3915 bytes) 000060 1787614910.049704 fallocate 0x01f22d thru 0x0293d8 (0xa1ac bytes) 000061 1787614910.054454 write 0x012f25 thru 0x01ed2a (0xbe06 bytes) 000062 1787614910.062422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) 000063 1787614910.341817 read 0x02e37c thru 0x038d81 (0xaa06 bytes) 000064 1787614910.374343 fallocate 0x03cfb6 thru 0x03ffff (0x304a bytes) 000065 1787614910.384882 mapwrite 0x02222d thru 0x0230db (0xeaf bytes) 000066 1787614910.421806 fallocate 0x0345ad thru 0x039f19 (0x596d bytes) 000067 1787614910.433269 MIRROR_EXTEND to 2 mirrors 000068 1787614911.037780 mapread 0x03f646 thru 0x03ffff (0x9ba bytes) 000069 1787614911.055373 punch from 0x002527 to 0x010a04 (0xe4de bytes) 000070 1787614911.087776 read 0x03e8d0 thru 0x03ffff (0x1730 bytes) 000071 1787614911.095130 punch from 0x01ddab to 0x01fac6 (0x1d1c bytes) 000072 1787614911.100287 trunc from 0x0353eb to 0x03cc13 (0x7829 bytes) 000073 1787614911.103865 mapwrite 0x019d8a thru 0x02013f (0x63b6 bytes) 000074 1787614911.150862 punch from 0x03cd5d to 0x03ffff (0x32a3 bytes) 000075 1787614911.159735 trunc from 0x040000 to 0x008fcb (0x37034 bytes) 000076 1787614911.237526 write 0x006c57 thru 0x010785 (0x9b2f bytes) 000077 1787614911.243322 mapwrite 0x021bdf thru 0x025ddf (0x4201 bytes) 000078 1787614911.281478 trunc from 0x015d92 to 0x023c43 (0xdeb2 bytes) 000079 1787614911.287201 trunc from 0x025de0 to 0x02b1aa (0x53cb bytes) 000080 1787614911.297998 trunc from 0x02b1ab to 0x016fd1 (0x141d9 bytes) 000081 1787614911.317017 trunc from 0x013c99 to 0x016fd1 (0x3339 bytes) 000082 1787614911.325910 read 0x0034f4 thru 0x004adc (0x15e9 bytes) 000083 1787614911.334291 write 0x036f8e thru 0x03b73f (0x47b2 bytes) 000084 1787614911.336636 mapwrite 0x035abd thru 0x0387c4 (0x2d08 bytes) 000086 1787614911.356760 read 0x00402a thru 0x00537b (0x1352 bytes) 000087 1787614911.358058 trunc from 0x03b740 to 0x02dc3e (0xdb01 bytes) 000088 1787614911.374891 fallocate 0x016362 thru 0x01c68e (0x632d bytes) 000089 1787614911.384173 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dc3f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dc3f): Mirror 2: 0xce7982e6 (stale) CRC-32 checksum value for chunk [0, 0x2dc3f): Mirror 1: 0x750b6a6f Mirror 3: 0x750b6a6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dc3f: skipped 000090 1787614912.080916 punch from 0x01072e to 0x01820d (0x7ae0 bytes) 000091 1787614912.117116 write 0x03f505 thru 0x03ffff (0xafb bytes) 000092 1787614912.127217 fallocate 0x01361b thru 0x01df66 (0xa94c bytes) 000094 1787614912.141444 read 0x011343 thru 0x0192cf (0x7f8d bytes) 000095 1787614912.161156 mapwrite 0x018d02 thru 0x0207ff (0x7afe bytes) 000096 1787614912.230795 write 0x03480c thru 0x03df66 (0x975b bytes) 000097 1787614912.244797 mapwrite 0x011c6c thru 0x01627a (0x460f bytes) 000098 1787614912.263431 fallocate 0x01ee6a thru 0x028646 (0x97dd bytes) 000099 1787614912.278587 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x923d98ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x923d98ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000100 1787614912.564698 trunc from 0x040000 to 0x022a4a (0x1d5b5 bytes) 000101 1787614912.604915 trunc from 0x00b832 to 0x0119cb (0x619a bytes) 000102 1787614912.609726 fallocate 0x0096ca thru 0x00c60a (0x2f41 bytes) 000103 1787614912.613987 punch from 0x01606a to 0x01bf60 (0x5ef7 bytes) 000104 1787614912.618314 read 0x007ee4 thru 0x00e4d3 (0x65f0 bytes) 000105 1787614912.634309 mapread 0x01fd0c thru 0x022a4a (0x2d3f bytes) 000106 1787614912.651866 write 0x03c06a thru 0x03ffff (0x3f96 bytes) 000107 1787614912.653562 write 0x015d26 thru 0x01acc6 (0x4fa1 bytes) 000108 1787614912.658717 mapwrite 0x00ca42 thru 0x00f02d (0x25ec bytes) 000109 1787614912.699145 mapread 0x00d62f thru 0x01bff9 (0xe9cb bytes) 000110 1787614912.700190 write 0x01c74e thru 0x0265d1 (0x9e84 bytes) 000111 1787614912.705507 fallocate 0x004d91 thru 0x00f919 (0xab89 bytes) 000112 1787614912.711450 punch from 0x02036c to 0x02c786 (0xc41b bytes) 000113 1787614912.729475 fallocate 0x008dbd thru 0x012439 (0x967d bytes) 000114 1787614912.735610 punch from 0x02eefd to 0x02ef4b (0x4f bytes) 000115 1787614912.744734 mapwrite 0x03ee24 thru 0x03f23e (0x41b bytes) 000116 1787614912.765007 read 0x02c332 thru 0x031681 (0x5350 bytes) 000117 1787614912.778409 trunc from 0x040000 to 0x036112 (0x9eed bytes) 000118 1787614912.800202 trunc from 0x036113 to 0x00ec44 (0x274ce bytes) 000119 1787614912.830782 mapwrite 0x036b8b thru 0x03e91f (0x7d95 bytes) 000120 1787614913.011820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e920): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e920): Mirror 2: 0xa170bae9 (stale) CRC-32 checksum value for chunk [0, 0x3e920): Mirror 1: 0x5dcd0fa8 Mirror 3: 0x5dcd0fa8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e920: skipped 000121 1787614913.829316 mapwrite 0x014232 thru 0x01531f (0x10ee bytes) 000122 1787614913.928287 mapwrite 0x00b41f thru 0x00daaf (0x2691 bytes) 000123 1787614913.957283 trunc from 0x02a5ce to 0x02cad0 (0x2503 bytes) 000124 1787614913.961639 mapwrite 0x0320c0 thru 0x03847e (0x63bf bytes) 000126 1787614913.993961 trunc from 0x03e920 to 0x003597 (0x3b388 bytes) 000127 1787614914.027844 mapwrite 0x034621 thru 0x037095 (0x2a75 bytes) 000128 1787614914.121807 mapread 0x020bab thru 0x028d7f (0x81d5 bytes) 000129 1787614914.157699 MIRROR_RESYNC 3 mirrors 000130 1787614914.895300 fallocate 0x025fab thru 0x02f2c2 (0x9318 bytes) 000131 1787614914.929973 read 0x022beb thru 0x02bbae (0x8fc4 bytes) 000132 1787614914.945875 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37096): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37096): Mirror 2: 0x50b04c82 (stale) Mirror 3: 0x50b04c82 (stale) CRC-32 checksum value for chunk [0, 0x37096): Mirror 1: 0x50b04c82 Mirror 4: 0x50b04c82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37096: skipped 000133 1787614915.619506 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37096): Mirror 2: 0x50b04c82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37096): Mirror 2: 0x50b04c82 (stale) CRC-32 checksum value for chunk [0, 0x37096): Mirror 1: 0x50b04c82 Mirror 4: 0x50b04c82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37096: skipped 000134 1787614916.331141 trunc from 0x037096 to 0x0015c8 (0x35acd bytes) 000135 1787614916.408275 mapread 0x00056b thru 0x0015c8 (0x105e bytes) 000136 1787614916.452724 mapwrite 0x032383 thru 0x03e45f (0xc0dd bytes) 000137 1787614916.558180 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e460): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 2: 0xa48b1326 (stale) Mirror 4: 0xa48b1326 (stale) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 1: 0xd9c34b53 Mirror 5: 0xd9c34b53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000138 1787614917.339492 mapread 0x0266ce thru 0x02c85d (0x6190 bytes) 000139 1787614917.378334 mapread 0x020282 thru 0x02b61f (0xb39e bytes) 000140 1787614917.409615 write 0x03799f thru 0x03d2b3 (0x5915 bytes) 000141 1787614917.466606 punch from 0x026e22 to 0x02bd5d (0x4f3c bytes) 000142 1787614917.473859 mapread 0x031953 thru 0x03e45f (0xcb0d bytes) 000143 1787614917.494396 fallocate 0x02b77d thru 0x03a78a (0xf00e bytes) 000144 1787614917.503706 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: 0xa48b1326 (stale) Mirror 5: 0xd9c34b53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: 0xa48b1326 (stale) Mirror 5: 0xd9c34b53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000145 1787614917.988123 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: 0xa48b1326 (stale) Mirror 5: 0xd9c34b53 (stale) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 1: 0x8839e45d Mirror 6: 0x8839e45d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000146 1787614918.970208 trunc from 0x03d240 to 0x03e45f (0x1220 bytes) 000147 1787614919.022283 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 4: 0xa48b1326 (stale) Mirror 5: 0xd9c34b53 (stale) Mirror 6: 0x8839e45d (stale) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 1: 0x13b7a6df Mirror 7: 0x13b7a6df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000148 1787614919.977504 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 5: 0xd9c34b53 (stale) Mirror 6: 0x8839e45d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 5: 0xd9c34b53 (stale) Mirror 6: 0x8839e45d (stale) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 1: 0x13b7a6df Mirror 7: 0x13b7a6df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000149 1787614920.706411 punch from 0x00749a to 0x0133da (0xbf41 bytes) 000151 1787614920.791892 read 0x02acd5 thru 0x02cda6 (0x20d2 bytes) 000153 1787614920.805188 punch from 0x00cdaa to 0x00ffd0 (0x3227 bytes) 000154 1787614920.815910 mapread 0x018d36 thru 0x023836 (0xab01 bytes) 000155 1787614920.862644 mapread 0x016335 thru 0x01ea7d (0x8749 bytes) 000156 1787614920.863749 fallocate 0x037bb0 thru 0x03c67d (0x4ace bytes) 000157 1787614920.888406 MIRROR_RESYNC 4 mirrors 000160 1787614921.825206 read 0x008926 thru 0x01792f (0xf00a bytes) 000161 1787614921.847617 fallocate 0x004109 thru 0x007e7b (0x3d73 bytes) 000162 1787614921.900865 read 0x02e3e3 thru 0x03a599 (0xc1b7 bytes) 000163 1787614921.922425 punch from 0x0231d9 to 0x02ad8a (0x7bb2 bytes) 000164 1787614921.929570 mapread 0x01b582 thru 0x021d29 (0x67a8 bytes) 000165 1787614921.931564 write 0x01e900 thru 0x0261de (0x78df bytes) 000166 1787614921.947629 trunc from 0x000b5d to 0x01089d (0xfd41 bytes) 000167 1787614921.967400 mapwrite 0x00cff7 thru 0x018878 (0xb882 bytes) 000168 1787614922.040446 mapread 0x021416 thru 0x02a194 (0x8d7f bytes) 000169 1787614922.060330 fallocate 0x004543 thru 0x0110d2 (0xcb90 bytes) 000170 1787614922.085284 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e460): Mirror 6: 0x13b7a6df (stale) Mirror 7: 0x13b7a6df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e460): Mirror 6: 0x13b7a6df (stale) Mirror 7: 0x13b7a6df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e460: skipped 000171 1787614922.403374 mapread 0x0176cc thru 0x01ddc7 (0x66fc bytes) 000172 1787614922.435742 mapwrite 0x030c5a thru 0x038205 (0x75ac bytes) 000173 1787614922.459164 write 0x01b501 thru 0x025297 (0x9d97 bytes) 000174 1787614922.468002 punch from 0x02a277 to 0x035349 (0xb0d3 bytes) 000175 1787614922.478820 read 0x01440d thru 0x01b50f (0x7103 bytes) 000176 1787614922.479556 trunc from 0x02db84 to 0x030da3 (0x3220 bytes) 000177 1787614922.485702 trunc from 0x03e460 to 0x03c113 (0x234c bytes) 000178 1787614922.496768 fallocate 0x03907e thru 0x03dfe9 (0x4f6c bytes) 000179 1787614922.504448 read 0x00473b thru 0x011ccf (0xd595 bytes) 000180 1787614922.516804 trunc from 0x03c114 to 0x036c22 (0x54f1 bytes) 000181 1787614922.531383 fallocate 0x01b043 thru 0x024ad0 (0x9a8e bytes) 000182 1787614922.536729 write 0x0183f3 thru 0x01ab9c (0x27aa bytes) 000183 1787614922.539696 punch from 0x00a899 to 0x016f6b (0xc6d3 bytes) 000184 1787614922.548812 trunc from 0x030bb4 to 0x032590 (0x19dd bytes) 000186 1787614922.553918 mapwrite 0x020cae thru 0x029480 (0x87d3 bytes) 000187 1787614922.569532 fallocate 0x003d1d thru 0x012ced (0xefd1 bytes) 000188 1787614922.574784 trunc from 0x036c23 to 0x018a81 (0x1e1a1 bytes) 000189 1787614922.592692 trunc from 0x018a82 to 0x017269 (0x1818 bytes) 000190 1787614922.610385 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1726a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1726a): Mirror 6: 0x9f8a6f4b (stale) Mirror 7: 0x9f8a6f4b (stale) CRC-32 checksum value for chunk [0, 0x1726a): Mirror 1: 0x4cb3fe0d Mirror 8: 0x4cb3fe0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1726a: skipped 000191 1787614923.515500 mapwrite 0x0110c6 thru 0x018b47 (0x7a82 bytes) 000192 1787614923.690598 mapread 0x00250e thru 0x00fd55 (0xd848 bytes) 000193 1787614923.711610 trunc from 0x018b48 to 0x01b6a4 (0x2b5d bytes) 000194 1787614923.744643 punch from 0x01808e to 0x01b6a4 (0x3617 bytes) 000195 1787614923.751205 read 0x009e60 thru 0x019c83 (0xfe24 bytes) 000196 1787614923.756969 fallocate 0x037aed thru 0x03ffff (0x8513 bytes) 000197 1787614923.764827 write 0x000d52 thru 0x00fa1d (0xeccc bytes) 000198 1787614923.776181 trunc from 0x03f439 to 0x03ffff (0xbc7 bytes) 000199 1787614923.792531 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf2db146b (stale) Mirror 7: 0xf2db146b (stale) Mirror 8: 0xafd9fc23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15d2a17 Mirror 9: 0x15d2a17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1787614924.707376 read 0x00e9fa thru 0x01bf2d (0xd534 bytes) 000201 1787614924.740411 MIRROR_RESYNC 5 mirrors 000202 1787614925.536313 mapread 0x0363dd thru 0x03ffff (0x9c23 bytes) 000203 1787614925.580483 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) 000204 1787614926.086699 fallocate 0x037c8d thru 0x03e21f (0x6593 bytes) 000205 1787614926.142229 trunc from 0x01676d to 0x016f62 (0x7f6 bytes) 000206 1787614926.154436 trunc from 0x0213e9 to 0x0263f2 (0x500a bytes) 000207 1787614926.167636 fallocate 0x01c01e thru 0x026ca6 (0xac89 bytes) 000208 1787614926.172582 fallocate 0x00168a thru 0x002f95 (0x190c bytes) 000209 1787614926.178214 punch from 0x01d210 to 0x02c9f1 (0xf7e2 bytes) 000210 1787614926.183557 trunc from 0x02869d to 0x0315a8 (0x8f0c bytes) 000211 1787614926.186986 punch from 0x03ca3f to 0x03ffff (0x35c1 bytes) 000212 1787614926.191288 mapwrite 0x0251c7 thru 0x028212 (0x304c bytes) 000213 1787614926.220042 trunc from 0x040000 to 0x03b364 (0x4c9b bytes) 000214 1787614926.243322 punch from 0x02149c to 0x0218fc (0x461 bytes) 000215 1787614926.247534 fallocate 0x02493e thru 0x02b80c (0x6ecf bytes) 000216 1787614926.251607 read 0x01e29d thru 0x0276ad (0x9411 bytes) 000217 1787614926.258570 punch from 0x02dd31 to 0x0369b3 (0x8c83 bytes) 000218 1787614926.262356 mapwrite 0x02dc2d thru 0x034d00 (0x70d4 bytes) 000219 1787614926.284234 mapwrite 0x0356af thru 0x035afc (0x44e bytes) 000220 1787614926.294380 fallocate 0x00fa50 thru 0x017639 (0x7bea bytes) 000221 1787614926.298593 trunc from 0x01b1a1 to 0x02359e (0x83fe bytes) 000222 1787614926.302726 fallocate 0x02bd27 thru 0x034852 (0x8b2c bytes) 000223 1787614926.306845 mapwrite 0x01faf6 thru 0x024b11 (0x501c bytes) 000224 1787614926.334716 trunc from 0x03b365 to 0x01b46f (0x1fef5 bytes) 000225 1787614926.349697 trunc from 0x0173f7 to 0x01b46f (0x4079 bytes) 000226 1787614926.355588 trunc from 0x01b470 to 0x0374e5 (0x1c076 bytes) 000228 1787614926.372731 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374e6): Mirror 7: 0x751bef14 (stale) Mirror 8: 0x751bef14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374e6): Mirror 7: 0x751bef14 (stale) Mirror 8: 0x751bef14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374e6: skipped 000230 1787614926.818245 trunc from 0x0374e6 to 0x001d11 (0x357d4 bytes) 000231 1787614926.859684 fallocate 0x029ce8 thru 0x039a2e (0xfd47 bytes) 000232 1787614926.873838 mapread 0x036e09 thru 0x039a2e (0x2c26 bytes) 000233 1787614926.895626 trunc from 0x039a2f to 0x008ed2 (0x30b5c bytes) 000234 1787614926.911512 mapwrite 0x00359d thru 0x0038f3 (0x357 bytes) 000235 1787614926.932697 read 0x002d49 thru 0x008ed2 (0x618a bytes) 000236 1787614926.933770 mapread 0x007dde thru 0x008ed2 (0x10f5 bytes) 000237 1787614926.935306 trunc from 0x0069da to 0x008ed2 (0x24f9 bytes) 000238 1787614926.941883 trunc from 0x007d0c to 0x008ed2 (0x11c7 bytes) 000239 1787614926.946569 write 0x0345d7 thru 0x03dc6f (0x9699 bytes) 000240 1787614926.949828 read 0x021428 thru 0x024064 (0x2c3d bytes) 000241 1787614926.975802 trunc from 0x0166ee to 0x021241 (0xab54 bytes) 000242 1787614926.990146 mapwrite 0x00a93d thru 0x00c7a3 (0x1e67 bytes) 000244 1787614927.033842 mapwrite 0x01bdfe thru 0x01c967 (0xb6a bytes) 000245 1787614927.066113 mapread 0x02d048 thru 0x03b155 (0xe10e bytes) 000246 1787614927.068860 mapread 0x0158bc thru 0x0243c7 (0xeb0c bytes) 000247 1787614927.137907 punch from 0x00089d to 0x00cf5b (0xc6bf bytes) 000248 1787614927.150552 mapread 0x0264a3 thru 0x02b3e2 (0x4f40 bytes) 000249 1787614927.151793 fallocate 0x02feea thru 0x03446f (0x4586 bytes) 000250 1787614927.157077 fallocate 0x01f6e7 thru 0x02dbee (0xe508 bytes) 000251 1787614927.166531 trunc from 0x03dc70 to 0x032b6f (0xb100 bytes) 000252 1787614927.184451 fallocate 0x02f744 thru 0x0338cf (0x418c bytes) 000253 1787614927.189692 punch from 0x00ab56 to 0x015759 (0xac04 bytes) 000254 1787614927.198409 trunc from 0x0338d0 to 0x000604 (0x332cb bytes) 000255 1787614927.220606 punch from 0x00006c to 0x000604 (0x599 bytes) 000256 1787614927.238877 read 0x000503 thru 0x000604 (0x102 bytes) 000257 1787614927.248537 read 0x00042d thru 0x000604 (0x1d8 bytes) 000258 1787614927.265303 MIRROR_RESYNC 3 mirrors 000259 1787614927.967862 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) 000260 1787614928.349885 punch from 0x000548 to 0x000604 (0xbd bytes) 000261 1787614928.416845 mapread 0x000232 thru 0x000604 (0x3d3 bytes) 000262 1787614928.429587 trunc from 0x000605 to 0x00e500 (0xdefc bytes) 000263 1787614928.491395 fallocate 0x032ba1 thru 0x03ffff (0xd45f bytes) 000264 1787614928.498394 punch from 0x008fdb to 0x014cee (0xbd14 bytes) 000265 1787614928.510378 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 8: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000266 1787614929.341102 mapread 0x008567 thru 0x00bfe4 (0x3a7e bytes) 000267 1787614929.397535 trunc from 0x004838 to 0x00a32b (0x5af4 bytes) 000268 1787614929.450861 trunc from 0x040000 to 0x01eab0 (0x2154f bytes) 000269 1787614929.483810 mapwrite 0x00fe5f thru 0x010421 (0x5c3 bytes) 000270 1787614929.509650 read 0x0014d4 thru 0x00f34a (0xde77 bytes) 000271 1787614929.521192 mapread 0x0161c2 thru 0x01a139 (0x3f78 bytes) 000272 1787614929.530738 fallocate 0x022e0a thru 0x029294 (0x648b bytes) 000273 1787614929.535793 fallocate 0x022222 thru 0x02dda2 (0xbb81 bytes) 000274 1787614929.542381 trunc from 0x0077f5 to 0x011182 (0x998e bytes) 000275 1787614929.547429 read 0x0123ea thru 0x016fc9 (0x4be0 bytes) 000276 1787614929.548330 mapwrite 0x024aba thru 0x026931 (0x1e78 bytes) 000277 1787614929.590880 trunc from 0x006456 to 0x009bbc (0x3767 bytes) 000278 1787614929.599217 trunc from 0x0257f2 to 0x0265c7 (0xdd6 bytes) 000279 1787614929.614187 mapread 0x013c0e thru 0x01cdcc (0x91bf bytes) 000280 1787614929.615097 mapread 0x0028f7 thru 0x00e078 (0xb782 bytes) 000281 1787614929.647403 trunc from 0x026932 to 0x01ee3b (0x7af6 bytes) 000282 1787614929.671849 trunc from 0x007e3c to 0x0141e0 (0xc3a5 bytes) 000283 1787614929.677642 read 0x016a6c thru 0x01856c (0x1b01 bytes) 000284 1787614929.678630 mapread 0x00cbb1 thru 0x019098 (0xc4e8 bytes) 000286 1787614929.694478 read 0x003ecb thru 0x005ad3 (0x1c09 bytes) 000287 1787614929.695705 read 0x00bf8f thru 0x018b7b (0xcbed bytes) 000289 1787614929.699261 mapwrite 0x03be4b thru 0x03ffff (0x41b5 bytes) 000290 1787614929.742765 mapwrite 0x024e33 thru 0x026d09 (0x1ed7 bytes) 000291 1787614929.755582 fallocate 0x031839 thru 0x036f8a (0x5752 bytes) 000292 1787614929.764830 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d8eb20e Mirror 9: 0x8d8eb20e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1787614930.773005 read 0x02fed4 thru 0x03e210 (0xe33d bytes) 000294 1787614930.786246 write 0x019cca thru 0x024ab6 (0xaded bytes) 000295 1787614930.837222 read 0x02cc5d thru 0x038fcb (0xc36f bytes) 000296 1787614930.848540 mapread 0x02dce0 thru 0x03bfb1 (0xe2d2 bytes) 000297 1787614930.850199 trunc from 0x040000 to 0x02c672 (0x1398d bytes) 000298 1787614930.869900 trunc from 0x004a89 to 0x013461 (0xe9d9 bytes) 000299 1787614930.875745 mapwrite 0x00a775 thru 0x013552 (0x8dde bytes) 000300 1787614930.934424 mapread 0x01f280 thru 0x02c03f (0xcdc0 bytes) 000301 1787614930.935553 mapwrite 0x038fc8 thru 0x03f791 (0x67ca bytes) 000302 1787614931.003821 mapwrite 0x00d19f thru 0x0123a2 (0x5204 bytes) 000303 1787614931.021926 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f792): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f792): Mirror 7: 0x73caa562 (stale) Mirror 8: 0x73caa562 (stale) Mirror 9: 0x494fc1f7 (stale) CRC-32 checksum value for chunk [0, 0x3f792): Mirror 1: 0x1717fa87 Mirror 10: 0x1717fa87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f792: skipped 000304 1787614932.071310 trunc from 0x025e55 to 0x0319da (0xbb86 bytes) 000305 1787614932.121905 fallocate 0x00f3de thru 0x012877 (0x349a bytes) 000306 1787614932.129092 write 0x02a1ea thru 0x032a43 (0x885a bytes) 000307 1787614932.148203 trunc from 0x01ca9f to 0x0221e9 (0x574b bytes) 000309 1787614932.164663 fallocate 0x025fde thru 0x027fbb (0x1fde bytes) 000310 1787614932.174168 punch from 0x026091 to 0x0283f2 (0x2362 bytes) 000311 1787614932.185613 fallocate 0x01c0de thru 0x02bf1b (0xfe3e bytes) 000312 1787614932.200126 mapread 0x03dddd thru 0x03f459 (0x167d bytes) 000313 1787614932.233643 mapread 0x025c87 thru 0x02d434 (0x77ae bytes) 000314 1787614932.250514 trunc from 0x03f792 to 0x031314 (0xe47d bytes) 000315 1787614932.318932 mapwrite 0x01c52d thru 0x029d7f (0xd853 bytes) 000317 1787614932.358155 trunc from 0x01d60c to 0x0244e7 (0x6edc bytes) 000318 1787614932.363429 write 0x0324a4 thru 0x0345cb (0x2128 bytes) 000319 1787614932.369463 MIRROR_RESYNC 5 mirrors 000320 1787614933.880703 trunc from 0x00085c to 0x00c687 (0xbe2c bytes) 000321 1787614933.927464 mapread 0x008b6c thru 0x0136f8 (0xab8d bytes) 000322 1787614933.969210 fallocate 0x00e622 thru 0x01954c (0xaf2b bytes) 000323 1787614933.974093 write 0x032bf7 thru 0x0391f0 (0x65fa bytes) 000324 1787614933.980156 punch from 0x027b5f to 0x02a9b0 (0x2e52 bytes) 000325 1787614933.984513 fallocate 0x01f0b5 thru 0x027491 (0x83dd bytes) 000326 1787614933.989046 trunc from 0x02677e to 0x030f61 (0xa7e4 bytes) 000327 1787614933.993972 trunc from 0x0271ed to 0x030a56 (0x986a bytes) 000328 1787614934.001521 fallocate 0x0149b2 thru 0x01fab1 (0xb100 bytes) 000329 1787614934.008316 read 0x008354 thru 0x0108a3 (0x8550 bytes) 000330 1787614934.012472 mapwrite 0x03d15a thru 0x03ffff (0x2ea6 bytes) 000332 1787614934.066468 read 0x030af8 thru 0x039302 (0x880b bytes) 000333 1787614934.075470 punch from 0x037dfe to 0x03bb6b (0x3d6e bytes) 000335 1787614934.080752 punch from 0x017818 to 0x026bf3 (0xf3dc bytes) 000336 1787614934.089156 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3f0b4dce (stale) Mirror 8: 0x3f0b4dce (stale) Mirror 9: 0x3f0b4dce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3f0b4dce (stale) Mirror 8: 0x3f0b4dce (stale) Mirror 9: 0x3f0b4dce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000338 1787614934.402396 mapread 0x000d6a thru 0x000f77 (0x20e bytes) 000339 1787614934.407931 mapread 0x017bd9 thru 0x02288c (0xacb4 bytes) 000340 1787614934.434351 trunc from 0x006d36 to 0x016177 (0xf442 bytes) 000341 1787614934.438598 trunc from 0x01896d to 0x0233f3 (0xaa87 bytes) 000342 1787614934.442596 trunc from 0x040000 to 0x02a0f1 (0x15f0e bytes) 000343 1787614934.459017 mapwrite 0x028668 thru 0x028915 (0x2ae bytes) 000346 1787614934.471410 punch from 0x00817c to 0x012de3 (0xac68 bytes) 000347 1787614934.478696 fallocate 0x02db58 thru 0x035cac (0x8155 bytes) 000348 1787614934.482698 trunc from 0x0164be to 0x016e93 (0x9d6 bytes) 000349 1787614934.486666 punch from 0x00020c to 0x00b13f (0xaf34 bytes) 000350 1787614934.492383 trunc from 0x02a0f2 to 0x03965f (0xf56e bytes) 000352 1787614934.508187 fallocate 0x019c61 thru 0x024e53 (0xb1f3 bytes) 000354 1787614934.517764 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39660): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39660): Mirror 7: 0x130d5ba5 (stale) Mirror 8: 0x130d5ba5 (stale) Mirror 9: 0x130d5ba5 (stale) CRC-32 checksum value for chunk [0, 0x39660): Mirror 1: 0x7a46194b Mirror 10: 0x7a46194b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39660: skipped 000355 1787614935.173623 punch from 0x02acc4 to 0x0386c0 (0xd9fd bytes) 000356 1787614935.249809 read 0x0031dc thru 0x00da45 (0xa86a bytes) 000357 1787614935.273222 trunc from 0x028cfb to 0x02e7c8 (0x5ace bytes) 000358 1787614935.283168 mapwrite 0x01ab49 thru 0x0289b9 (0xde71 bytes) 000359 1787614935.371433 punch from 0x03715e to 0x03965f (0x2502 bytes) 000360 1787614935.377806 mapread 0x02b36f thru 0x036add (0xb76f bytes) 000361 1787614935.396754 trunc from 0x037ac4 to 0x03965f (0x1b9c bytes) 000362 1787614935.405626 mapread 0x013c2b thru 0x018fe1 (0x53b7 bytes) 000363 1787614935.421656 trunc from 0x0356b5 to 0x03965f (0x3fab bytes) 000364 1787614935.427830 trunc from 0x00b08f to 0x0181b6 (0xd128 bytes) 000365 1787614935.443781 write 0x005ef7 thru 0x00f0e6 (0x91f0 bytes) 000366 1787614935.467334 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39660): Mirror 7: 0x130d5ba5 (stale) Mirror 8: 0x130d5ba5 (stale) Mirror 10: 0x7a46194b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39660): Mirror 7: 0x130d5ba5 (stale) Mirror 8: 0x130d5ba5 (stale) Mirror 10: 0x7a46194b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39660: skipped 000367 1787614935.960314 trunc from 0x039660 to 0x037847 (0x1e18 bytes) 000368 1787614936.019310 read 0x02fbc2 thru 0x0307b4 (0xbf3 bytes) 000369 1787614936.025502 fallocate 0x00e8eb thru 0x017ef0 (0x9606 bytes) 000370 1787614936.031343 mapwrite 0x039a8f thru 0x03dd53 (0x42c5 bytes) 000371 1787614936.085557 write 0x00fef0 thru 0x01e356 (0xe467 bytes) 000372 1787614936.094732 punch from 0x007c68 to 0x008b91 (0xf2a bytes) 000373 1787614936.098451 read 0x02852a thru 0x030f70 (0x8a47 bytes) 000375 1787614936.107016 mapwrite 0x030352 thru 0x03e92f (0xe5de bytes) 000376 1787614936.164499 MIRROR_RESYNC 4 mirrors 000377 1787614937.167723 write 0x019e21 thru 0x01b5bc (0x179c bytes) 000378 1787614937.215329 write 0x010e9d thru 0x01a48e (0x95f2 bytes) 000379 1787614937.221518 trunc from 0x03e930 to 0x004023 (0x3a90c bytes) 000380 1787614937.266623 mapread 0x000471 thru 0x000808 (0x398 bytes) 000381 1787614937.273115 trunc from 0x000928 to 0x004023 (0x36fc bytes) 000382 1787614937.277821 mapwrite 0x02e860 thru 0x03d1ae (0xe94f bytes) 000383 1787614937.344097 mapread 0x038dab thru 0x03d1ae (0x4404 bytes) 000384 1787614937.345125 mapwrite 0x032f66 thru 0x03ce88 (0x9f23 bytes) 000385 1787614937.365810 punch from 0x039631 to 0x03d1ae (0x3b7e bytes) 000386 1787614937.376621 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d1af): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1af): Mirror 7: 0xbf6c6c19 (stale) Mirror 8: 0xbf6c6c19 (stale) Mirror 10: 0xbf6c6c19 (stale) CRC-32 checksum value for chunk [0, 0x3d1af): Mirror 1: 0x232fb9b9 Mirror 11: 0x232fb9b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1af: skipped 000387 1787614938.397887 trunc from 0x00f9ff to 0x01beff (0xc501 bytes) 000388 1787614938.451468 punch from 0x000bed to 0x00ff6e (0xf382 bytes) 000389 1787614938.459246 trunc from 0x011bd9 to 0x016358 (0x4780 bytes) 000390 1787614938.463553 fallocate 0x000abd thru 0x00fe51 (0xf395 bytes) 000392 1787614938.470558 mapwrite 0x01e3e3 thru 0x024352 (0x5f70 bytes) 000393 1787614938.533270 trunc from 0x03b45e to 0x03d1ae (0x1d51 bytes) 000394 1787614938.541397 write 0x0253f2 thru 0x02ac01 (0x5810 bytes) 000395 1787614938.551562 read 0x028dc6 thru 0x029a5b (0xc96 bytes) 000396 1787614938.552291 write 0x01251b thru 0x020a3e (0xe524 bytes) 000397 1787614938.562125 write 0x033e6f thru 0x03ffff (0xc191 bytes) 000398 1787614938.568502 trunc from 0x01a361 to 0x024f44 (0xabe4 bytes) 000399 1787614938.594070 mapwrite 0x018001 thru 0x01af02 (0x2f02 bytes) 000400 1787614938.619646 fallocate 0x00b609 thru 0x0176f4 (0xc0ec bytes) 000402 1787614938.626656 punch from 0x038b0c to 0x03e2e0 (0x57d5 bytes) 000403 1787614938.632237 trunc from 0x008f43 to 0x0100d4 (0x7192 bytes) 000404 1787614938.636637 trunc from 0x0000b8 to 0x00781e (0x7767 bytes) 000406 1787614938.645914 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbb2752b3 (stale) Mirror 10: 0xbb2752b3 (stale) Mirror 11: 0xfba39cdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbb2752b3 (stale) Mirror 10: 0xbb2752b3 (stale) Mirror 11: 0xfba39cdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000407 1787614939.087728 punch from 0x02ab8a to 0x02c600 (0x1a77 bytes) 000408 1787614939.093813 mapwrite 0x032904 thru 0x0344f2 (0x1bef bytes) 000409 1787614939.127578 fallocate 0x02b340 thru 0x037ee1 (0xcba2 bytes) 000410 1787614939.132894 read 0x016481 thru 0x017a51 (0x15d1 bytes) 000411 1787614939.140887 mapwrite 0x02e1e8 thru 0x0369dc (0x87f5 bytes) 000412 1787614939.159642 mapread 0x01f595 thru 0x02a451 (0xaebd bytes) 000413 1787614939.179238 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbb2752b3 (stale) Mirror 10: 0xbb2752b3 (stale) Mirror 11: 0xfba39cdc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xefc51951 Mirror 12: 0xefc51951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000414 1787614940.058271 mapwrite 0x021ad3 thru 0x02ff4e (0xe47c bytes) 000415 1787614940.197613 read 0x030d57 thru 0x03f7f3 (0xea9d bytes) 000417 1787614940.229688 MIRROR_RESYNC 5 mirrors 000418 1787614941.988752 mapread 0x034c29 thru 0x03cf4b (0x8323 bytes) 000419 1787614942.006296 trunc from 0x0114cb to 0x0119a6 (0x4dc bytes) 000420 1787614942.049032 trunc from 0x040000 to 0x01904f (0x26fb0 bytes) 000421 1787614942.085132 mapwrite 0x00089c thru 0x0083bd (0x7b22 bytes) 000422 1787614942.117204 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19050): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19050): Mirror 7: 0x44406f11 (stale) Mirror 10: 0x44406f11 (stale) Mirror 11: 0x44406f11 (stale) Mirror 12: 0x44406f11 (stale) CRC-32 checksum value for chunk [0, 0x19050): Mirror 1: 0x953b2781 Mirror 13: 0x953b2781 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19050: skipped 000423 1787614942.959219 mapread 0x0015a6 thru 0x00f5fd (0xe058 bytes) 000424 1787614943.049730 trunc from 0x017f46 to 0x01869c (0x757 bytes) 000425 1787614943.084802 write 0x008c46 thru 0x018bc3 (0xff7e bytes) 000426 1787614943.099011 punch from 0x0140e3 to 0x015288 (0x11a6 bytes) 000427 1787614943.117388 mapwrite 0x02a371 thru 0x036d4a (0xc9da bytes) 000428 1787614943.192426 punch from 0x0118a4 to 0x01d48b (0xbbe8 bytes) 000429 1787614943.199597 fallocate 0x019fd6 thru 0x0235f0 (0x961b bytes) 000430 1787614943.204117 write 0x027e6b thru 0x02f255 (0x73eb bytes) 000431 1787614943.207174 trunc from 0x036d4b to 0x0005c7 (0x36783 bytes) 000432 1787614943.221819 mapread 0x00052c thru 0x0005c7 (0x9c bytes) 000433 1787614943.229821 punch from 0x00058c to 0x0005c7 (0x3c bytes) 000434 1787614943.234747 fallocate 0x02ca66 thru 0x02ef41 (0x24dc bytes) 000435 1787614943.249822 mapwrite 0x011266 thru 0x012022 (0xdbd bytes) 000436 1787614943.285071 mapread 0x005c7e thru 0x00f82c (0x9baf bytes) 000437 1787614943.292214 trunc from 0x0035db to 0x003bd5 (0x5fb bytes) 000438 1787614943.296130 mapread 0x000105 thru 0x000694 (0x590 bytes) 000439 1787614943.302839 trunc from 0x012023 to 0x00415f (0xdec3 bytes) 000440 1787614943.317742 fallocate 0x0009f6 thru 0x00ae64 (0xa46f bytes) 000441 1787614943.322745 fallocate 0x0237c3 thru 0x02a62b (0x6e69 bytes) 000442 1787614943.327554 fallocate 0x024c60 thru 0x0293f2 (0x4793 bytes) 000443 1787614943.331715 punch from 0x00ce93 to 0x01a342 (0xd4b0 bytes) 000444 1787614943.335904 trunc from 0x019625 to 0x01a9b1 (0x138d bytes) 000445 1787614943.340480 read 0x009af2 thru 0x0162bb (0xc7ca bytes) 000446 1787614943.348769 read 0x0200a9 thru 0x027290 (0x71e8 bytes) 000447 1787614943.357104 mapread 0x020500 thru 0x0272ef (0x6df0 bytes) 000448 1787614943.358350 read 0x004fd8 thru 0x00bcb6 (0x6cdf bytes) 000449 1787614943.359130 trunc from 0x0293f3 to 0x00532c (0x240c6 bytes) 000450 1787614943.379734 MIRROR_RESYNC 6 mirrors 000451 1787614944.344790 trunc from 0x00532d to 0x011a45 (0xc719 bytes) 000452 1787614944.413746 mapread 0x00c316 thru 0x00ca63 (0x74e bytes) 000453 1787614944.419401 mapread 0x00e5f9 thru 0x011a45 (0x344d bytes) 000454 1787614944.432565 fallocate 0x0249e3 thru 0x024bf9 (0x217 bytes) 000455 1787614944.436786 write 0x03f697 thru 0x03ffff (0x969 bytes) 000456 1787614944.439008 write 0x003943 thru 0x006a41 (0x30ff bytes) 000458 1787614944.452344 mapwrite 0x023ad2 thru 0x032fcc (0xf4fb bytes) 000459 1787614944.486810 read 0x0339fb thru 0x036d5f (0x3365 bytes) 000460 1787614944.493298 mapwrite 0x005316 thru 0x00f439 (0xa124 bytes) 000461 1787614944.527821 mapwrite 0x0178f3 thru 0x021c60 (0xa36e bytes) 000462 1787614944.551722 read 0x01bd39 thru 0x0258c2 (0x9b8a bytes) 000463 1787614944.556739 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000464 1787614944.938364 trunc from 0x03b694 to 0x03ffff (0x496c bytes) 000465 1787614944.943277 trunc from 0x027074 to 0x02b616 (0x45a3 bytes) 000466 1787614944.947709 mapread 0x034d84 thru 0x03c14f (0x73cc bytes) 000469 1787614944.969918 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000470 1787614945.362215 read 0x008f49 thru 0x010241 (0x72f9 bytes) 000472 1787614945.390452 trunc from 0x03b9ae to 0x03ffff (0x4652 bytes) 000473 1787614945.396441 mapread 0x01832b thru 0x01b200 (0x2ed6 bytes) 000474 1787614945.425654 punch from 0x027bbc to 0x0379af (0xfdf4 bytes) 000475 1787614945.440005 fallocate 0x01b7bb thru 0x02a706 (0xef4c bytes) 000476 1787614945.445398 mapread 0x03ef29 thru 0x03ffff (0x10d7 bytes) 000477 1787614945.456559 mapwrite 0x009ae0 thru 0x0101bb (0x66dc bytes) 000478 1787614945.471854 mapwrite 0x00ead6 thru 0x01adf1 (0xc31c bytes) 000479 1787614945.501448 fallocate 0x007db2 thru 0x0154c6 (0xd715 bytes) 000480 1787614945.506025 read 0x01fd8d thru 0x02b47d (0xb6f1 bytes) 000482 1787614945.525449 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000483 1787614945.838647 fallocate 0x033754 thru 0x03da7f (0xa32c bytes) 000485 1787614945.847868 mapread 0x03b277 thru 0x03ffff (0x4d89 bytes) 000486 1787614945.880325 mapread 0x02630f thru 0x035445 (0xf137 bytes) 000487 1787614945.903055 punch from 0x03e2b0 to 0x03ffff (0x1d50 bytes) 000489 1787614945.907168 mapwrite 0x005616 thru 0x00ba99 (0x6484 bytes) 000490 1787614945.936550 punch from 0x0343d7 to 0x03facd (0xb6f7 bytes) 000491 1787614945.941120 read 0x02011c thru 0x02645c (0x6341 bytes) 000492 1787614945.941806 write 0x03704d thru 0x03ffff (0x8fb3 bytes) 000493 1787614945.951759 fallocate 0x015515 thru 0x01ef62 (0x9a4e bytes) 000494 1787614945.956036 read 0x022167 thru 0x022cb6 (0xb50 bytes) 000495 1787614945.957157 write 0x03d60d thru 0x03ffff (0x29f3 bytes) 000496 1787614945.958334 fallocate 0x025df4 thru 0x02ebe1 (0x8dee bytes) 000498 1787614945.966117 mapread 0x02093a thru 0x02c962 (0xc029 bytes) 000499 1787614945.967480 read 0x031735 thru 0x03da1e (0xc2ea bytes) 000500 1787614945.981508 mapwrite 0x00232f thru 0x002fbf (0xc91 bytes) 000501 1787614945.994522 mapread 0x03b4cd thru 0x03ffff (0x4b33 bytes) 000502 1787614945.995470 punch from 0x025e36 to 0x032816 (0xc9e1 bytes) 000503 1787614946.000600 write 0x02f5e3 thru 0x0350ea (0x5b08 bytes) 000504 1787614946.005503 read 0x011224 thru 0x01edec (0xdbc9 bytes) 000505 1787614946.010263 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000506 1787614946.372809 fallocate 0x01c17e thru 0x021922 (0x57a5 bytes) 000507 1787614946.377885 punch from 0x028231 to 0x033cfd (0xbacd bytes) 000508 1787614946.384324 trunc from 0x001339 to 0x003543 (0x220b bytes) 000509 1787614946.389594 trunc from 0x03f177 to 0x03ffff (0xe89 bytes) 000510 1787614946.399592 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) 000512 1787614946.532221 trunc from 0x019370 to 0x026337 (0xcfc8 bytes) 000513 1787614946.540457 read 0x008f46 thru 0x018eb3 (0xff6e bytes) 000515 1787614946.564844 punch from 0x03eb7b to 0x03ffff (0x1485 bytes) 000516 1787614946.572478 trunc from 0x03801e to 0x03b425 (0x3408 bytes) 000517 1787614946.581334 punch from 0x01a657 to 0x025fa7 (0xb951 bytes) 000518 1787614946.585406 mapwrite 0x022805 thru 0x02895a (0x6156 bytes) 000519 1787614946.656344 read 0x035431 thru 0x03cf72 (0x7b42 bytes) 000521 1787614946.689523 write 0x00ed74 thru 0x01d632 (0xe8bf bytes) 000523 1787614946.701213 punch from 0x01bc3e to 0x02a27a (0xe63d bytes) 000524 1787614946.734185 trunc from 0x040000 to 0x034b22 (0xb4dd bytes) 000525 1787614946.753748 trunc from 0x034b23 to 0x022798 (0x1238a bytes) 000527 1787614946.776506 mapread 0x011965 thru 0x01b766 (0x9e02 bytes) 000528 1787614946.782834 trunc from 0x00898e to 0x00e32f (0x59a2 bytes) 000529 1787614946.790494 trunc from 0x022799 to 0x003531 (0x1f267 bytes) 000530 1787614946.814121 write 0x02af98 thru 0x0396b0 (0xe719 bytes) 000531 1787614946.816435 mapread 0x025b15 thru 0x02657c (0xa68 bytes) 000532 1787614946.827164 mapwrite 0x002b22 thru 0x012757 (0xfc36 bytes) 000533 1787614946.930545 mapread 0x025675 thru 0x02764b (0x1fd7 bytes) 000534 1787614946.938102 punch from 0x012079 to 0x014b63 (0x2aeb bytes) 000535 1787614946.945097 mapwrite 0x00483f thru 0x00d9c6 (0x9188 bytes) 000537 1787614946.965458 punch from 0x024309 to 0x02f069 (0xad61 bytes) 000538 1787614946.970711 trunc from 0x02a3e9 to 0x038335 (0xdf4d bytes) 000539 1787614946.978762 write 0x001bd8 thru 0x00e2c1 (0xc6ea bytes) 000540 1787614946.982398 fallocate 0x033382 thru 0x034653 (0x12d2 bytes) 000541 1787614946.995107 trunc from 0x0396b1 to 0x02fb4e (0x9b62 bytes) 000542 1787614947.021819 mapwrite 0x010c7d thru 0x0126fa (0x1a7e bytes) 000543 1787614947.050567 MIRROR_EXTEND to 2 mirrors 000544 1787614947.720171 trunc from 0x00c71b to 0x010c29 (0x450f bytes) 000545 1787614947.775918 trunc from 0x02fb4f to 0x009b08 (0x26046 bytes) 000546 1787614947.850179 write 0x022a5a thru 0x02bfb7 (0x955e bytes) 000547 1787614947.852106 mapwrite 0x013135 thru 0x014ba1 (0x1a6d bytes) 000548 1787614947.878761 punch from 0x0037a4 to 0x00a477 (0x6cd4 bytes) 000549 1787614947.884351 mapwrite 0x02ba8c thru 0x0367f8 (0xad6d bytes) 000550 1787614947.966256 punch from 0x00b3cb to 0x018f5b (0xdb91 bytes) 000551 1787614947.976207 fallocate 0x0269f5 thru 0x034811 (0xde1d bytes) 000553 1787614947.981369 mapwrite 0x00c121 thru 0x018c98 (0xcb78 bytes) 000554 1787614948.032853 read 0x000c03 thru 0x00e198 (0xd596 bytes) 000556 1787614948.044238 trunc from 0x0367f9 to 0x028e3a (0xd9be bytes) 000558 1787614948.074675 MIRROR_RESYNC 2 mirrors 000559 1787614948.809276 fallocate 0x024224 thru 0x0288d1 (0x46ae bytes) 000560 1787614948.850210 write 0x0224ac thru 0x02fa72 (0xd5c7 bytes) 000561 1787614948.860083 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2fa73): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa73): Mirror 2: 0x32accae7 (stale) CRC-32 checksum value for chunk [0, 0x2fa73): Mirror 1: 0x84b4d7b7 Mirror 3: 0x84b4d7b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa73: skipped 000562 1787614949.653318 mapread 0x01202e thru 0x012040 (0x13 bytes) 000564 1787614949.688201 trunc from 0x00fec2 to 0x01c57e (0xc6bd bytes) 000567 1787614949.736262 mapread 0x02e93e thru 0x02fa72 (0x1135 bytes) 000568 1787614949.755456 punch from 0x0038f4 to 0x00ddb4 (0xa4c1 bytes) 000569 1787614949.762151 read 0x01ad92 thru 0x01e1dd (0x344c bytes) 000571 1787614949.771746 read 0x02aa9b thru 0x02fa72 (0x4fd8 bytes) 000572 1787614949.775757 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fa73): Mirror 3: 0x84b4d7b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa73): Mirror 3: 0x84b4d7b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa73: skipped 000573 1787614950.056990 mapread 0x004131 thru 0x009251 (0x5121 bytes) 000574 1787614950.077557 read 0x0250f1 thru 0x02b7c2 (0x66d2 bytes) 000575 1787614950.084889 fallocate 0x01acb3 thru 0x01d05c (0x23aa bytes) 000576 1787614950.089541 write 0x01c24c thru 0x02710b (0xaec0 bytes) 000577 1787614950.099332 fallocate 0x0273ad thru 0x0369ed (0xf641 bytes) 000578 1787614950.105698 mapread 0x02e985 thru 0x0369ed (0x8069 bytes) 000579 1787614950.116094 mapwrite 0x019ce8 thru 0x0241bc (0xa4d5 bytes) 000580 1787614950.151644 mapread 0x034308 thru 0x0369ed (0x26e6 bytes) 000581 1787614950.152508 trunc from 0x0369ee to 0x013fa7 (0x22a46 bytes) 000582 1787614950.180117 mapwrite 0x03e64a thru 0x03ffff (0x19b6 bytes) 000583 1787614950.257765 fallocate 0x022c23 thru 0x024bdf (0x1fbd bytes) 000584 1787614950.272426 write 0x00bfb9 thru 0x01b397 (0xf3df bytes) 000585 1787614950.283096 trunc from 0x020c19 to 0x02d90c (0xccf4 bytes) 000586 1787614950.288054 mapread 0x0040e5 thru 0x009648 (0x5564 bytes) 000587 1787614950.288981 write 0x036f3b thru 0x03da3d (0x6b03 bytes) 000588 1787614950.293633 read 0x011fef thru 0x01a39e (0x83b0 bytes) 000589 1787614950.298797 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf05ba546 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x77a3c831 Mirror 4: 0x77a3c831 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000590 1787614951.286027 punch from 0x02b6b3 to 0x03a714 (0xf062 bytes) 000591 1787614951.354191 fallocate 0x00088f thru 0x009bb5 (0x9327 bytes) 000592 1787614951.360725 mapread 0x025f06 thru 0x0344d4 (0xe5cf bytes) 000593 1787614951.422696 fallocate 0x00824c thru 0x0149ef (0xc7a4 bytes) 000594 1787614951.430446 fallocate 0x00ddde thru 0x012464 (0x4687 bytes) 000595 1787614951.444472 read 0x00e9ba thru 0x010bc5 (0x220c bytes) 000596 1787614951.459454 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf05ba546 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf05ba546 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1787614951.753313 trunc from 0x040000 to 0x027e9b (0x18164 bytes) 000598 1787614951.793453 trunc from 0x01fcbd to 0x027e9b (0x81df bytes) 000599 1787614951.797039 trunc from 0x01d628 to 0x027e9b (0xa874 bytes) 000600 1787614951.801684 punch from 0x01b533 to 0x01bccf (0x79d bytes) 000601 1787614951.810141 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27e9c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27e9c): Mirror 3: 0xefa1fe15 (stale) CRC-32 checksum value for chunk [0, 0x27e9c): Mirror 1: 0x92dedf3 Mirror 4: 0x92dedf3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27e9c: skipped 000602 1787614952.755613 punch from 0x019272 to 0x022136 (0x8ec5 bytes) 000603 1787614952.805843 trunc from 0x027e9c to 0x036416 (0xe57b bytes) 000604 1787614952.847099 read 0x03525f thru 0x036416 (0x11b8 bytes) 000606 1787614952.854706 fallocate 0x031724 thru 0x03cab7 (0xb394 bytes) 000607 1787614952.869794 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000608 1787614953.202183 punch from 0x022f69 to 0x0323ea (0xf482 bytes) 000609 1787614953.217861 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) CRC-32 checksum value for chunk [0, 0x36417): Mirror 1: 0xb0c183cb Mirror 4: 0xb0c183cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000610 1787614954.189362 read 0x001ac0 thru 0x00b110 (0x9651 bytes) 000611 1787614954.205616 read 0x034f49 thru 0x036416 (0x14ce bytes) 000612 1787614954.221348 read 0x027029 thru 0x02a3a6 (0x337e bytes) 000613 1787614954.228387 read 0x01dd0a thru 0x02da1f (0xfd16 bytes) 000614 1787614954.236283 read 0x023a2e thru 0x031d30 (0xe303 bytes) 000615 1787614954.245712 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) CRC-32 checksum value for chunk [0, 0x36417): Mirror 1: 0xb0c183cb Mirror 4: 0xb0c183cb Mirror 5: 0xb0c183cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000616 1787614955.552502 mapwrite 0x026ca2 thru 0x02e311 (0x7670 bytes) 000617 1787614955.652392 read 0x0138ec thru 0x023875 (0xff8a bytes) 000618 1787614955.678138 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) Mirror 5: 0xb0c183cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) Mirror 5: 0xb0c183cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000619 1787614955.989118 mapwrite 0x008f2c thru 0x00ba95 (0x2b6a bytes) 000620 1787614956.041364 write 0x026188 thru 0x0271ee (0x1067 bytes) 000621 1787614956.055520 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) Mirror 5: 0xb0c183cb (stale) CRC-32 checksum value for chunk [0, 0x36417): Mirror 1: 0x64630bda Mirror 6: 0x64630bda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000622 1787614956.921547 trunc from 0x00fbfb to 0x011df0 (0x21f6 bytes) 000623 1787614956.977081 trunc from 0x01eac0 to 0x02653d (0x7a7e bytes) 000624 1787614956.984560 mapwrite 0x0176c8 thru 0x01e593 (0x6ecc bytes) 000625 1787614957.024752 trunc from 0x01f0b0 to 0x028cef (0x9c40 bytes) 000626 1787614957.030643 read 0x0094ce thru 0x013e52 (0xa985 bytes) 000627 1787614957.045987 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36417): Mirror 3: 0xf2c8f7ca (stale) Mirror 5: 0xb0c183cb (stale) Mirror 6: 0x64630bda (stale) CRC-32 checksum value for chunk [0, 0x36417): Mirror 1: 0xef3e2c83 Mirror 7: 0xef3e2c83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36417: skipped 000628 1787614958.273183 write 0x033b41 thru 0x03a8b2 (0x6d72 bytes) 000630 1787614958.326659 trunc from 0x03a8b3 to 0x00543a (0x35478 bytes) 000631 1787614958.361114 fallocate 0x011384 thru 0x012350 (0xfcd bytes) 000632 1787614958.365771 trunc from 0x000770 to 0x00144b (0xcdc bytes) 000633 1787614958.377763 write 0x01af69 thru 0x01d54e (0x25e6 bytes) 000634 1787614958.378896 read 0x00be01 thru 0x00e478 (0x2678 bytes) 000635 1787614958.385155 punch from 0x006517 to 0x014ec8 (0xe9b2 bytes) 000636 1787614958.389979 trunc from 0x014f51 to 0x016651 (0x1701 bytes) 000637 1787614958.421532 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d54f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d54f): Mirror 3: 0x92fda90a (stale) Mirror 5: 0xdce9637c (stale) Mirror 6: 0xe885c974 (stale) Mirror 7: 0x4b541097 (stale) CRC-32 checksum value for chunk [0, 0x1d54f): Mirror 1: 0xb0c33468 Mirror 8: 0xb0c33468 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d54f: skipped 000638 1787614959.359493 trunc from 0x00545f to 0x00c460 (0x7002 bytes) 000639 1787614959.477247 mapwrite 0x02be54 thru 0x037771 (0xb91e bytes) 000640 1787614959.626761 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37772): Mirror 5: 0xde0e7f32 (stale) Mirror 6: 0x6418aa56 (stale) Mirror 7: 0x548762d0 (stale) Mirror 8: 0xbce329a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37772): Mirror 5: 0xde0e7f32 (stale) Mirror 6: 0x6418aa56 (stale) Mirror 7: 0x548762d0 (stale) Mirror 8: 0xbce329a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37772: skipped 000641 1787614960.026815 fallocate 0x007062 thru 0x00cb4b (0x5aea bytes) 000642 1787614960.035414 fallocate 0x0115dc thru 0x0156d7 (0x40fc bytes) 000643 1787614960.039415 read 0x009abc thru 0x00cdbe (0x3303 bytes) 000644 1787614960.049242 mapread 0x00995f thru 0x019376 (0xfa18 bytes) 000645 1787614960.077769 mapwrite 0x00da9b thru 0x0149b3 (0x6f19 bytes) 000646 1787614960.105388 trunc from 0x037772 to 0x02a0f4 (0xd67d bytes) 000647 1787614960.129894 write 0x008f54 thru 0x00faa7 (0x6b54 bytes) 000648 1787614960.132194 write 0x00a5c1 thru 0x012126 (0x7b66 bytes) 000649 1787614960.138318 MIRROR_RESYNC 5 mirrors 000650 1787614961.488021 trunc from 0x02a0f5 to 0x02a429 (0x335 bytes) 000651 1787614961.532710 mapwrite 0x008b86 thru 0x008fa4 (0x41f bytes) 000652 1787614961.552606 mapread 0x020070 thru 0x02a429 (0xa3ba bytes) 000653 1787614961.582642 punch from 0x026603 to 0x02a429 (0x3e27 bytes) 000654 1787614961.591629 trunc from 0x028771 to 0x02a429 (0x1cb9 bytes) 000656 1787614961.601388 fallocate 0x0278ba thru 0x03064f (0x8d96 bytes) 000657 1787614961.608283 trunc from 0x015f77 to 0x01a4d2 (0x455c bytes) 000658 1787614961.615565 write 0x0314c1 thru 0x03c310 (0xae50 bytes) 000659 1787614961.618004 mapwrite 0x009ad5 thru 0x014e7b (0xb3a7 bytes) 000660 1787614961.652642 read 0x00b5d0 thru 0x014ccb (0x96fc bytes) 000662 1787614961.653774 mapwrite 0x031cd3 thru 0x037d92 (0x60c0 bytes) 000663 1787614961.701019 punch from 0x022cc1 to 0x02deac (0xb1ec bytes) 000664 1787614961.712992 punch from 0x00eeb1 to 0x01e6f1 (0xf841 bytes) 000665 1787614961.723261 fallocate 0x01f339 thru 0x0299d6 (0xa69e bytes) 000666 1787614961.732803 punch from 0x03114f to 0x037a96 (0x6948 bytes) 000667 1787614961.740409 mapread 0x00b9a6 thru 0x00fc04 (0x425f bytes) 000668 1787614961.750187 fallocate 0x00359d thru 0x011f30 (0xe994 bytes) 000669 1787614961.754495 trunc from 0x02b520 to 0x03a98b (0xf46c bytes) 000670 1787614961.759780 fallocate 0x009f8f thru 0x0116b9 (0x772b bytes) 000671 1787614961.768644 punch from 0x01ddf5 to 0x01e8aa (0xab6 bytes) 000672 1787614961.779987 fallocate 0x030ab7 thru 0x03952d (0x8a77 bytes) 000673 1787614961.791222 trunc from 0x03c311 to 0x013077 (0x29299 bytes) 000674 1787614961.818472 mapwrite 0x027cfc thru 0x031652 (0x9957 bytes) 000675 1787614961.923066 trunc from 0x006556 to 0x011d7c (0xb827 bytes) 000676 1787614961.930252 fallocate 0x01b3d7 thru 0x023deb (0x8a15 bytes) 000677 1787614961.944019 mapwrite 0x012668 thru 0x01a02f (0x79c8 bytes) 000678 1787614961.979334 mapread 0x023445 thru 0x02cba1 (0x975d bytes) 000680 1787614961.982071 mapread 0x02bb98 thru 0x02f0e8 (0x3551 bytes) 000681 1787614961.983110 trunc from 0x019413 to 0x019f3d (0xb2b bytes) 000682 1787614961.988282 mapread 0x00e581 thru 0x00ef34 (0x9b4 bytes) 000683 1787614961.991066 fallocate 0x00bae2 thru 0x0118dd (0x5dfc bytes) 000684 1787614961.998399 trunc from 0x0103d5 to 0x01222a (0x1e56 bytes) 000685 1787614962.014887 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31653): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31653): Mirror 5: 0x6e6807a0 (stale) Mirror 6: 0x6e6807a0 (stale) Mirror 7: 0x6e6807a0 (stale) Mirror 8: 0x6e6807a0 (stale) CRC-32 checksum value for chunk [0, 0x31653): Mirror 1: 0x6ac136f9 Mirror 9: 0x6ac136f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31653: skipped 000686 1787614963.081322 write 0x021774 thru 0x028b8c (0x7419 bytes) 000687 1787614963.150346 trunc from 0x030274 to 0x031652 (0x13df bytes) 000688 1787614963.155885 read 0x010c91 thru 0x0134ca (0x283a bytes) 000689 1787614963.161823 punch from 0x030fd1 to 0x031652 (0x682 bytes) 000690 1787614963.167395 fallocate 0x01aa28 thru 0x028e1d (0xe3f6 bytes) 000692 1787614963.172294 write 0x01c052 thru 0x029504 (0xd4b3 bytes) 000693 1787614963.182551 trunc from 0x0307bc to 0x031652 (0xe97 bytes) 000694 1787614963.187198 fallocate 0x015a18 thru 0x022a64 (0xd04d bytes) 000695 1787614963.192089 trunc from 0x02aab3 to 0x02b407 (0x955 bytes) 000696 1787614963.196611 read 0x0007e4 thru 0x008bc8 (0x83e5 bytes) 000697 1787614963.209553 trunc from 0x02a40a to 0x031652 (0x7249 bytes) 000699 1787614963.215561 mapwrite 0x017167 thru 0x01a76b (0x3605 bytes) 000701 1787614963.231382 trunc from 0x031653 to 0x00db23 (0x23b2f bytes) 000702 1787614963.250269 trunc from 0x00a367 to 0x00db23 (0x37bd bytes) 000703 1787614963.253986 trunc from 0x00764c to 0x00db23 (0x64d8 bytes) 000704 1787614963.258309 read 0x0099b1 thru 0x00db23 (0x4173 bytes) 000705 1787614963.266299 mapwrite 0x01c07c thru 0x01f1d6 (0x315b bytes) 000707 1787614963.301070 punch from 0x019537 to 0x01f1d6 (0x5ca0 bytes) 000708 1787614963.305847 trunc from 0x01f1d7 to 0x02ce96 (0xdcc0 bytes) 000709 1787614963.335613 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ce97): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce97): Mirror 5: 0xc79d6737 (stale) Mirror 6: 0xc79d6737 (stale) Mirror 7: 0xc79d6737 (stale) Mirror 8: 0xc79d6737 (stale) Mirror 9: 0xb6cbe403 (stale) CRC-32 checksum value for chunk [0, 0x2ce97): Mirror 1: 0x299ed728 Mirror 10: 0x299ed728 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce97: skipped 000710 1787614964.129721 mapwrite 0x0295c1 thru 0x0325e1 (0x9021 bytes) 000711 1787614964.256639 fallocate 0x012b3e thru 0x01f2b3 (0xc776 bytes) 000712 1787614964.263203 write 0x02fd47 thru 0x03ed0c (0xefc6 bytes) 000713 1787614964.266245 read 0x00d61b thru 0x01b036 (0xda1c bytes) 000714 1787614964.274726 read 0x0281ce thru 0x0318e6 (0x9719 bytes) 000715 1787614964.275453 mapwrite 0x020f62 thru 0x024922 (0x39c1 bytes) 000716 1787614964.312261 read 0x013198 thru 0x01d4a5 (0xa30e bytes) 000717 1787614964.314198 trunc from 0x03ed0d to 0x01f2d3 (0x1fa39 bytes) 000718 1787614964.338866 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f2d4): Mirror 6: 0xcda12b58 (stale) Mirror 7: 0xcda12b58 (stale) Mirror 8: 0xcda12b58 (stale) Mirror 9: 0xfc09af37 (stale) Mirror 10: 0x92061a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2d4): Mirror 6: 0xcda12b58 (stale) Mirror 7: 0xcda12b58 (stale) Mirror 8: 0xcda12b58 (stale) Mirror 9: 0xfc09af37 (stale) Mirror 10: 0x92061a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2d4: skipped 000719 1787614964.739329 read 0x004951 thru 0x008d28 (0x43d8 bytes) 000720 1787614964.750410 trunc from 0x01f2d4 to 0x00d4e4 (0x11def bytes) 000721 1787614964.765354 write 0x034770 thru 0x03891d (0x41ae bytes) 000722 1787614964.766701 read 0x014b1f thru 0x015422 (0x904 bytes) 000723 1787614964.773106 fallocate 0x00bd2f thru 0x00f79e (0x3a70 bytes) 000724 1787614964.779568 trunc from 0x016409 to 0x023498 (0xd090 bytes) 000725 1787614964.785639 trunc from 0x0326d7 to 0x03891d (0x6247 bytes) 000726 1787614964.809441 read 0x037963 thru 0x03891d (0xfbb bytes) 000727 1787614964.830191 trunc from 0x03891e to 0x0233aa (0x15573 bytes) 000729 1787614964.857350 read 0x00a0d6 thru 0x0146cb (0xa5f6 bytes) 000730 1787614964.858309 read 0x01eb16 thru 0x0233aa (0x4895 bytes) 000731 1787614964.859199 fallocate 0x0322ec thru 0x03ee8a (0xcb9f bytes) 000732 1787614964.863882 write 0x002a6d thru 0x00b13e (0x86d2 bytes) 000733 1787614964.865553 write 0x010f13 thru 0x015ac9 (0x4bb7 bytes) 000735 1787614964.868343 trunc from 0x010c89 to 0x01a6ff (0x9a77 bytes) 000736 1787614964.882929 fallocate 0x00407d thru 0x00eaae (0xaa32 bytes) 000737 1787614964.897719 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x233ab): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233ab): Mirror 6: 0xe2ea63c (stale) Mirror 7: 0xe2ea63c (stale) Mirror 8: 0xe2ea63c (stale) Mirror 9: 0x731991cf (stale) Mirror 10: 0xc2e5ee67 (stale) CRC-32 checksum value for chunk [0, 0x233ab): Mirror 1: 0xf84564e9 Mirror 11: 0xf84564e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233ab: skipped 000738 1787614965.998349 read 0x010c2c thru 0x01c751 (0xbb26 bytes) 000739 1787614966.015386 trunc from 0x0233ab to 0x03b315 (0x17f6b bytes) 000740 1787614966.083899 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b316): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b316): Mirror 6: 0xbd8a6610 (stale) Mirror 7: 0xbd8a6610 (stale) Mirror 8: 0xbd8a6610 (stale) Mirror 9: 0x3908edf6 (stale) Mirror 10: 0x2411e6d6 (stale) Mirror 11: 0xf59b3b8f (stale) CRC-32 checksum value for chunk [0, 0x3b316): Mirror 1: 0xf59b3b8f Mirror 12: 0xf59b3b8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b316: skipped 000741 1787614967.205416 fallocate 0x01b990 thru 0x0252ec (0x995d bytes) 000742 1787614967.283881 write 0x01e7b4 thru 0x0212dc (0x2b29 bytes) 000743 1787614967.322877 punch from 0x0088af to 0x00ad4e (0x24a0 bytes) 000744 1787614967.333063 fallocate 0x010fed thru 0x01eaa1 (0xdab5 bytes) 000745 1787614967.350267 MIRROR_RESYNC 8 mirrors 000747 1787614969.663925 fallocate 0x01554d thru 0x0200ee (0xaba2 bytes) 000749 1787614969.695389 mapread 0x00478b thru 0x01417c (0xf9f2 bytes) 000750 1787614969.718441 write 0x03ac50 thru 0x03ffff (0x53b0 bytes) 000751 1787614969.722608 fallocate 0x03a622 thru 0x03ffff (0x59de bytes) 000752 1787614969.727624 fallocate 0x03232c thru 0x035969 (0x363e bytes) 000753 1787614969.731451 trunc from 0x03491f to 0x034d32 (0x414 bytes) 000754 1787614969.735577 read 0x0236ae thru 0x03041c (0xcd6f bytes) 000755 1787614969.745517 punch from 0x03c6a0 to 0x03db54 (0x14b5 bytes) 000756 1787614969.758054 punch from 0x00eddf to 0x017c7f (0x8ea1 bytes) 000757 1787614969.762493 punch from 0x01d9d7 to 0x0280b1 (0xa6db bytes) 000758 1787614969.769021 trunc from 0x040000 to 0x00f4f5 (0x30b0a bytes) 000759 1787614969.790229 trunc from 0x0052d4 to 0x00c9bd (0x76ea bytes) 000760 1787614969.795410 read 0x004726 thru 0x00f4f5 (0xadd0 bytes) 000761 1787614969.803329 mapread 0x000436 thru 0x009662 (0x922d bytes) 000762 1787614969.804326 trunc from 0x00f4f6 to 0x013c55 (0x4760 bytes) 000763 1787614969.817131 trunc from 0x011adc to 0x013c55 (0x217a bytes) 000764 1787614969.822052 mapwrite 0x028f66 thru 0x029666 (0x701 bytes) 000765 1787614969.854852 trunc from 0x002cad to 0x00bc4c (0x8fa0 bytes) 000766 1787614969.860221 write 0x00ac8d thru 0x00d419 (0x278d bytes) 000767 1787614969.871546 mapwrite 0x03caad thru 0x03ffff (0x3553 bytes) 000768 1787614969.916706 write 0x030819 thru 0x036ccf (0x64b7 bytes) 000769 1787614969.923047 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5589aab7 (stale) Mirror 7: 0x5589aab7 (stale) Mirror 9: 0x5589aab7 (stale) Mirror 10: 0x5589aab7 (stale) Mirror 11: 0x5589aab7 (stale) Mirror 12: 0x5589aab7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5589aab7 (stale) Mirror 7: 0x5589aab7 (stale) Mirror 9: 0x5589aab7 (stale) Mirror 10: 0x5589aab7 (stale) Mirror 11: 0x5589aab7 (stale) Mirror 12: 0x5589aab7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000770 1787614970.427714 punch from 0x00fd74 to 0x013760 (0x39ed bytes) 000771 1787614970.432714 read 0x01884b thru 0x024fb4 (0xc76a bytes) 000772 1787614970.451338 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5589aab7 (stale) Mirror 7: 0x5589aab7 (stale) Mirror 9: 0x5589aab7 (stale) Mirror 10: 0x5589aab7 (stale) Mirror 11: 0x5589aab7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5589aab7 (stale) Mirror 7: 0x5589aab7 (stale) Mirror 9: 0x5589aab7 (stale) Mirror 10: 0x5589aab7 (stale) Mirror 11: 0x5589aab7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1787614970.894432 read 0x023f1c thru 0x02d3de (0x94c3 bytes) 000774 1787614970.903288 mapread 0x03ac3b thru 0x03ffff (0x53c5 bytes) 000775 1787614970.921398 write 0x038357 thru 0x03e2c8 (0x5f72 bytes) 000776 1787614970.926566 punch from 0x03254d to 0x033809 (0x12bd bytes) 000777 1787614970.935810 MIRROR_RESYNC 6 mirrors 000778 1787614972.885563 fallocate 0x00be6c thru 0x01674b (0xa8e0 bytes) 000779 1787614972.910241 punch from 0x01cd69 to 0x021040 (0x42d8 bytes) 000780 1787614972.926454 write 0x006e57 thru 0x0130d6 (0xc280 bytes) 000781 1787614972.941376 trunc from 0x03c1ea to 0x03ffff (0x3e16 bytes) 000782 1787614972.951766 trunc from 0x040000 to 0x03e765 (0x189a bytes) 000783 1787614972.989104 fallocate 0x00bca7 thru 0x00c790 (0xaea bytes) 000784 1787614972.995323 fallocate 0x000a5c thru 0x003dec (0x3391 bytes) 000785 1787614972.998760 read 0x02585d thru 0x02a82b (0x4fcf bytes) 000787 1787614973.012302 mapwrite 0x01d104 thru 0x028b0c (0xba09 bytes) 000789 1787614973.069330 read 0x020fba thru 0x029422 (0x8469 bytes) 000790 1787614973.070368 read 0x00270b thru 0x00cc35 (0xa52b bytes) 000791 1787614973.082199 write 0x012240 thru 0x01a8e9 (0x86aa bytes) 000792 1787614973.083415 read 0x022496 thru 0x022de4 (0x94f bytes) 000794 1787614973.084371 read 0x02eaed thru 0x030cd7 (0x21eb bytes) 000795 1787614973.084863 mapread 0x023862 thru 0x03087b (0xd01a bytes) 000796 1787614973.086437 mapread 0x01d75a thru 0x021f4c (0x47f3 bytes) 000797 1787614973.087793 fallocate 0x0240d0 thru 0x030539 (0xc46a bytes) 000798 1787614973.091877 fallocate 0x02b28b thru 0x02ef0e (0x3c84 bytes) 000799 1787614973.106534 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e766): Mirror 6: 0x11063e18 (stale) Mirror 9: 0x11063e18 (stale) Mirror 10: 0x11063e18 (stale) Mirror 11: 0x11063e18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e766): Mirror 6: 0x11063e18 (stale) Mirror 9: 0x11063e18 (stale) Mirror 10: 0x11063e18 (stale) Mirror 11: 0x11063e18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e766: skipped 000800 1787614973.529211 read 0x009c17 thru 0x00a82d (0xc17 bytes) 000802 1787614973.540072 fallocate 0x00e9b2 thru 0x011288 (0x28d7 bytes) 000803 1787614973.547194 mapwrite 0x00991d thru 0x014ea6 (0xb58a bytes) 000804 1787614973.602067 write 0x029476 thru 0x02e2d1 (0x4e5c bytes) 000806 1787614973.641399 mapwrite 0x02f3c8 thru 0x0345c8 (0x5201 bytes) 000807 1787614973.676782 punch from 0x0168b1 to 0x02063b (0x9d8b bytes) 000808 1787614973.693898 read 0x03bd8c thru 0x03e765 (0x29da bytes) 000809 1787614973.709157 trunc from 0x03e766 to 0x02e0d9 (0x1068c bytes) 000810 1787614973.730646 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e0da): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0da): Mirror 6: 0x399eb311 (stale) Mirror 9: 0x399eb311 (stale) Mirror 10: 0x399eb311 (stale) Mirror 11: 0x399eb311 (stale) CRC-32 checksum value for chunk [0, 0x2e0da): Mirror 1: 0x8ea0293b Mirror 12: 0x8ea0293b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0da: skipped 000811 1787614974.472128 write 0x01b561 thru 0x026f3c (0xb9dc bytes) 000812 1787614974.533687 trunc from 0x022dfe to 0x02a151 (0x7354 bytes) 000813 1787614974.553856 read 0x01f1ee thru 0x025167 (0x5f7a bytes) 000814 1787614974.565902 write 0x0264bb thru 0x0329d1 (0xc517 bytes) 000815 1787614974.586871 write 0x03fe2c thru 0x03ffff (0x1d4 bytes) 000816 1787614974.594999 MIRROR_RESYNC 6 mirrors 000817 1787614976.515384 mapwrite 0x027676 thru 0x0300ec (0x8a77 bytes) 000818 1787614976.633634 punch from 0x020f3f to 0x029d31 (0x8df3 bytes) 000819 1787614976.641429 read 0x004015 thru 0x00adf6 (0x6de2 bytes) 000820 1787614976.652720 trunc from 0x040000 to 0x00dbdf (0x32420 bytes) 000821 1787614976.692395 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdbe0): Mirror 9: 0xdb1be824 (stale) Mirror 10: 0xdb1be824 (stale) Mirror 11: 0xdb1be824 (stale) Mirror 12: 0xdb1be824 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdbe0): Mirror 9: 0xdb1be824 (stale) Mirror 10: 0xdb1be824 (stale) Mirror 11: 0xdb1be824 (stale) Mirror 12: 0xdb1be824 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdbe0: skipped 000822 1787614977.123427 trunc from 0x00164a to 0x00dbdf (0xc596 bytes) 000823 1787614977.128428 write 0x02343d thru 0x025e23 (0x29e7 bytes) 000824 1787614977.134650 mapread 0x01805d thru 0x01adce (0x2d72 bytes) 000825 1787614977.154536 punch from 0x00fbd7 to 0x01abd5 (0xafff bytes) 000826 1787614977.162699 trunc from 0x008413 to 0x013cc2 (0xb8b0 bytes) 000829 1787614977.172877 trunc from 0x01d932 to 0x025e23 (0x84f2 bytes) 000830 1787614977.188464 fallocate 0x000b38 thru 0x00222a (0x16f3 bytes) 000831 1787614977.197247 trunc from 0x025e24 to 0x02d120 (0x72fd bytes) 000832 1787614977.239265 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d121): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d121): Mirror 9: 0x9ae35e54 (stale) Mirror 10: 0x9ae35e54 (stale) Mirror 11: 0x9ae35e54 (stale) Mirror 12: 0x9ae35e54 (stale) CRC-32 checksum value for chunk [0, 0x2d121): Mirror 1: 0xd7ae117e Mirror 13: 0xd7ae117e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d121: skipped 000833 1787614978.173667 mapread 0x0244c0 thru 0x02ac01 (0x6742 bytes) 000834 1787614978.202349 fallocate 0x03d6d6 thru 0x03ffff (0x292a bytes) 000835 1787614978.243742 mapread 0x015d21 thru 0x01acd7 (0x4fb7 bytes) 000836 1787614978.258104 MIRROR_RESYNC 6 mirrors 000837 1787614979.413654 trunc from 0x01218e to 0x01efaa (0xce1d bytes) 000838 1787614979.458449 trunc from 0x02cdbb to 0x02d120 (0x366 bytes) 000839 1787614979.463541 fallocate 0x036cb1 thru 0x03ffff (0x934f bytes) 000840 1787614979.470185 read 0x023f38 thru 0x0251fd (0x12c6 bytes) 000841 1787614979.477908 read 0x0092d1 thru 0x00c9b1 (0x36e1 bytes) 000842 1787614979.499439 MIRROR_RESYNC 6 mirrors 000843 1787614980.849515 read 0x018242 thru 0x021204 (0x8fc3 bytes) 000844 1787614980.866129 fallocate 0x005c29 thru 0x011e38 (0xc210 bytes) 000845 1787614980.934205 fallocate 0x03b03f thru 0x03ffff (0x4fc1 bytes) 000846 1787614980.948451 read 0x005325 thru 0x013172 (0xde4e bytes) 000847 1787614980.964033 trunc from 0x02d121 to 0x03591c (0x87fc bytes) 000848 1787614981.005088 MIRROR_RESYNC 6 mirrors 000849 1787614982.228364 MIRROR_RESYNC 6 mirrors 000850 1787614982.665724 MIRROR_RESYNC 6 mirrors 000851 1787614983.033313 MIRROR_EXTEND to 7 mirrors 000852 1787614983.774725 fallocate 0x02abf4 thru 0x038f71 (0xe37e bytes) 000853 1787614983.821295 read 0x00760e thru 0x0144bb (0xceae bytes) 000854 1787614983.832992 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3591d): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3591d): Mirror 9: 0x6839d9bf (stale) Mirror 10: 0x6839d9bf (stale) Mirror 11: 0x6839d9bf (stale) Mirror 12: 0x6839d9bf (stale) Mirror 13: 0x6839d9bf (stale) Mirror 14: 0x6839d9bf (stale) CRC-32 checksum value for chunk [0, 0x3591d): Mirror 1: 0x6839d9bf Mirror 15: 0x6839d9bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3591d: skipped 000855 1787614984.778842 read 0x01c665 thru 0x027034 (0xa9d0 bytes) 000856 1787614984.813818 punch from 0x02ed80 to 0x03591c (0x6b9d bytes) 000857 1787614984.869451 trunc from 0x019822 to 0x01afa7 (0x1786 bytes) 000858 1787614984.880399 read 0x01cd42 thru 0x02c1fb (0xf4ba bytes) 000859 1787614984.897572 fallocate 0x039ccf thru 0x03ffff (0x6331 bytes) 000860 1787614984.906514 punch from 0x01bdf3 to 0x01c6f1 (0x8ff bytes) 000861 1787614984.918351 trunc from 0x03591d to 0x01c1c8 (0x19754 bytes) 000862 1787614984.967413 trunc from 0x01c1c9 to 0x032223 (0x1605b bytes) 000863 1787614984.993736 write 0x035b61 thru 0x03f51b (0x99bb bytes) 000865 1787614985.001032 read 0x02ff55 thru 0x032776 (0x2822 bytes) 000866 1787614985.033814 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: 0x542f130 (stale) Mirror 10: 0x542f130 (stale) Mirror 11: 0x542f130 (stale) Mirror 12: 0x542f130 (stale) Mirror 13: 0x542f130 (stale) Mirror 14: 0x542f130 (stale) Mirror 15: 0x542f130 (stale) CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 1: 0xfe41b8ef Mirror 16: 0xfe41b8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f51c: skipped 000867 1787614986.066601 read 0x0331ca thru 0x038672 (0x54a9 bytes) 000868 1787614986.078125 mapread 0x006e7e thru 0x00764a (0x7cd bytes) 000869 1787614986.099499 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: 0x542f130 (stale) Mirror 10: 0x542f130 (stale) Mirror 11: 0x542f130 (stale) Mirror 12: 0x542f130 (stale) Mirror 13: 0x542f130 (stale) Mirror 14: 0x542f130 (stale) Mirror 15: 0x542f130 (stale) CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 1: 0xfe41b8ef Mirror 16: 0xfe41b8ef Mirror 17: 0xfe41b8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f51c: skipped 000870 1787614987.080113 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 9: 0x542f130 (stale) Mirror 10: 0x542f130 (stale) Mirror 11: 0x542f130 (stale) Mirror 12: 0x542f130 (stale) Mirror 13: 0x542f130 (stale) Mirror 14: 0x542f130 (stale) Mirror 15: 0x542f130 (stale) CRC-32 checksum value for chunk [0, 0x3f51c): Mirror 1: 0xfe41b8ef Mirror 16: 0xfe41b8ef Mirror 17: 0xfe41b8ef Mirror 18: 0xfe41b8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f51c: skipped 000871 1787614988.049861 trunc from 0x03f51c to 0x024e16 (0x1a705 bytes) 000873 1787614988.114306 read 0x0067fa thru 0x00ba54 (0x525b bytes) 000874 1787614988.122157 trunc from 0x009f98 to 0x01973e (0xf7a7 bytes) 000875 1787614988.127472 trunc from 0x024e17 to 0x0200e6 (0x4d30 bytes) 000876 1787614988.152452 trunc from 0x00c10e to 0x019081 (0xcf74 bytes) 000877 1787614988.157235 mapwrite 0x03026f thru 0x03b47c (0xb20e bytes) 000878 1787614988.223113 mapread 0x008e4b thru 0x00b0f5 (0x22ab bytes) 000879 1787614988.230828 fallocate 0x01d2aa thru 0x01d2e6 (0x3d bytes) 000881 1787614988.235554 trunc from 0x006aae to 0x00db0f (0x7062 bytes) 000882 1787614988.244175 trunc from 0x0105a0 to 0x01d3ea (0xce4b bytes) 000883 1787614988.256241 write 0x0164b2 thru 0x01c8c4 (0x6413 bytes) 000884 1787614988.276917 trunc from 0x03b47d to 0x0289b0 (0x12acc bytes) 000885 1787614988.298578 punch from 0x001b9e to 0x00fd06 (0xe169 bytes) 000886 1787614988.306090 punch from 0x009a13 to 0x017a22 (0xe010 bytes) 000887 1787614988.335461 punch from 0x0158c1 to 0x01a294 (0x49d4 bytes) 000888 1787614988.340397 write 0x01caa5 thru 0x01e677 (0x1bd3 bytes) 000889 1787614988.345854 fallocate 0x013a55 thru 0x019ddd (0x6389 bytes) 000890 1787614988.350389 punch from 0x013d0d to 0x0232b7 (0xf5ab bytes) 000891 1787614988.359801 trunc from 0x026ede to 0x0289b0 (0x1ad3 bytes) 000892 1787614988.364491 write 0x035156 thru 0x036407 (0x12b2 bytes) 000893 1787614988.366135 mapread 0x0174ec thru 0x024727 (0xd23c bytes) 000894 1787614988.383891 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36408): Mirror 9: 0xb454c2ee (stale) Mirror 10: 0xb454c2ee (stale) Mirror 11: 0xb454c2ee (stale) Mirror 12: 0xb454c2ee (stale) Mirror 13: 0xb454c2ee (stale) Mirror 14: 0xb454c2ee (stale) Mirror 15: 0xb454c2ee (stale) Mirror 16: 0xa67838c4 (stale) Mirror 17: 0xa67838c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36408): Mirror 9: 0xb454c2ee (stale) Mirror 10: 0xb454c2ee (stale) Mirror 11: 0xb454c2ee (stale) Mirror 12: 0xb454c2ee (stale) Mirror 13: 0xb454c2ee (stale) Mirror 14: 0xb454c2ee (stale) Mirror 15: 0xb454c2ee (stale) Mirror 16: 0xa67838c4 (stale) Mirror 17: 0xa67838c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36408: skipped 000895 1787614988.894435 write 0x03c108 thru 0x03ffff (0x3ef8 bytes) 000896 1787614988.900056 read 0x01bd78 thru 0x01cd4d (0xfd6 bytes) 000897 1787614988.907117 fallocate 0x00374e thru 0x00c1dd (0x8a90 bytes) 000898 1787614988.920301 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0x6843af98 (stale) Mirror 17: 0x6843af98 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b6a39f1 Mirror 18: 0x1b6a39f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000899 1787614990.098796 trunc from 0x040000 to 0x030143 (0xfebc bytes) 000900 1787614990.196292 trunc from 0x023955 to 0x027f30 (0x45dc bytes) 000901 1787614990.201478 punch from 0x02a14e to 0x02f154 (0x5007 bytes) 000902 1787614990.205748 read 0x00d2c1 thru 0x00d8be (0x5fe bytes) 000904 1787614990.212820 mapread 0x0121c4 thru 0x0213de (0xf21b bytes) 000905 1787614990.251381 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30144): Mirror 9: 0x9137c1f7 (stale) Mirror 10: 0x9137c1f7 (stale) Mirror 11: 0x9137c1f7 (stale) Mirror 12: 0x9137c1f7 (stale) Mirror 14: 0x9137c1f7 (stale) Mirror 15: 0x9137c1f7 (stale) Mirror 16: 0x9137c1f7 (stale) Mirror 17: 0x9137c1f7 (stale) Mirror 18: 0x9137c1f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30144): Mirror 9: 0x9137c1f7 (stale) Mirror 10: 0x9137c1f7 (stale) Mirror 11: 0x9137c1f7 (stale) Mirror 12: 0x9137c1f7 (stale) Mirror 14: 0x9137c1f7 (stale) Mirror 15: 0x9137c1f7 (stale) Mirror 16: 0x9137c1f7 (stale) Mirror 17: 0x9137c1f7 (stale) Mirror 18: 0x9137c1f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30144: skipped 000907 1787614990.908235 mapwrite 0x01a029 thru 0x01cb8d (0x2b65 bytes) 000909 1787614990.982264 fallocate 0x01aa5b thru 0x0270aa (0xc650 bytes) 000910 1787614990.995826 mapread 0x00d15a thru 0x014488 (0x732f bytes) 000911 1787614991.040715 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30144): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30144): Mirror 9: 0x9137c1f7 (stale) Mirror 10: 0x9137c1f7 (stale) Mirror 11: 0x9137c1f7 (stale) Mirror 12: 0x9137c1f7 (stale) Mirror 14: 0x9137c1f7 (stale) Mirror 15: 0x9137c1f7 (stale) Mirror 16: 0x9137c1f7 (stale) Mirror 17: 0x9137c1f7 (stale) Mirror 18: 0x9137c1f7 (stale) CRC-32 checksum value for chunk [0, 0x30144): Mirror 1: 0xf663b3f5 Mirror 19: 0xf663b3f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30144: skipped 000912 1787614992.325928 write 0x035bf9 thru 0x038765 (0x2b6d bytes) 000913 1787614992.412402 write 0x03f307 thru 0x03ffff (0xcf9 bytes) 000914 1787614992.413633 fallocate 0x024add thru 0x0329a6 (0xdeca bytes) 000915 1787614992.436256 mapread 0x0119bc thru 0x013359 (0x199e bytes) 000916 1787614992.460884 mapwrite 0x015b90 thru 0x020030 (0xa4a1 bytes) 000917 1787614992.492518 trunc from 0x040000 to 0x010f40 (0x2f0bf bytes) 000918 1787614992.520250 fallocate 0x024f9a thru 0x0321be (0xd225 bytes) 000919 1787614992.529376 mapread 0x00434f thru 0x005ef8 (0x1baa bytes) 000920 1787614992.569282 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x321bf): Mirror 9: 0x47b23892 (stale) Mirror 10: 0x47b23892 (stale) Mirror 11: 0x47b23892 (stale) Mirror 12: 0x47b23892 (stale) Mirror 14: 0x47b23892 (stale) Mirror 16: 0x47b23892 (stale) Mirror 17: 0x47b23892 (stale) Mirror 18: 0x47b23892 (stale) Mirror 19: 0x8bfabb4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321bf): Mirror 9: 0x47b23892 (stale) Mirror 10: 0x47b23892 (stale) Mirror 11: 0x47b23892 (stale) Mirror 12: 0x47b23892 (stale) Mirror 14: 0x47b23892 (stale) Mirror 16: 0x47b23892 (stale) Mirror 17: 0x47b23892 (stale) Mirror 18: 0x47b23892 (stale) Mirror 19: 0x8bfabb4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321bf: skipped 000921 1787614993.141925 write 0x012536 thru 0x014fee (0x2ab9 bytes) 000922 1787614993.169455 trunc from 0x0321bf to 0x00ab0f (0x276af bytes) 000923 1787614993.216774 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xab10): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xab10): Mirror 9: 0xb35ff014 (stale) Mirror 10: 0xb35ff014 (stale) Mirror 11: 0xb35ff014 (stale) Mirror 12: 0xb35ff014 (stale) Mirror 14: 0xb35ff014 (stale) Mirror 16: 0xb35ff014 (stale) Mirror 17: 0xb35ff014 (stale) Mirror 18: 0xb35ff014 (stale) Mirror 19: 0xb35ff014 (stale) CRC-32 checksum value for chunk [0, 0xab10): Mirror 1: 0xb35ff014 Mirror 20: 0xb35ff014 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xab10: skipped 000924 1787614994.187121 trunc from 0x004b91 to 0x00869d (0x3b0d bytes) 000925 1787614994.239159 read 0x009efe thru 0x00ab0f (0xc12 bytes) 000926 1787614994.245166 read 0x00734b thru 0x00ab0f (0x37c5 bytes) 000927 1787614994.258852 trunc from 0x001dc9 to 0x002121 (0x359 bytes) 000929 1787614994.264113 mapwrite 0x02367d thru 0x026e26 (0x37aa bytes) 000930 1787614994.345920 write 0x020f06 thru 0x022caa (0x1da5 bytes) 000931 1787614994.347312 punch from 0x001910 to 0x008d9a (0x748b bytes) 000932 1787614994.372527 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x26e27): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e27): Mirror 9: 0xd34e37c2 (stale) Mirror 10: 0xd34e37c2 (stale) Mirror 11: 0xd34e37c2 (stale) Mirror 12: 0xd34e37c2 (stale) Mirror 14: 0xd34e37c2 (stale) Mirror 16: 0xd34e37c2 (stale) Mirror 17: 0xd34e37c2 (stale) Mirror 18: 0xd34e37c2 (stale) Mirror 19: 0x4abbf94e (stale) Mirror 20: 0xd34e37c2 (stale) CRC-32 checksum value for chunk [0, 0x26e27): Mirror 1: 0x20cf9fcd Mirror 21: 0x20cf9fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e27: skipped 000933 1787614995.586316 fallocate 0x01875d thru 0x0226d7 (0x9f7b bytes) 000934 1787614995.660854 mapread 0x026439 thru 0x026e26 (0x9ee bytes) 000935 1787614995.668678 fallocate 0x02ff0b thru 0x03f407 (0xf4fd bytes) 000936 1787614995.676424 punch from 0x013b0b to 0x01fa80 (0xbf76 bytes) 000937 1787614995.691404 mapread 0x0054dc thru 0x014093 (0xebb8 bytes) 000938 1787614995.757124 mapread 0x003166 thru 0x009d4b (0x6be6 bytes) 000939 1787614995.762494 trunc from 0x03a296 to 0x03f407 (0x5172 bytes) 000940 1787614995.769093 write 0x01853b thru 0x01eaad (0x6573 bytes) 000941 1787614995.784254 trunc from 0x03f408 to 0x0279ab (0x17a5c bytes) 000942 1787614995.851852 mapwrite 0x0264e8 thru 0x02acd9 (0x47f2 bytes) 000943 1787614995.913745 mapread 0x01479c thru 0x0230c4 (0xe929 bytes) 000944 1787614995.927998 mapwrite 0x00e897 thru 0x0177a3 (0x8f0d bytes) 000945 1787614995.973257 mapwrite 0x002023 thru 0x004785 (0x2763 bytes) 000946 1787614996.002789 mapwrite 0x01da38 thru 0x02b5bf (0xdb88 bytes) 000947 1787614996.109196 punch from 0x01ebf0 to 0x02b5bf (0xc9d0 bytes) 000948 1787614996.120747 read 0x00512f thru 0x00bafc (0x69ce bytes) 000950 1787614996.126106 MIRROR_RESYNC 12 mirrors 000951 1787614998.395163 punch from 0x01c8d8 to 0x01d692 (0xdbb bytes) 000952 1787614998.434130 mapwrite 0x01390c thru 0x01991b (0x6010 bytes) 000953 1787614998.467990 trunc from 0x01ff09 to 0x025b7a (0x5c72 bytes) 000956 1787614998.473760 punch from 0x017a44 to 0x024701 (0xccbe bytes) 000957 1787614998.479185 fallocate 0x014646 thru 0x02421c (0xfbd7 bytes) 000958 1787614998.488131 MIRROR_RESYNC 12 mirrors 000959 1787615001.125164 MIRROR_RESYNC 12 mirrors 000960 1787615001.922549 punch from 0x01814b to 0x01f229 (0x70df bytes) 000961 1787615001.989091 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b5c0): Mirror 9: 0xa532bf14 (stale) Mirror 10: 0xa532bf14 (stale) Mirror 11: 0xa532bf14 (stale) Mirror 12: 0xa532bf14 (stale) Mirror 14: 0xa532bf14 (stale) Mirror 16: 0xa532bf14 (stale) Mirror 17: 0xa532bf14 (stale) Mirror 18: 0xa532bf14 (stale) Mirror 19: 0xa532bf14 (stale) Mirror 20: 0xa532bf14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b5c0): Mirror 9: 0xa532bf14 (stale) Mirror 10: 0xa532bf14 (stale) Mirror 11: 0xa532bf14 (stale) Mirror 12: 0xa532bf14 (stale) Mirror 14: 0xa532bf14 (stale) Mirror 16: 0xa532bf14 (stale) Mirror 17: 0xa532bf14 (stale) Mirror 18: 0xa532bf14 (stale) Mirror 19: 0xa532bf14 (stale) Mirror 20: 0xa532bf14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b5c0: skipped 000962 1787615002.637434 trunc from 0x0223c6 to 0x02826e (0x5ea9 bytes) 000963 1787615002.651200 trunc from 0x02b5c0 to 0x0095fb (0x21fc4 bytes) 000964 1787615002.702196 write 0x006113 thru 0x00cf50 (0x6e3e bytes) 000965 1787615002.715167 fallocate 0x00952d thru 0x00df66 (0x4a3a bytes) 000966 1787615002.726351 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdf67): Mirror 9: 0x4aa6f6e (stale) Mirror 10: 0x4aa6f6e (stale) Mirror 11: 0x4aa6f6e (stale) Mirror 12: 0x4aa6f6e (stale) Mirror 14: 0x4aa6f6e (stale) Mirror 16: 0x4aa6f6e (stale) Mirror 17: 0x4aa6f6e (stale) Mirror 18: 0x4aa6f6e (stale) Mirror 19: 0x4aa6f6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdf67): Mirror 9: 0x4aa6f6e (stale) Mirror 10: 0x4aa6f6e (stale) Mirror 11: 0x4aa6f6e (stale) Mirror 12: 0x4aa6f6e (stale) Mirror 14: 0x4aa6f6e (stale) Mirror 16: 0x4aa6f6e (stale) Mirror 17: 0x4aa6f6e (stale) Mirror 18: 0x4aa6f6e (stale) Mirror 19: 0x4aa6f6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdf67: skipped 000967 1787615003.325733 punch from 0x006527 to 0x00df66 (0x7a40 bytes) 000968 1787615003.330567 trunc from 0x0044d7 to 0x00df66 (0x9a90 bytes) 000969 1787615003.334597 trunc from 0x00df67 to 0x0261b8 (0x18252 bytes) 000970 1787615003.351285 punch from 0x0035a8 to 0x007d09 (0x4762 bytes) 000971 1787615003.356782 mapread 0x023127 thru 0x025f67 (0x2e41 bytes) 000972 1787615003.374051 trunc from 0x0261b9 to 0x0130b0 (0x13108 bytes) 000973 1787615003.390349 write 0x00f105 thru 0x018863 (0x975f bytes) 000975 1787615003.399736 punch from 0x001b37 to 0x00334c (0x1816 bytes) 000976 1787615003.404120 mapwrite 0x023243 thru 0x0283c3 (0x5181 bytes) 000977 1787615003.446180 punch from 0x02518a to 0x0283c3 (0x323a bytes) 000978 1787615003.451115 trunc from 0x0283c4 to 0x02d0f2 (0x4d2f bytes) 000979 1787615003.463492 write 0x0275cc thru 0x0304e7 (0x8f1c bytes) 000980 1787615003.473138 MIRROR_RESYNC 10 mirrors 000981 1787615006.857776 trunc from 0x0304e8 to 0x02f909 (0xbde bytes) 000982 1787615006.916835 trunc from 0x011cc8 to 0x02093c (0xec75 bytes) 000983 1787615006.921738 fallocate 0x006729 thru 0x0129ac (0xc284 bytes) 000984 1787615006.945709 MIRROR_RESYNC 10 mirrors 000985 1787615009.834048 trunc from 0x0109b6 to 0x01b510 (0xab5b bytes) 000987 1787615009.872270 read 0x0116bf thru 0x017d4e (0x6690 bytes) 000988 1787615009.885149 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 9: 0x395f2131 (stale) Mirror 10: 0x395f2131 (stale) Mirror 11: 0x395f2131 (stale) Mirror 12: 0x395f2131 (stale) Mirror 14: 0x395f2131 (stale) Mirror 16: 0x395f2131 (stale) Mirror 17: 0x395f2131 (stale) Mirror 18: 0x395f2131 (stale) Mirror 19: 0x395f2131 (stale) CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 1: 0x157797a3 Mirror 20: 0x157797a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f90a: skipped 000989 1787615011.087928 mapread 0x023952 thru 0x02e644 (0xacf3 bytes) 000990 1787615011.171649 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 9: 0x395f2131 (stale) Mirror 10: 0x395f2131 (stale) Mirror 12: 0x395f2131 (stale) Mirror 14: 0x395f2131 (stale) Mirror 16: 0x395f2131 (stale) Mirror 17: 0x395f2131 (stale) Mirror 18: 0x395f2131 (stale) Mirror 19: 0x395f2131 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 9: 0x395f2131 (stale) Mirror 10: 0x395f2131 (stale) Mirror 12: 0x395f2131 (stale) Mirror 14: 0x395f2131 (stale) Mirror 16: 0x395f2131 (stale) Mirror 17: 0x395f2131 (stale) Mirror 18: 0x395f2131 (stale) Mirror 19: 0x395f2131 (stale) CRC-32 checksum value for chunk [0, 0x2f90a): Mirror 1: 0x157797a3 Mirror 20: 0x157797a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f90a: skipped 000991 1787615012.174705 mapread 0x029947 thru 0x02f5a9 (0x5c63 bytes) 000992 1787615012.229132 read 0x01a187 thru 0x02115d (0x6fd7 bytes) 000993 1787615012.259599 read 0x00baa6 thru 0x00c74a (0xca5 bytes) 000994 1787615012.288698 mapread 0x01b4b4 thru 0x026bf9 (0xb746 bytes) 000995 1787615012.309128 write 0x022d26 thru 0x0265ac (0x3887 bytes) 000996 1787615012.394916 fallocate 0x036d9e thru 0x03ffff (0x9262 bytes) 000997 1787615012.404628 trunc from 0x02f90a to 0x01403a (0x1b8cf bytes) 000998 1787615012.430796 fallocate 0x001759 thru 0x008108 (0x69b0 bytes) 000999 1787615012.435804 mapread 0x01223a thru 0x01403a (0x1e01 bytes) 001000 1787615012.451031 mapwrite 0x01e5f8 thru 0x028565 (0x9f6e bytes) All operations completed A-OK! PASS 70a (110s) == sanity-flr test 71: check mirror extend parent fid ==== 19:43:37 (1787615017) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1fe:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 19:43:45 (1787615025) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223571 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112222 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 19:43:52 (1787615032) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7509696 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7191424 .. done Extending file size to 8799200 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7921600 .. Extending file size to 9234432 .. Extending file size to 8890464 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195264 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9253344 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9292896 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61537 61538 61539 61543 61544 done 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:45:14 (1787615114) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3427200 .. Extending file size to 4940032 .. done Extending file size to 7615104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8514944 .. failed Extending file size to 3674176 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3865568 .. Extending file size to 8340864 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9090976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9095808 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9128352 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63417 63419 63420 63421 63422 failed 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:46:29 (1787615189) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17816 3589204 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25596 3581424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43412 7170628 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17816 3589204 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25596 3581424 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43412 7170628 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103673 s, 20.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183959 s, 11.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x80b12305 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:46:47 (1787615207) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124405 s, 16.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 19:46:54 (1787615214) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:47:00 (1787615220) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0881641 s, 23.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:47:06 (1787615226) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0785088 s, 26.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x275:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x275:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x276:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:47:13 (1787615233) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136045 s, 15.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x278:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:47:44 (1787615264) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660743 s, 15.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27b:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:48:26 (1787615306) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131945 s, 15.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:49:14 (1787615354) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg307-server: Pool lustre.test_204f created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122383 s, 17.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg307-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:50:21 (1787615421) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:50:27 (1787615427) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:50:34 (1787615434) oleg307-server: Pool lustre.test_206 created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg307-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:51:02 (1787615462) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645777 s, 16.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406472 s, 25.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6dfb7b1d Mirror 3: 0x6dfb7b1d PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:51:11 (1787615471) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:51:13 (1787615473) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:51:16 (1787615476) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0605208 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1010000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1010000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1010000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d60000 Read this (4 bytes): BBBB 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:51:49 (1787615509) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0197574 s, 415 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:51:55 (1787615515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354959 s, 29.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611260 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 19:51:56.000000000 -0400 Modify: 2026-08-24 19:51:56.000000000 -0400 Change: 2026-08-24 19:51:57.000000000 -0400 Birth: 2026-08-24 19:51:55.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:52:01 (1787615521) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447017 s, 23.5 MB/s oleg307-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg307-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg307-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:52:20 (1787615540) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.179196 s, 229 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.030717 s, 133 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 19:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x81:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 19:52 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:52:26 (1787615546) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:52:33 (1787615553) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1694 sec =========== 19:52:42 (1787615562) === sanity-flr: start cleanup 19:52:43 (1787615563) === === sanity-flr: finish cleanup 19:52:51 (1787615571) ===